LIAS Storage Layer

Concepts and Design

BobKrzaczek 2008-Jun-09

Goals

• No Special Protocols or Software Required
  • Basic Access From a Web Browser
  • No Assumptions about Data Processing Intentions

• Low Overhead
  • Don't Consume Resources Unnecessarily
    • Network Bandwidth
    • CPU Power
  • Accessible by Mobile Platforms

• Useful for Multiple Projects in Multiple Contexts
  • WASP
  • Krem's Ground Sensors
  • LARCH

Low Overhead, Low Complexity

• Past Experience
  • CORBA
  • XML-RPC
  • RPC

• Others
  • WS (Web Services)
  • SOAP
  • DCOM

• Those Aren't It!
  • Special Libraries
  • Complexity
  • Favored Platforms (e.g., hardware, operating systems)
  • "Bonus": Many Surprising Assumptions Lurk Within Each...

REST

Representational State Transfer

• First Appeared in Roy Fielding's 2000 Ph.D. Dissertation

• Roy Fielding?
  • Principal Author of RFC 1945, 2068, 2145, 2616 (what eventually became HTTP 1.1) with Tim Berners-Lee, Jim Gettys, et. al.
  • Co-founder of the Apache Server Project
  • Member of the IETF Working Groups covering web infrastructure (e.g., HTTP, URI, HTML)

• Yes, "that" Roy Fielding...

REST, Representational State Transfer

• Introduced the Concept of "Web Services"
  (But don't let WS-* distract you)

• An Approach to Distributed Computing

• A Distinct Alternative to the RPC Paradigm (e.g., CORBA)
  • Low Overhead
  • Low Complexity
  • Widespread Support
  • Builds on Existing Infrastructure
  • No Favored Platforms

REST, Representational State Transfer

Three Orthogonal Elements of REST

• Verbs
  • Constrained
  • HTTP methods (GET, POST, PUT, DELETE)
  • Optional: Include WebDAV

• Nouns
  • Unconstrained
  • URI

• Content Type
  • Somewhat Constrained (MIME, Encodings, Transfer Types)
  • Text: Usually XML, but can use HTML, JSON, ...
  • Interesting Possibilities for On-The-Fly Image Conversion

REST Verbs

• Implemented as HTTP Methods
  • GET
  • POST
  • PUT
  • DELETE

• Extended with WebDAV
  • COPY
  • MOVE
  • LOCK
  • ... more ...

REST Verbs

• Implemented as HTTP Methods
  • GET
  • POST
  • PUT
  • DELETE

• Extended with WebDAV
  • COPY
  • MOVE
  • LOCK
  • ... more ...

• But We Don't Need or Want All of That
  • LIAS Archive Only Stores and Provides Data
  • Never Delete
  • Never Rename or Relocate

REST Data (Content)

• Document-Oriented, or Data-Oriented, rather than Object-Oriented

• Usually Based in XML or Something Similar in Intent

• Data, Formats, Encodings, and Transfers are Advertised at Runtime

• Can Be Negotiated
  • Client Can Say, "I prefer XML but will accept HTML."
  • Server Can Make Decisions Based on Client Preferences

• Leads to Interesting non-Text Application Possibilities
  • Image Conversion
  • Data Resampling

REST Nouns

"The interesting part..."_

• URI are Nouns

• When a Regular Naming Scheme is Adopted, only a few Verbs are Required for a Rich Interface

GET /people
GET /people/krzaczek
GET /people/krzaczek/phone

PUT /people/krzaczek/photo

POST /people/krzaczek/new-email

REST Nouns

Name Space is Partitioned However it Best Fits the Application

GET  /people/krzaczek/emailaddress
GET  /people/krzaczek/inbox
POST /people/krzaczek/send-message

Or...

GET  /emailaddresses/krzaczek
GET  /inboxes/krzaczek
POST /sendmail/krzaczek

As Long As the Names are Regular and Predictable, REST Implementations Remain Easy

REST Is Simple

This Is Key!

• Stateless

• Easily Layered With or Forwarded Through Other Protocols and Devices
  • Caches
  • Gateways
  • Firewalls
  • Proxies
  • ... all invisible to the end REST clients and servers!

• Much Easier to Change/Update/Evolve Interfaces
  • No IDL
  • No rpcgen
  • No ...

REST Scales

• Multiple Servers
  • Service Partitioning
  • Load Balancing (e.g., Round Robin)
  • Fault Tolerance

• Caching
  • Static/Dynamic Content Management
  • Proxies

Remember: It Builds on Existing Web Technologies

LIAS Storage System

• Resource Discovery, returns text with hypertext (links)

GET /lss
GET /lss/MISI
GET /lss/WASP
GET /lss/WASP/2003
GET /lss/WASP/2003/07
GET /lss/search?q="Spencerport"&p="WASP"
GET /lss/search?lat="43.0"&lon="-77.6"&rad="10km"

LIAS Storage System

• Image Access, returning data

GET /lss/WASP/2003/07/29/raw/SWIR099.hdr
GET /lss/WASP/2003/07/29/raw/SWIR099.img

LIAS Storage System

• Upload of Many Files from the Field

PUT /lss/Krem/2008/06/15/ground.csv
PUT /lss/Krem/2008/06/15/cam1.avi
PUT /lss/Krem/2008/06/15/cam2.avi
PUT /lss/Krem/2008/06/15/cam3.avi

• Single Upload from the Field
  • We can be smart enough to "unpack" the single upload into a tree

PUT /lss/Krem/2008/06/15/data.zip

• Easily Implemented with your Favorite Scripting Language

• Or Even Performed Manually from a Web Browser

Metadata

• We Cannot Predict the Names of Files used in Future Products

• We Cannot Predict All Future Metadata Needs

So... Store Simple Text Associations in a Parallel Tree

• No Matter What Names are Used, No Conflicts

• No Fixed Keys in the Associations

• Future Additions Do Not Invalidate Previous Expressions

Metadata

• Metadata is Stored in a Matching Directory

• Metadata is Contained in a Single File

• Easy to Extend Later
  • Add New Files
  • Take "metadata" as the sum of all available content

• Newer Files Override Older Files

• Maintains Our Stance on Never Editing nor Deleting Data

Metadata

/ARCHIVE/data/WASP/Spencerport 10k/raw/SWIR010.hdr
/ARCHIVE/data/WASP/Spencerport 10k/raw/SWIR010.img
/ARCHIVE/data/WASP/Spencerport 10k/raw/MWIR010.hdr
/ARCHIVE/data/WASP/Spencerport 10k/raw/MWIR010.img
/ARCHIVE/data/WASP/Spencerport 10k/applanix/DEFAULT-001.dat

/ARCHIVE/meta/WASP/Spencerport 10k/info

Metadata

Metadata

Later, we can add a second metadata file in the meta tree.

/ARCHIVE/meta/WASP/Spencerport 10k/info
/ARCHIVE/meta/WASP/Spencerport 10k/geo

And now the metadata for "Spencerport 10k" is the aggregate of the info and geo files.

• Set TITLE = LIAS Storage Layer