S3 implementation

Upcoming. This page describes the s3 branch. It is not part of a release, and details may change.

These pages are for a reader who wants to open the code of the S3 endpoint or to build on it. The gateway is a Maven module of its own with the HTTP server, Signature Version 4, the namespace index of buckets, versions and uploads, and a local cache. It knows nothing of gridnodes: the application module starts it and hands it network storage through two small interfaces.

The module

The gateway is the Maven module hedgehog-s3, in the directory s3/, and the JPMS module org.unigrid.hedgehog.s3. It uses no CDI and depends on hedgehog-common only among the Hedgehog modules. It is built on Netty, Jersey 3.0.18, JAXB for the XML of the S3 API, and Jackson for the JSON of POST policies. The application module depends on it (requires org.unigrid.hedgehog.s3).

Package Holds Detail
spi The interfaces towards the node and their exceptions below
server S3Server, the Netty pipeline and the flow control of request and response bodies S3 server and authentication
auth SignatureFilter, Signature Version 4, aws-chunked bodies, presigned URLs and POST policies S3 server and authentication
http The virtual-host and request-id filters, the subresource names, payload checks and form parsing S3 server and authentication
resource The Jersey resources ServiceResource, BucketResource and ObjectResource and their helpers S3 blob layer and cache, S3 features
request Bucket names, keys, byte ranges and preconditions as a request states them S3 features
entity The XML documents of the S3 API, read and written by S3Xml S3 features
error S3ErrorCode, S3Exception and S3ExceptionMapper S3 server and authentication
index The namespace index: Namespace, Tree, Journal, runs, manifests and listings S3 namespace index
cache CachingBlobStore, the local cache over any blob store S3 blob layer and cache
checksum The checksum algorithms, among them Crc64Nvme S3 features
lifecycle LifecycleSweeper and the lifecycle rules S3 features

The interfaces towards the node

The package spi is the only boundary between the gateway and the node.

Type Job
BlobStore Stores a stream as an immutable blob and answers its BlobId, opens a Blob, and deletes one
Blob size(), writeTo(out), and a default ranged writeTo(out, offset, length) that streams the whole blob and drops what lies outside the range
BlobId The id of a blob. Whoever holds it can read the blob, so its toString() is BlobId[redacted]
CredentialStore Answers the secret of an access key id
BlobUnavailableException, BlobNotFoundException Storage that cannot serve now, answered with 503, and a blob that does not exist

In the daemon

Three classes of the application module tie the gateway to the node. Paths are relative to application/src/main/java/org/unigrid/hedgehog/.

S3Gateway (server/s3/S3Gateway.java) is an @Eager @ApplicationScoped bean. With --no-s3 it does nothing more than log that the endpoint is disabled, so there is no endpoint, index or lifecycle sweeper. Otherwise it puts a CachingBlobStore over a StorageBlobStore and builds the rest bottom up: the namespace, opened from s3/index in the data directory or mounted from --s3root, then the lifecycle sweeper, and the server last.

Opening the namespace may read its manifest from network storage, which a freshly started node cannot reach until it knows enough gridnodes. A BlobStoreException therefore leaves the daemon running, and the gateway tries again every 10 seconds on the daemon thread s3-start, so the port is bound once the index opens. Any other failure stops the daemon at start, or stops the retries later. On shutdown the gateway cancels the retries and, if a server started, closes the sweeper, then the server, flushes the namespace and closes it. A failed flush is only logged, as its writes stay in the journal for the next start.

StorageBlobStore (service/storage/StorageBlobStore.java) implements BlobStore over the StorageService of Network storage; see S3 blob layer and cache.

S3Credentials (server/s3/S3Credentials.java) implements CredentialStore over the file s3/credentials, which the hedgehog s3 key commands in command/s3/ also change. The options are in command/option/S3Options.java.

Table of contents