3 S3 (Storage)

S3 provides a fairly simple and REST-ful interface. Uploading an object to S3 is an HTTP PUT request. Download an object is a GET request. And so on. As a result, you may feel you don’t need a lot of “wrapper” around this.

Where you definitely will want help is constructing the Authorization header S3 uses to authenticate requests. Doing so requires making a string out of specific elements of your request and “signing” it with your AWS private key. Even a small discrepancy will cause the request to fail authentication. As a result, aws/s3 makes it easy for you to create the authentication header correctly and successfully.

Plus, aws/s3 does provide wrappers and tries to help with some wrinkles. For example, S3 may give you a 302 redirect when you do a PUT or POST. You don’t want to transmit the entire body, only to have S3 ignore it and you have to transmit it all over again. Instead, you want to supply the request header Expect: 100-continue, which lets S3 respond before you transmit the body.

3.1 Request Method

3.2 Endpoint

Added in version 1.1 of package aws.

When S3 returns certain 50x response codes, an additional (sub1 (s3-max-tries)) attempts will be made. If none succeed, then a exn:fail:aws exception is raised.

A value of 1 means try just once, in other words do not retry.

Added in version 1.7 of package aws.

3.3 Authentication signatures

3.4 Conveniences

Keep in mind that bucket names on S3 are global—shared among all users of S3. You may want to make your bucket names include a domain name that you own.

If you try to create a bucket with a name that is already used by another AWS account, you will get a 409 Conflict response.

If you create a bucket that already exists under your own account, this operation is idempotent (it’s not an error, it’s simply a no-op).

Use valid-bucket-name? to check the validity of the bucket-name you want to use.

This operation is idempotent (it is a no-op to delete a bucket that has already been deleted).

When dealing with arbitrary buckets, you might need to parameterize s3-region to this value because AWS Signature v4 requires a region to be specified. For example:

(parameterize ([s3-region (bucket-location "my-bucket")])

(ls "my-bucket/"))

Added in version 1.6 of package aws.

For each such batch, proc is called. The first time proc is called, its first argument is the init value; subsequent times it’s given the previous value that it returned.

The second argument to proc is a (listof xexpr?), where each xexpr? respresents XML returned by S3 for each object. The XML is the Contents or CommonPrefixes portions of the ListBucketResults XML response, where CommonPrefixes are produced only for a non-#f delimiter.

The return value of ls/proc is the final return value of proc.

For example, ls is implemented as simply:

(map (λ (x) (se-path* '(Key) x))

(ls/proc b+p append '()))

• an xexpr (as with ls/proc)

• response headers from a HEAD request (as with head)

• an xexpr representing the ACL (as with get-acl)

• name (as with ls)

• response headers from a HEAD request (as with head)

• an xexpr representing the ACL (as with get-acl)

Copy an existing S3 object bucket+path/from to bucket+path/to, including its metadata. Both names are of the form "bucket/path/to/resource".

It is not an error to copy to an existing object (it will be replaced). It is even OK to copy an existing object to itself, as long as heads implies metadata changes.

Changed in version 1.8 of package aws: Added the heads argument.

S3 responds with an XML representation of the ACL, which is returned as an xexpr?.

If acl is #f, then the request body is empty and ACL changes must be provided by heads (e.g., as a canned ACL using 'x-amz-acl).

Changed in version 1.8 of package aws: Added the heads argument and allow #f for acl.

Make a GET request for bucket+path (which is the form "bucket/path/to/resource").

The reader procedure is called with an input-port? and a string? respresenting the response headers. The reader should read the response body from the port, being careful to read the exact number of bytes as specified in the response header’s Content-Length field. The return value of reader is the return value of get/proc.

You may pass request headers in the optional heads argument.

The optional arguments range-begin and range-end are used to supply an HTTP Range request header. This header, which Amazon S3 supports, enables a getting only a subset of the bytes. Note that range-end is exclusive to be consistent with the Racket convention, e.g. subbytes. (The HTTP Range header specifies the end as inclusive, so your range-end argument is decremented to make the value for the header.)

You may pass request headers in the optional heads argument.

The optional arguments range-begin and range-end are used to supply an optional Range request header. This header, which Amazon S3 supports, enables a getting only a subset of the bytes. Note that range-end is exclusive to be consistent with the Racket convention, e.g. subbytes. (The HTTP Range header specifies the end as inclusive, so your range-end argument is decremented to make the value for the header.)

The response body is held in memory; if it is very large and you want to "stream" it instead, consider using get/proc.

You may pass request headers in the optional heads argument.

Makes a PUT request for bucket+path (which is the form "bucket/path/to/resource"), using the writer procedure to write the request body and the reader procedure to read the response body. Returns the response header (unless it raises exn:fail:aws).

The writer procedure is given an output-port? and a string? representing the response headers. It should write the request body to the port. The amount written should be exactly the same as data-length, which is used to create a Content-Length request header. You must also supply mime-type (for example "text/plain") which is used to create a Content-Type request header.

The reader procedure is the same as for get/proc. The response body for a PUT request usually isn’t interesting, but you should read it anyway.

Note: If you want a Content-MD5 request header, you must calculate and supply it yourself in heads. Supplying this allows S3 to verify the upload integrity.

chunk-len determines the length of the Content-Encoding: aws-chunked chunks used to perform AWS Signature v4 chunked uploads.

To use reduced redundancy storage, supply (hasheq 'x-amz-storage-class "REDUCED_REDUNDANCY") for heads.

Makes a PUT request for bucket+path (which is the form "bucket/path/to/resource"), sending data as the request body and creating a Content-Type header from mime-type. Returns the response header (unless it raises exn:fail:aws).

A Content-MD5 request header is automatically created from data. To ensure data integrity, S3 will reject the request if the bytes it receives do not match the MD5 checksum.

To use reduced redundancy storage, supply (hasheq 'x-amz-storage-class "REDUCED_REDUNDANCY") for heads.

Upload file to bucket+path and return the response header (or raise exn:fail:aws).

The #:mode-flag argument is identical to that of call-with-input-file*.

If #:mime-type is #f, then the Content-Type header is guessed from the file extension, using a (very short!) list of common extensions. If no match is found, then "application/x-unknown-content-type" is used. You can customize the MIME type guessing by setting the path->mime-proc parameter to your own procedure.

A Content-MD5 request header is automatically created from the contents of file. To ensure data integrity, S3 will reject the request if the bytes it receives do not match the MD5 checksum.

A Content-Disposition request header is automatically created from file. For example if file is "/foo/bar/test.txt" or "c:\\foo\\bar\\test.txt" then the header "Content-Disposition:attachment; filename=\"test.txt\"" is created. This is helpful because a web browser that is given the URI for the object will prompt the user to download it as a file.

To use reduced redundancy storage, supply (hasheq 'x-amz-storage-class "REDUCED_REDUNDANCY") for heads.

3.5 Multipart uploads

In addition to uploading an entire object in a single PUT request, S3 lets you upload it in multiple 5 MB or larger chunks, using the multipart upload API. Amazon recommends using this when the total data to upload is bigger than about 100 MB.

3.5.1 Convenience

Each part must be at least s3-multipart-size-minimum, except the last part.

The parts are uploaded using a small number of worker threads, to get some parallelism and probably better performance.

Changed in version 1.7 of package aws: Worker threads handle exceptions by returning work to the end of the to-do list to retry later, but no sooner than a delay that increases after each such retry.

The parts are uploaded using a small number of worker threads, to get some parallelism and probably better performance.

Although it’s usually desirable for part-size to be as small as possible, it must be at least 5 MB, and large enough that no more than 10,000 parts are required. When part-size is #f, the default, a suitable minimal size is calculated based on the file-size of file.

Use list-multipart-uploads to look for a multipart-put/file of bucket+path and file that was interrupted (neither complete-multipart-upload nor abort-multipart-upload was called and succeeded). If such an upload is found, use list-multipart-upload-parts to determine which of the previously uploaded parts have MD5 checksums that match the corresponding parts of file (that is, parts that do not remain to be uploaded). If any do, return the upload ID and a list of those parts. Otherwise return #f.

You may call this to determine whether resume-multipart-put/file would attempt to do anything, for example if you want to get user confirmation.

Added in version 1.5 of package aws.

If incomplete-multipart-put/file returns a non #f value, use the information to resume the upload by upload-part-ing the remaining parts, calling complete-multipart-upload, and returning the upload ID. Otherwise return #f.

Added in version 1.5 of package aws.

3.5.2 Building blocks

Use these if the data you’re uploading is computed on the fly and you don’t know the total size in advance. Otherwise you may simply use multipart-put or multipart-put/file.

Note that S3 part numbers start with 1 (not 0).

bstr must be at least s3-multipart-size-minimum, unless it’s the last part.

Returns a cons of part-number and the ETag for the part. You will need to supply a list of these, one for each part, to complete-multipart-upload.

Added in version 1.3 of package aws.

Returns S3’s XML response in the form of an xexpr?.

3.6 S3 examples