Upload and download objects
Browse a bucket by folder, upload files of up to 5 TiB from the browser, resume an interrupted upload, and download objects, without an access key.
The console acts as you, with a credential that lives 20 minutes, so what you see and may do is exactly what your policies and the bucket policy allow; no access key is needed. Uploads and downloads go straight between your browser and the bucket with a presigned URL for one object, valid 15 minutes. Replace example-uploads with your bucket's name.
Prerequisites
- To browse:
s3:ListObjects. To download:s3:GetObject. To upload:s3:PutObject. To cancel or discard an unfinished upload:s3:AbortMultipartUpload. - In the console, also
s3:GetBucketon the bucket to open it (without it the bucket's page says "Access denied"). - The bucket is not one the platform writes: a database's backup bucket can be browsed and downloaded, not changed.
Browse
- Open the bucket, then choose Objects. The list shows folders (prefixes ending in
/) and objects, 200 at a time; Show more loads the next 200. - Choose a folder's name to open it; the breadcrumb leads back.
dcloud s3 bucket-objects list example-uploads --prefix photos/GET /s3/api/buckets/example-uploads/objects?prefix=photos/Pass nextContinuationToken back as continuationToken for the next page.
Upload
- Open the folder to upload into.
- Choose Upload and pick the files, or drop files anywhere on the page.
Each file's row shows the bytes sent (and for a large file the parts). Three files upload at a time.
dcloud s3 bucket-objects upload-url example-uploads --key photos/cat.jpg --content-type image/jpegThen PUT the file to the returned url with the returned headers.
POST /s3/api/buckets/example-uploads/objects/upload-url
{ "key": "photos/cat.jpg", "contentType": "image/jpeg" }Large files
In the console, a file of 100 MiB or more is uploaded in parts (a multipart upload): 16 MiB parts, larger for very large files so that there are at most 10,000, four at a time, each tried up to three times, waiting longer after each failure. Files up to 5 TiB work; a larger one fails with "larger than an object can be (5 TiB)".
- × on the row cancels; the parts sent so far are deleted.
- If the page is reloaded or the connection drops, the upload is listed under Unfinished uploads in that browser. Resume… asks for the same file again (same name, size and modification time; another file is refused with "That is not the file this upload was started with: …"), sends only the parts the bucket does not have yet, and completes the object.
- Discard deletes the parts, at once.
- Until an upload is completed or discarded, its parts count as stored data.
Over the API the same is multipart-create (answers uploadId), multipart-part-urls
(presigned part URLs, up to 100 per call, valid 15 minutes), PUT of each part (keep the ETag
it answers), multipart-parts (what is stored so far), multipart-complete with every part in
ascending order, and multipart-abort. A part other than the last that is too small fails the
completion with EntityTooSmall; the console's 16 MiB parts never are.
Download
- In the object's menu, choose Download.
dcloud s3 bucket-objects download-url example-uploads --key photos/cat.jpgPOST /s3/api/buckets/example-uploads/objects/download-url
{ "key": "photos/cat.jpg" }The answer's url downloads the object for 900 seconds.
How the browser reaches the bucket
The bucket lets the Deutsche Cloud Console's own origin, and only it, read the answers to presigned PUT,
GET and HEAD requests on object keys once their signature has verified, including the ETag of
each uploaded part. So browser uploads work without a CORS rule on the bucket. Every other origin is
decided by the bucket's CORS rules (see Let a web page use the bucket). The
console's origin never gets DELETE, POST or bucket settings from the browser: starting,
completing and aborting a multipart upload go through the console's API.
For many files, an S3 client is faster (see Use the AWS CLI).
If it is refused
- 403
The bucket refused it: … (your policies or the bucket policy). - Without the permission, the console disables the button and says why, for example "Uploading needs s3:PutObject, which your policies do not grant."
Related
Related
- GET /s3/api/buckets/{name}/objects
- POST /s3/api/buckets/{name}/objects/upload-url
- POST /s3/api/buckets/{name}/objects/download-url
- POST /s3/api/buckets/{name}/objects/multipart
- POST /s3/api/buckets/{name}/objects/multipart/part-urls
- POST /s3/api/buckets/{name}/objects/multipart/parts
- POST /s3/api/buckets/{name}/objects/multipart/complete
- POST /s3/api/buckets/{name}/objects/multipart/abort