Troubleshooting Object Storage
Errors S3 clients and the console can show for a bucket, with their cause and fix.
SignatureDoesNotMatch
Cause: the wrong secret, or a client that rewrites the host.
Fix: copy the key again; set the endpoint to https://s3.<domain> exactly.
InvalidAccessKeyId
Cause: the key was deleted or is inactive ("The access key ID you provided does not exist in our records").
Fix: activate the key, or create a new one (see Create an access key).
AccessDenied "This key has expired" or "The key’s owner may no longer act in this organization"
Cause: the access key expired, or its IAM user was deactivated or the person left the organization.
Fix: create a new key for an active user.
AccessDenied on a bucket you can see in the console
Cause: your policies, or the key's scope, do not grant the action on that object.
Fix: add the action for the bucket or prefix (see Access control). A key limited to one bucket cannot list your buckets: name the bucket in every command.
AccessDenied "This bucket is managed by the platform and is read-only"
Cause: a write to a bucket the platform manages, such as a database's backup bucket (rds-…).
Fix: only read it; restore a database through the database.
RequestTimeTooSkewed
Cause: the client's clock is more than 15 minutes off.
Fix: synchronise the clock.
AccessDenied on put-bucket-policy
Cause: the policy is public and Block public access is on.
Fix: allow public access on the bucket first (see Make objects public).
The website answers 403 for every page
Cause: nothing in the bucket is public; the console shows On, not public.
Fix: allow public access and add a public-read policy.
A custom domain has no certificate
Cause: the name does not resolve to Deutsche Cloud yet, was not added, or the bucket's website is off.
Fix: check the CNAME; turn the website on and add the domain on the bucket's Website tab (see Use a custom domain for a website).
ExpiredToken or InvalidToken
Cause: temporary credentials past their expiresAt (ExpiredToken); a session token missing,
wrong, or sent with a permanent key (InvalidToken).
Fix: create new temporary credentials; set AWS_SESSION_TOKEN for temporary credentials only.
KeyTooLongError
Cause: an object key longer than 1024 bytes.
Fix: use a shorter key.
AccessControlListNotSupported
Cause: an upload with an ACL that grants anyone else (--acl public-read, or any
x-amz-grant-* header); only private and bucket-owner-full-control are accepted.
Fix: leave out the ACL; publish with a bucket policy (see Make objects public).
NotImplemented on CreateBucket or DeleteBucket
Cause: buckets are not created or deleted over S3.
Fix: use the console, dcloud s3 buckets create or dcloud s3 buckets delete. For rclone,
set no_check_bucket = true.
KMS.DisabledException
Cause: the bucket's KMS key is disabled. Disabling a key stops its buckets within 30 seconds.
Fix: enable the key; reads and writes work again with the next request.
InsufficientStorage (507)
Cause: on a storage cluster, its fullest building reached its fill limit; on your own hosts, the store is 95 % full.
Fix: delete objects (reads and deletes still work); ask for more capacity.
QuotaExceeded (403) on own storage
Cause: your organization reached its s3.stored-bytes quota.
Fix: delete objects, or ask for a higher quota (see Quotas).
ServiceUnavailable (503) on a write
Cause: a building of the storage cluster is down or cut off (Retry-After: 10), the cluster
is unavailable, or your hosts' object store is down.
Fix: retry with backoff; every S3 SDK does.
SlowDown (503)
Cause: the object store kept asking to slow down after the gateway's own retries.
Fix: retry with backoff.
Versioning or lifecycle is refused with 409 on a storage-cluster bucket
Cause: the bucket shares storage with others and already holds objects.
Fix: set it on an empty bucket, or copy the objects into a new bucket and set it there first (see Storage locations).
A bucket shows entries storage cannot read
Cause: rarely, the storage underneath keeps listing an object whose data is gone, and every request for it fails. The bucket shows "N entries in this bucket cannot be read or deleted." A delete that hits such an entry is never reported as done.
Fix: none on your side. Deutsche Cloud is notified and repairs the bucket by moving everything readable to fresh storage; until then it cannot be deleted. Nothing else in it is affected.