Replicate a bucket
Copy every object written to a bucket into another bucket of your organization, usually in the other zone, so the data survives the loss of a zone.
Replication is S3's cross-region replication (CRR), between the zones of a region. Replace example-uploads with the source bucket and example-uploads-nbg with the destination.
Prerequisites
- Versioning is on in both buckets (see Turn on versioning).
- Permission for
s3:PutReplicationConfigurationon the source bucket, ands3:PutObjecton the destination's objects: the platform writes into it on your behalf. Stopping replication needss3:DeleteReplicationConfiguration. - In the console, also
s3:GetBucketon the source bucket to open it (without it the bucket's page says "Access denied"). - Neither bucket is one the platform manages.
- Both buckets are in a zone, not on your organization's own storage.
- A destination in Helsinki (
fi1a) for a source in Germany needs Allow EU locations outside Germany turned on.
Add a replication rule
- Open the source bucket, then choose Replication.
- Choose Add rule. It is unavailable until versioning is on in this bucket.
- (Optional) In Add a replication rule, for Name, enter a name for the rule in place of
the suggested
rule-1. - For Destination bucket, choose the destination. Buckets are grouped under Same zone and Other zones; the first bucket in another zone is chosen for you.
- (Optional) For Prefix, enter the prefix to copy, for example
photos/. - (Optional) Clear Copy existing objects too to copy only objects written from now on.
- (Optional) Select Replicate deletes.
- Choose Add rule.
The Status section appears and shows what is still to copy.
dcloud s3 bucket-replication set example-uploads \
--rules '{"destination":"example-uploads-nbg","prefix":"","deleteMarkerReplication":true}'PUT /s3/api/buckets/example-uploads/replication
{ "rules": [{ "destination": "example-uploads-nbg", "prefix": "", "deleteMarkerReplication": true }] }resource "dcloud_bucket_replication" "uploads" {
bucket = dcloud_bucket.uploads.id
rules = jsonencode([
{
destination = dcloud_bucket.uploads_copy.id
prefix = ""
deleteMarkerReplication = true
existingObjects = true
},
])
}A set replaces every rule. Over S3, aws s3api put-bucket-replication --bucket example-uploads --replication-configuration file://replication.json works too; its Role is accepted and not
used.
Rule fields
| Field | Default | Meaning |
|---|---|---|
destination |
(required) | A bucket name, arn:aws:s3:::<name> or its rn:ews: name |
prefix |
"" |
Only keys starting with it |
existingObjects |
true |
Also copy what the bucket held before; false copies new writes only |
deleteMarkerReplication |
false |
Copy a delete as a delete marker; the destination keeps the versions |
enabled |
true |
|
id, priority |
rule-1…; the first rule highest |
Each rule needs its own id and a different priority; for one destination, the matching rule with the highest priority applies |
Up to 100 rules per bucket.
How it behaves
- A write is copied within seconds. The platform also compares each replicating bucket with its destinations in full once an hour, so nothing missed stays missed.
- Existing objects are copied too, unlike on AWS, unless
existingObjectsisfalse. - A delete is copied only with
deleteMarkerReplication, and then as a delete marker. Deleting a specific version is never copied. - The Status section shows Still to copy, Failed, Oldest waiting, Last full
comparison and the recent failures with their error. Retry failed (or
dcloud s3 bucket-replication retry example-uploads) gives failed keys new attempts. - While a bucket replicates, its versioning cannot be paused, and the destination cannot be deleted.
Stop replicating
Delete the last rule in the console and confirm Stop replicating, or run
dcloud s3 bucket-replication delete example-uploads. Replicas already made stay.
If it is refused
All with 400:
Versioning must be enabled on example-uploads before replication can be configured, orVersioning must be enabled on the destination bucket example-uploads-nbg.The destination bucket example-uploads-nbg does not exist.A bucket cannot replicate to itself.example-uploads-nbg already replicates to example-uploads; two-way replication is not supported.example-uploads-nbg is in fi1a (…), outside Germany: an administrator has to allow EU locations outside Germany first (Settings → Data location).… is on its organization's own object store, which does not replicate.… is managed by the platform.Rule ID … is used twice,Each rule needs a different Priority.
Without s3:PutObject on the destination the request is refused with 403.