In Ceph S3 RGW, “replication policy” can mean three different things, and mixing them up causes most of the confusion:
- Ceph storage replication underneath RGW
- RGW multisite sync policy between zones
- S3 bucket replication rules exposed through the S3 API
They are related, but they solve different problems.
Start with the two kinds of replication
Think of it as two layers:
Layer 1: inside one Ceph cluster. Ceph protects object data in pools using either replicated pools or erasure coding. That is storage durability, not S3-style cross-site replication.
Layer 2: between RGW sites/zones. RGW multisite moves bucket metadata and object changes between zones in a zonegroup. A multi-zone setup is the foundation for disaster recovery and geo-distribution.
So when someone says “replication policy” for RGW S3, they usually mean Layer 2.
What a replication policy is in RGW
In modern RGW multisite, the policy is called a sync policy. It gives bucket-level, fine-grained control over how data moves between zones. Unlike older “everything mirrors everywhere” behavior, sync policy allows buckets to diverge and lets data flow be symmetric or one-way/directional.
That means you can express patterns like:
- replicate bucket A from Zone A → Zone B only
- replicate between A ↔ B
- allow replication only for selected buckets
- choose specific destination zones for a bucket’s sync behavior
How this relates to the S3 API
Ceph RGW implements S3 bucket replication only partially. The Ceph docs say bucket replication is supported partially and is permitted only across zones. RGW does not currently support AWS-style replication within the same zone. It also adds a Ceph-specific Zone array so users can choose which zones receive replicated bucket data.
So the practical translation is:
- AWS S3 CRR/SRR mindset: “replicate this bucket according to replication rules”
- Ceph RGW reality: “replicate this bucket across RGW multisite zones, controlled by sync policy”
Important distinction: replication policy is not a bucket policy
A bucket policy in S3/RGW is an access-control document: who can read, write, list, or modify objects. It is managed through standard S3 operations. It does not control multisite data movement.
So: Bucket policy = permissions. Replication/sync policy = where object changes get copied. Pool replication = how Ceph stores copies internally.
Topology terms that matter
To understand RGW replication policies, these are the core objects:
- Realm: global multisite namespace/scope
- Zonegroup: grouping of zones
- Zone: an RGW site/location with its own backing Ceph cluster
- Bucket: where fine-grained sync policy is applied
A multi-zone configuration has one zonegroup with multiple zones, and each zone can accept writes. Multisite then synchronizes changes between them.
What a policy can practically control
At a high level, a replication/sync policy lets you define which buckets are affected, which zones are sources and destinations, whether data movement is bidirectional or one-way, and whether a given rule/group is enabled, allowed, or forbidden.
Ceph’s sync policy model includes groups, data flows, and pipes: data flow describes the allowed direction of movement between zones; pipes define specific source/destination bucket relationships; groups collect those rules and have a status such as enabled/allowed/forbidden.
Why this exists
Default multisite behavior tends toward “mirror the bucket across zones.” Sync policies exist because real deployments often need more nuance, such as DR-only replication, locality-specific buckets, selective replication for compliance or cost, hub-and-spoke or fan-out topologies, and intentionally different bucket contents in different zones.
A simple example
Say you have Zone A in Amsterdam, Zone B in Frankfurt, and Zone C in New York. You might want:
logs/bucket: A → B → Cmedia/bucket: A ↔ B, but not Cbackups/bucket: A → C only
In Ceph terms, that is exactly the kind of bucket-granular behavior that sync policy is meant to express.
Where people get surprised
A few practical caveats:
- RGW bucket replication is not full AWS parity; Ceph documents it as partial.
- Replication is a multisite / cross-zone concept, not “same zone” bucket-to-bucket copying in the AWS sense.
- Disabling an RGW daemon’s sync traffic only stops it from sending replication traffic; it may still receive replication unless removed from replication endpoints.
The short version
In Ceph S3 RGW, a replication policy usually means a multisite sync policy that controls which buckets replicate between which zones, and in what direction, while being separate from both access bucket policies and Ceph’s internal pool replication.
Official references
Use the Ceph multisite guide together with the sync-policy documentation for your release when designing real replication flows.