Skip to content
ScaleStore
Menu
← Blog
Technical field note
9 min read

Putting Ceph RGW on the Internet: An Operator’s Checklist

A public S3 endpoint deserves the same discipline as any other internet-facing infrastructure: narrow exposure, strong identity, bounded abuse, useful logs and tested recovery.

Operator note: Ceph RGW features and supported policy/API behavior vary by release. Check the documentation for your deployed version before turning an example into production policy.

Here is a full practical checklist for exposing Ceph RGW to the internet.

A. Platform and version

  • Run a currently supported Ceph release
  • Apply the latest stable point release for your chosen branch
  • Prefer the beast frontend
  • Do not use deprecated civetweb for new deployments
  • Document your rollback/upgrade path before go-live

Ceph’s active maintained release lines currently include Tentacle, Squid, and Reef, and Ceph documents that civetweb is deprecated while beast is the modern frontend.

B. Network design

  • Expose only the RGW public endpoint to the internet
  • Keep MON, MGR, OSD, dashboard, and cluster/backend networks private
  • Put RGW behind a dedicated reverse proxy / load balancer
  • Restrict inbound internet access to TCP 443 only
  • Restrict proxy-to-RGW traffic with host firewall/security groups
  • Separate public ingress from storage/backend traffic
  • Use dedicated DNS names for RGW
  • Decide whether you need path-style or virtual-host-style bucket addressing and configure DNS accordingly

C. TLS and certificates

  • Enforce HTTPS only
  • Redirect HTTP to HTTPS
  • Use trusted certificates with automated renewal
  • Disable weak TLS versions/ciphers at the edge
  • Use HSTS if browser access is expected and safe for your use case
  • Confirm proxy/WAF does not break S3 signing behavior
  • Test presigned URLs through the edge path
  • Test multipart uploads over TLS

Ceph supports RGW frontend configuration through cephadm and frontend args, but public deployments should still terminate TLS cleanly at the edge or on RGW itself.

D. Edge proxy / load balancer

  • Place Nginx / HAProxy / Envoy / Traefik / cloud LB in front of RGW
  • Enable access logs on the edge
  • Set max header size
  • Set max request body size appropriate for your upload pattern
  • Limit idle keepalive abuse
  • Limit concurrent connections per source where practical
  • Allow only required HTTP methods: GET, PUT, POST, DELETE, HEAD
  • Deny unexpected methods
  • Preserve the correct Host header if your signing model depends on it
  • Avoid rewriting signed headers, query strings, and canonical URI components

E. WAF / abuse filtering

  • Put a WAF or edge filtering layer in front of RGW
  • Start in monitor/log mode
  • Promote only validated rules to block/challenge
  • Add bot mitigation / IP reputation filtering
  • Add geo/ASN rules only if your client base is narrow
  • Rate limit request floods at the edge
  • Block obvious exploit/probe paths that should never be valid
  • Confirm WAF does not alter S3 signed requests
  • Keep exception rules for valid large uploads and presigned URLs
  • If using CrowdSec, make sure the AppSec component and a WAF-capable bouncer are actually in the request path

OWASP defines a WAF as an HTTP application firewall that applies rules to HTTP traffic, and current NIST API-protection guidance recommends perimeter protections such as WAFs, bot detection, and DDoS mitigation in front of APIs.

F. DDoS and rate controls

  • Enable RGW-native rate limiting
  • Set per-user read/write/list/delete limits
  • Set per-bucket rate limits where needed
  • Consider global rate limits for shared/public workloads
  • Add edge-side rate limits as a first filter
  • Define separate thresholds for auth failures, object download floods, bucket listing abuse, and multipart upload abuse
  • Test with burst traffic before production

Ceph documents RGW rate limiting by user and bucket, including ops and byte limits.

G. Identity and credential design

  • Create one identity per application/service
  • Do not share access keys across teams/apps
  • Use least-privilege policies only
  • Disable and remove unused users/keys
  • Rotate keys regularly
  • Prefer STS temporary credentials where possible
  • Use account-based IAM instead of broad flat admin users
  • Document key ownership and rotation owner
  • Separate human admin credentials from application credentials
  • Prevent internet-facing apps from using admin-capable credentials

Ceph RGW supports account-based IAM and STS, and account users begin without permissions until policies are attached.

H. Authorization and public access

  • Default all buckets to private
  • Review ACL usage; avoid legacy ACL sprawl
  • Review bucket policies for every public bucket
  • Explicitly inventory every anonymous/public object path
  • Use short-lived signed URLs instead of public buckets where possible
  • Review CORS settings and allow only specific origins/methods/headers
  • Do not allow wildcard CORS unless truly necessary
  • Revalidate bucket policies after each app change

Ceph documents that RGW supports authenticated and unauthenticated requests, treating unauthenticated access as anonymous, and supports bucket policies and signed URLs.

I. Storage quotas and consumption control

  • Set user quotas
  • Set bucket quotas
  • Monitor pool fullness and capacity thresholds
  • Alert before quotas and pools are near full
  • Decide which workloads may exceed quotas and how exceptions are handled
  • Confirm large clients fail predictably when quota is reached

Ceph’s RGW admin tooling and health checks document quotas and pool-full behavior.

J. Encryption

  • Enforce TLS in transit
  • Enable server-side encryption for sensitive buckets
  • Prefer external KMS for higher assurance environments
  • Decide on SSE mode per tenant/workload
  • Verify encrypted objects replicate correctly if multisite is used
  • Test download/read paths for encrypted multipart objects

Ceph RGW supports server-side encryption options for stored objects.

K. Logging and observability

  • Enable edge proxy access logs
  • Enable RGW usage logging if needed
  • Enable bucket logging for sensitive or regulated buckets
  • Send logs to centralized storage/SIEM
  • Build dashboards for request rate, 4xx/5xx rate, auth failures, top source IPs, egress bandwidth, bucket listing volume, and upload error rate
  • Create alerts for traffic spikes, repeated auth failures, unusual egress, sudden increases in anonymous requests, latency spikes, and backend pool pressure

Ceph documents bucket logging, usage logging, and monitoring via Prometheus/Alertmanager/Grafana.

L. Admin and management exposure

  • Keep Admin Ops off the public internet
  • Keep OIDC/IAM admin workflows off the public edge
  • Restrict admin actions to internal networks or VPN
  • Require MFA for admin access where possible
  • Separate admin DNS names/endpoints from public client endpoints
  • Audit all admin caps and remove anything unnecessary
  • Verify no app credential has admin caps

Ceph documents privileged admin-style APIs and admin caps for some management features; those should not be exposed through the same public path as tenant traffic.

M. Tenant isolation and multi-tenancy

  • Decide whether you need RGW multi-tenancy
  • Isolate tenants/accounts intentionally
  • Avoid naming collisions and policy bleed between customers
  • Review IAM boundaries and tenant-level controls
  • Document per-tenant logging, quotas, and encryption defaults

Ceph supports RGW multi-tenancy and account isolation features.

N. Backup, recovery, and resilience

  • Define RPO/RTO for object workloads
  • Test object restore procedures
  • Test accidental deletion recovery
  • Decide whether versioning is required
  • Consider object lock / retention for critical buckets
  • If multisite is used, test failover and resync
  • Test behavior of encrypted objects under replication

Ceph’s newer RGW releases include support related to object lock and multisite/encryption fixes, so recovery behavior should be tested explicitly.

O. Secure defaults for applications

  • Use least privilege per app
  • Do not embed root/admin credentials in code or CI variables
  • Prefer temporary credentials
  • Set sensible retry/backoff in clients
  • Avoid unlimited parallel multipart uploads
  • Validate application handling of 403, 429, 5xx, and quota errors
  • Log request IDs for troubleshooting

P. Pre-production validation

  • Test normal uploads/downloads through the full internet path
  • Test presigned URLs
  • Test virtual-hosted bucket names
  • Test very large uploads
  • Test multipart resume/abort paths
  • Test object listing at scale
  • Test anonymous access is denied where expected
  • Test public bucket behavior where intentionally enabled
  • Test edge rate limits and RGW-native rate limits
  • Test WAF false positives on real clients
  • Test key rotation without downtime
  • Test failure of one RGW node
  • Test certificate renewal
  • Run external scanning against the public endpoint

Q. Operational hygiene

  • Maintain a written owner for certificates, DNS, edge proxy, WAF/CrowdSec, Ceph upgrades, IAM/policy review, and logging/alerting
  • Review public exposure quarterly
  • Review policies and keys quarterly
  • Patch edge and RGW regularly
  • Re-test after every major change
  • Keep a runbook for incident response and abuse handling

Minimum go-live standard

I would personally require all of these before opening to the internet:

  • Supported Ceph release
  • Beast frontend
  • Reverse proxy/LB
  • HTTPS only
  • WAF/CrowdSec in front
  • RGW quotas
  • RGW rate limits
  • Least-privilege IAM
  • Private-by-default buckets
  • Logging and alerting
  • No public admin exposure
  • Presigned URL and multipart testing completed

Strongly recommended reference architecture

Internet → CDN/WAF/Proxy → RGW nodes → private Ceph cluster network

With edge TLS, edge logs, edge rate limits, a CrowdSec or other WAF layer, RGW native quotas/rate limits, IAM/STS, and SSE for sensitive buckets.

The failures I would worry about most

  • Exposing Ceph internals instead of only RGW
  • Using old Ceph releases
  • Shared long-lived keys
  • Public buckets by accident
  • No quotas/rate limits
  • WAF/proxy rewriting signed S3 requests
  • No logging
  • Admin caps on public-facing app users
  • No restore/failover testing

Keep the checklist tied to your release

Ceph RGW changes over time. Before go-live, compare your design with the current RGW S3 compatibility documentation and the documentation for the exact Ceph release in production.