“Why did this request fail?” sounds like a logging question until you try to answer it for somebody else’s S3 request.
The customer sees an error from a client or application. The support engineer sees an account and a rough timestamp. Somewhere underneath are a dashboard, control-plane services, a provider-side bridge, Ceph RGW and whatever network or policy layer sits in front of it.
Giving support raw access to all of those systems is not a solution. It is a security problem with a search box.
The useful unit is a story, not a log line
A single RGW log entry can tell you a lot, but support usually needs a sequence:
- Which customer and provider were involved?
- What operation were they trying to perform?
- Which credential or temporary session was in play—without revealing the secret?
- Did ScaleStore issue a control action?
- Did the Provider Bridge execute it?
- What did RGW or the edge return?
- Was a quota, policy or availability condition responsible?
That sequence is the difference between observability data and a support workflow.
Correlation IDs earn their keep here
Correlation IDs can feel like plumbing when everything is healthy. During an incident they are one of the cheapest ways to avoid guessing.
If a user action starts in a web interface and results in a provider-side operation, the identifier should survive that journey. The operator can then move from the customer-visible event to the control action and provider-side result without searching by timestamps and hoping clocks line up.
Not every component will use the same internal request ID, especially once RGW and proxies are involved. That is fine. The platform can record the relationships between identifiers rather than pretending one magic ID exists everywhere.
“More detail” can make support less safe
S3 workflows contain sensitive material: access keys, session tokens, signed request details, object names, headers and sometimes customer data in diagnostic payloads.
A support interface should be designed around what an operator needs to know, not around everything the backend happens to know.
Useful examples are: credential fingerprint rather than secret value; bounded resource identifier rather than a full payload; error class and provider-side status rather than a raw environment dump.
The same rule applies to logs. Redaction after the fact is much harder than avoiding sensitive logging in the first place.
Failures cross product boundaries
Consider a failed upload. The cause might be:
- an expired temporary credential;
- a bucket policy denying the action;
- a quota that has actually been reached;
- an edge or TLS problem;
- RGW being unavailable;
- an earlier provisioning action that never completed.
Those causes belong to different subsystems, but the customer does not care. They asked one question.
This is why we think diagnostics are part of the product layer rather than an operator-only afterthought. The platform is already the place that understands customer context, entitlement and provider integration. It is in the best position to connect the pieces.
Support should degrade gracefully
There will always be cases where the answer is “we need to go deeper.” A safe support workflow does not have to replace Ceph logs, packet captures or expert debugging.
Its job is to make escalation deliberate. Give the first-line operator enough evidence to classify the problem, preserve the identifiers an engineer will need, and avoid dangerous copy-paste troubleshooting.
The standard we want
When a customer reports a failure, an operator should be able to get to a credible explanation without administrator credentials and without reading five unrelated dashboards.
Sometimes the answer will still be “RGW returned an unexpected error and this needs engineering.” That is acceptable. The important part is knowing how we got there, what was ruled out, and which evidence follows the escalation.
That is much closer to a support product than “here are the logs.”