tenant isolation in AWS XRay backend

Question

I require support for preserving traces from different teams in Xray with the following attributes. The XRay backend used to store all traces will be provisioned in a single centralized account. Note that applications are integrated with OpenTelemetry libraries and will be submitting traces to XRay using ADOT collector.

* maintain separate keys, one per team, for encryption of trace data at rest 
* separate IAM roles to provision access of keys and trace data such a team cannot access traces from other teams.
* maintain separate configurations for sampling rates, rate limits, quotas, etc.

A feasible option for implementing these requirements seems to be using tracing groups. Groups can be created using filter expressions based on certain pre-defined attributes in a trace such as service prefix, unique per team. However, XRay lacks configuring multiple keys for encrypting traces and IAM policies to control read access to traces and service graphs don't support resource level permissions or condition filters. There is also a limit of 25 groups per account. Due to these limitations, we cannot use groups.

Are there alternate approaches for implementing my aforementioned requirements?

Answer

Good to receive this proposal. Could you clarify the requirement, make sure I understand correctly and see if there is workaround.

The trace data to be separated is from a single application or different applications?
* If from different applications, the solution is by setup multiple otel collectors with different IAM role/credentials. So trace data can be emitted to different account and use different sampling rules.
* If from a single application, though I think mostly single application does not make sense, can use [filterprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor#filter-spans-from-traces) to dispatch trace data to different awsxray exporters. The sampling rules have to be in one account because collector supports just one [awsproxy extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/awsproxy#aws-proxy).

```
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filterprocessor]
      exporters: [awsxray]
    traces/2:
      receivers: [otlp]
      processors: [filterprocessor/2]
      exporters: [awsxray/2]
```

tenant isolation in AWS XRay backend

Relevant content