Skip to content

Enabling ADOT addon on multiple EKS clusters generates "out of order sample" error

0

I have multiple EKS clusters and I'm trying to enable the ADOT EKS addon to write metrics to a single Amazon Managed Prometheus instance.

After enabling the addon on the second cluster, the opentelemetry collector generates the following error:

Exporting failed. Dropping data.    {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Permanent error: remote write returned HTTP status 400 Bad Request; err = %!w(<nil>): maxFailure (quorum) on a given error family, rpc error: code = Code(400)  ... err: out of order sample.

I'm pretty sure the reason for this is the two collectors are generating conflicting samples and Prometheus can't differentiate them, and so it's rejecting samples from the second collector that don't match the expected ordering from the first.

This is normally fixed in other Prometheus collectors by applying an external_labels identifier to signify the cluster name, and indeed this is advised in this documentation page for the ADOT collector.

However, even though the OTEL prometheusremotewriteexporter supports the externals_labels attribute, it appears that the same configuration isn't exposed for the ADOT EKS addon, and only these specific configuration values are supported. Attempting to add that value into the configuration of the addon anyway, e.g.:

{
  "collector": {
    "prometheusMetrics": {
      "exporters": {
        "prometheusremotewrite": {
          "endpoint": "${PROMETHEUS_REMOTE_WRITE_ENDPOINT}",
          "external_labels": {
            "cluster": "${CLUSTER_NAME}",
            "__replica__": "primary"
          }
        }
      },
...

fails validation with the following error:

Error: updating EKS Add-On ... operation error EKS: UpdateAddon, https response error StatusCode: 400, RequestID: ..., InvalidParameterException: ConfigurationValue provided in request is not supported: Json schema validation failed with error: [$.collector.prometheusMetrics.exporters.prometheusremotewrite.external_labels: is not defined in the schema and the schema does not allow additional properties]

Is there a way to apply external_labels values to an ADOT collector deployed via the EKS addon, or should I be architecting this in a different manner?

1 Answer
0

It is expected that in a multi-cluster environment you need to add a label to uniquely identify each cluster

AWS
answered a year ago
  • OK, but as I said above, I can't find a mechanism to add that label within the configuration schema of the ADOT EKS addon - how should it be done?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.