Prometheus scrapping

0

while scrapping and exporting the metrics awsprometheusremotewrite exporter throwing an error like "error exporterhelper/queued_retry.go:183 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "name": "awsprometheusremotewrite", "error": "Permanent error: remote write returned HTTP status 400 Bad Request; err = <nil>: timestamp too old: 1714982876646 metric: "current_active_requests"\n", "dropped_items": 8907}" but for in my development env I have installed Prometheus and configured Prometheus with storage block with out of order time window : 1d, is there is any equivalent configuration that I have to make in aws side or in my adot-collector.yml configuration side?

Akash
asked a month ago73 views
1 Answer
0
Accepted Answer

Based on the AMP Troubleshooting page, here: https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-troubleshooting.html#AMP-troubleshoot-duplicate-ordering and some other pages I was able to find online from users who have attempted to use this service (https://dev.to/authress/aws-advanced-serverless-prometheus-in-action-j1h), it appears that Amazon Managed Service for Prometheus does not yet support Out-of-order timestamps. So there's no way to configure this option when using AWS-managed Prometheus.

Checking on this official Prometheus page linked from the AWS docs: https://promlabs.com/blog/2022/12/15/understanding-duplicate-samples-and-out-of-order-timestamp-errors-in-prometheus/ and searches of the Prometheus issue tracker, it seems that the project stills considers the feature "experimental" and it may have some outstanding bugs (like https://github.com/prometheus/prometheus/issues/11496) which may be why AWS has not chosen to support it.

Instead, you'll have to fix the root cause of the timestamps at their source, by either overwriting the timestamps or fixing the misconfigurations that led to incorrect timestamps in the first place.

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions