- Newest
- Most votes
- Most comments
The HTTP 401 error you're encountering suggests an authentication issue between your AWS OpenTelemetry (AWS-OTel) Collector and the OpenSearch ingestion pipeline. This problem likely arose after restarting the pipeline, indicating a possible change or expiration in the authentication credentials. Here are some potential causes and steps to resolve the issue:
-
IAM Role Permissions: Ensure that the IAM role associated with your ECS task running the AWS-OTel Collector has the necessary permissions to interact with the OpenSearch ingestion pipeline. Check if the role's permissions have been modified or if any temporary credentials have expired.
-
SigV4 Authentication: Verify that the SigV4 authentication in your OpenTelemetry Collector configuration is correctly set up. Make sure the region and service name in the SigV4 configuration match your OpenSearch ingestion pipeline settings.
-
Pipeline Role: Check the pipeline role configuration in your OpenSearch ingestion pipeline. Ensure it has the correct permissions to write to the OpenSearch Service domain or OpenSearch Serverless collection, as well as permission to read secrets from Secrets Manager if applicable.
-
Secrets Manager: If you're using Secrets Manager for authentication, verify that the secrets haven't expired or been modified.
-
Network Configuration: Confirm that there haven't been any changes to the network configuration that might be preventing the collector from reaching the pipeline endpoint.
-
Pipeline Configuration: Review your pipeline configuration to ensure that the authentication settings haven't been altered during the restart process.
To troubleshoot:
- Double-check all IAM roles and policies associated with both the AWS-OTel Collector and the OpenSearch ingestion pipeline.
- Verify the SigV4 configuration in your OpenTelemetry Collector setup.
- Review the pipeline configuration, particularly the authentication and endpoint settings.
- Check the CloudWatch logs for both the AWS-OTel Collector and the OpenSearch ingestion pipeline for more detailed error messages.
- Consider recreating the pipeline with the same configuration to see if it resolves the issue.
If the problem persists after these checks, you may need to engage AWS support for more in-depth troubleshooting, as there could be an issue with the OpenSearch ingestion pipeline service itself.
Sources
Using an OpenSearch Ingestion pipeline with OpenSearch - Amazon OpenSearch Service
Using an OpenSearch Ingestion pipeline with OpenTelemetry Collector - Amazon OpenSearch Service
answered 2 years ago
Hello,
The Validation failed exception is mostly related to the configuration change however the error you are facing with OpenSearch ingestion pipeline is related to Authentication - "ERROR com.amazon.osis.HttpAuthorization - Authentication status: 401"
If you have reviewed the above provided options to check the credentials and still facing the same issue, please open a support case with the AWS team with the resource ARN and pipeline configuration file.
Hey I am also facing this issue, I am using sigv4auth extension in opentelemetry collector and in between (rarely), I am seeing these errors - error internal/queue_sender.go:51 Exporting failed. Dropping data. {"resource": {"service.instance.id": "", "service.name": "otelcol-contrib", "service.version": "0.130.1"}, "otelcol.component.id": "otlphttp", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://<pipeline_name>.us-east-2.osis.amazonaws.com/v1/traces responded with HTTP Status Code 401", "dropped_items": 5}
answered 8 months ago
Relevant content
asked 2 years ago
asked 3 years ago

There is a Validation failed due to shard size of 50GB error under Configuration change status. could this possibly cause the issue?
Would change in pipeline cause the change in Domain config change?