Skip to content

OpenSearch Pipeline Responds with HTTP 401 Error

0

I have an AWS OpenTelemetry (AWS-OTel) Collector running in ECS, which publishes logs to an OpenSearch ingestion pipeline. This setup was working as expected until I restarted the pipeline.

After stopping the pipeline a few days ago and restarting it today , I began seeing the following error in the collector logs: Permanent error: error exporting items, request to https://<pipeline>/v1/logs responded with HTTP Status Code 401.

Additionally, the OpenSearch pipeline logs show the following error: [armeria-common-worker-epoll-3-3] ERROR com.amazon.osis.HttpAuthorization - Authentication status: 401.

I am unable to determine whether this error originates from the OpenSearch ingestion pipeline itself or from the OpenSearch domain. Could you help investigate the root cause of this authentication failure and suggest potential steps for resolution?

asked 2 years ago985 views

3 Answers
0

The HTTP 401 error you're encountering suggests an authentication issue between your AWS OpenTelemetry (AWS-OTel) Collector and the OpenSearch ingestion pipeline. This problem likely arose after restarting the pipeline, indicating a possible change or expiration in the authentication credentials. Here are some potential causes and steps to resolve the issue:

  1. IAM Role Permissions: Ensure that the IAM role associated with your ECS task running the AWS-OTel Collector has the necessary permissions to interact with the OpenSearch ingestion pipeline. Check if the role's permissions have been modified or if any temporary credentials have expired.

  2. SigV4 Authentication: Verify that the SigV4 authentication in your OpenTelemetry Collector configuration is correctly set up. Make sure the region and service name in the SigV4 configuration match your OpenSearch ingestion pipeline settings.

  3. Pipeline Role: Check the pipeline role configuration in your OpenSearch ingestion pipeline. Ensure it has the correct permissions to write to the OpenSearch Service domain or OpenSearch Serverless collection, as well as permission to read secrets from Secrets Manager if applicable.

  4. Secrets Manager: If you're using Secrets Manager for authentication, verify that the secrets haven't expired or been modified.

  5. Network Configuration: Confirm that there haven't been any changes to the network configuration that might be preventing the collector from reaching the pipeline endpoint.

  6. Pipeline Configuration: Review your pipeline configuration to ensure that the authentication settings haven't been altered during the restart process.

To troubleshoot:

  1. Double-check all IAM roles and policies associated with both the AWS-OTel Collector and the OpenSearch ingestion pipeline.
  2. Verify the SigV4 configuration in your OpenTelemetry Collector setup.
  3. Review the pipeline configuration, particularly the authentication and endpoint settings.
  4. Check the CloudWatch logs for both the AWS-OTel Collector and the OpenSearch ingestion pipeline for more detailed error messages.
  5. Consider recreating the pipeline with the same configuration to see if it resolves the issue.

If the problem persists after these checks, you may need to engage AWS support for more in-depth troubleshooting, as there could be an issue with the OpenSearch ingestion pipeline service itself.
Sources
Using an OpenSearch Ingestion pipeline with OpenSearch - Amazon OpenSearch Service
Using an OpenSearch Ingestion pipeline with OpenTelemetry Collector - Amazon OpenSearch Service

answered 2 years ago

  • There is a Validation failed due to shard size of 50GB error under Configuration change status. could this possibly cause the issue?

    Would change in pipeline cause the change in Domain config change?

0

Hello,

The Validation failed exception is mostly related to the configuration change however the error you are facing with OpenSearch ingestion pipeline is related to Authentication - "ERROR com.amazon.osis.HttpAuthorization - Authentication status: 401"

If you have reviewed the above provided options to check the credentials and still facing the same issue, please open a support case with the AWS team with the resource ARN and pipeline configuration file.

AWS
SUPPORT ENGINEER

answered 2 years ago

0

Hey I am also facing this issue, I am using sigv4auth extension in opentelemetry collector and in between (rarely), I am seeing these errors - error internal/queue_sender.go:51 Exporting failed. Dropping data. {"resource": {"service.instance.id": "", "service.name": "otelcol-contrib", "service.version": "0.130.1"}, "otelcol.component.id": "otlphttp", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://<pipeline_name>.us-east-2.osis.amazonaws.com/v1/traces responded with HTTP Status Code 401", "dropped_items": 5}

answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.