OpenSearch Ingestion Pipeline for S3 logs stops working after an hour

0

Hello,

I have created an OpenSearch Ingestion Pipeline for getting ALB Logs from S3 bucket, and pushing them to OpenSearch domain. The S3 source is setup with SQS queue.

When stopping/starting the pipeline it works fine and logs appear in OpenSearch domain. However after an hour or so ingestion stops. Looking at pipeline logs in CloudWatch I can see multiple errors from S3 like: Error processing from S3: Unable to execute HTTP request: Timeout waiting for connection from pool. Retrying with exponential backoff.

If I stop/start the pipeline it will again work for sometime and then throw these errors again. How can I solve this issue? Thanks

No hay respuestas

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas