logstash pipeline not working for OpenSearch serverless

0

Hi All,

We are trying to send the some sample logs to OpenSearch Serverless collection using logstash from EC2 instance, we are getting this below error, can someone please help to advise how to troubleshoot this. (From same EC2 instance we are able to send some logs via OpenSearch client for same collection endpoint, so i guess we have all the required access iam policies)

[2023-05-24T06:12:42,684][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://myendpoint.my-region-1.aoss.amazonaws.com:443/"}
[2023-05-24T06:12:42,704][INFO ][logstash.outputs.opensearch][main] Cluster version determined (2.0.0) {:version=>2}
[2023-05-24T06:12:42,713][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-05-24T06:12:42,783][ERROR][logstash.outputs.opensearch][main] Unable to retrieve OpenSearch cluster uuid {:message=>"undefined method `[]' for nil:NilClass", :exception=>NoMethodError, :backtrace=>["/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/plugin_mixins/opensearch/common.rb:91:in `discover_cluster_uuid'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch.rb:253:in `finish_register'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch.rb:231:in `block in register'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/plugin_mixins/opensearch/common.rb:83:in `block in after_successful_connection'"]}
[2023-05-24T06:12:42,792][INFO ][logstash.outputs.opensearch][main] Using a default mapping template {:version=>2, :ecs_compatibility=>:disabled}
[2023-05-24T06:12:42,867][ERROR][logstash.outputs.opensearch][main] Failed to install template {:message=>"Got response code '403' contacting OpenSearch at URL 'https://myendpoint.my-region-1.aoss.amazonaws.com:443/_index_template/logstash'", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:178:in `perform_request'", 

[2023-05-24T06:12:42,949][INFO ][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/home/ec2-user/logstash-8.6.1/logstash-generator.conf"], :thread=>"#<Thread:0x56582b65@/home/ec2-user/logstash-8.6.1/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-05-24T06:12:44,239][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.29}

[2023-05-24T06:12:45,458][ERROR][logstash.outputs.opensearch][main][94d7b43e9ddc82bd66b8069c6178c6e3c5f41ed993c59831bb213b4295ecaf15] Encountered a retryable error (will retry with exponential backoff) {:code=>403, :url=>"https://myendpoint.my-region-1.aoss.amazonaws.com:443/_bulk", :content_length=>87843}
2 Respuestas
0

Have you tried following this guide?

Hope this tutorial will help

profile pictureAWS
EXPERTO
Roi
respondido hace un año
0

I have tried that guide and have the same problem. After sometime the connection just seems to be dropped.

[logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://lv2g
[logstash.outputs.opensearch][main][37a5786b5056b771860f5f5d8b409ecca37dc54d771140a61d10c588f0e54c02] Marking url as dead. Last error: [LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError] OpenSearch Unreachable: [https://lv2g

This is OpenSearch Serverless.

acruz
respondido hace 4 días

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas