logstash pipeline not working for OpenSearch serverless

0

Hi All,

We are trying to send the some sample logs to OpenSearch Serverless collection using logstash from EC2 instance, we are getting this below error, can someone please help to advise how to troubleshoot this. (From same EC2 instance we are able to send some logs via OpenSearch client for same collection endpoint, so i guess we have all the required access iam policies)

[2023-05-24T06:12:42,684][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://myendpoint.my-region-1.aoss.amazonaws.com:443/"}
[2023-05-24T06:12:42,704][INFO ][logstash.outputs.opensearch][main] Cluster version determined (2.0.0) {:version=>2}
[2023-05-24T06:12:42,713][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-05-24T06:12:42,783][ERROR][logstash.outputs.opensearch][main] Unable to retrieve OpenSearch cluster uuid {:message=>"undefined method `[]' for nil:NilClass", :exception=>NoMethodError, :backtrace=>["/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/plugin_mixins/opensearch/common.rb:91:in `discover_cluster_uuid'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch.rb:253:in `finish_register'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch.rb:231:in `block in register'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/plugin_mixins/opensearch/common.rb:83:in `block in after_successful_connection'"]}
[2023-05-24T06:12:42,792][INFO ][logstash.outputs.opensearch][main] Using a default mapping template {:version=>2, :ecs_compatibility=>:disabled}
[2023-05-24T06:12:42,867][ERROR][logstash.outputs.opensearch][main] Failed to install template {:message=>"Got response code '403' contacting OpenSearch at URL 'https://myendpoint.my-region-1.aoss.amazonaws.com:443/_index_template/logstash'", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:178:in `perform_request'", 

[2023-05-24T06:12:42,949][INFO ][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/home/ec2-user/logstash-8.6.1/logstash-generator.conf"], :thread=>"#<Thread:0x56582b65@/home/ec2-user/logstash-8.6.1/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-05-24T06:12:44,239][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.29}

[2023-05-24T06:12:45,458][ERROR][logstash.outputs.opensearch][main][94d7b43e9ddc82bd66b8069c6178c6e3c5f41ed993c59831bb213b4295ecaf15] Encountered a retryable error (will retry with exponential backoff) {:code=>403, :url=>"https://myendpoint.my-region-1.aoss.amazonaws.com:443/_bulk", :content_length=>87843}
2 回答
0

Have you tried following this guide?

Hope this tutorial will help

profile pictureAWS
专家
Roi
已回答 1 年前
0

I have tried that guide and have the same problem. After sometime the connection just seems to be dropped.

[logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://lv2g
[logstash.outputs.opensearch][main][37a5786b5056b771860f5f5d8b409ecca37dc54d771140a61d10c588f0e54c02] Marking url as dead. Last error: [LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError] OpenSearch Unreachable: [https://lv2g

This is OpenSearch Serverless.

acruz
已回答 4 天前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则