logstash pipeline not working for OpenSearch serverless

0

Hi All,

We are trying to send the some sample logs to OpenSearch Serverless collection using logstash from EC2 instance, we are getting this below error, can someone please help to advise how to troubleshoot this. (From same EC2 instance we are able to send some logs via OpenSearch client for same collection endpoint, so i guess we have all the required access iam policies)

[2023-05-24T06:12:42,684][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://myendpoint.my-region-1.aoss.amazonaws.com:443/"}
[2023-05-24T06:12:42,704][INFO ][logstash.outputs.opensearch][main] Cluster version determined (2.0.0) {:version=>2}
[2023-05-24T06:12:42,713][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-05-24T06:12:42,783][ERROR][logstash.outputs.opensearch][main] Unable to retrieve OpenSearch cluster uuid {:message=>"undefined method `[]' for nil:NilClass", :exception=>NoMethodError, :backtrace=>["/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/plugin_mixins/opensearch/common.rb:91:in `discover_cluster_uuid'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch.rb:253:in `finish_register'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch.rb:231:in `block in register'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/plugin_mixins/opensearch/common.rb:83:in `block in after_successful_connection'"]}
[2023-05-24T06:12:42,792][INFO ][logstash.outputs.opensearch][main] Using a default mapping template {:version=>2, :ecs_compatibility=>:disabled}
[2023-05-24T06:12:42,867][ERROR][logstash.outputs.opensearch][main] Failed to install template {:message=>"Got response code '403' contacting OpenSearch at URL 'https://myendpoint.my-region-1.aoss.amazonaws.com:443/_index_template/logstash'", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:178:in `perform_request'", 

[2023-05-24T06:12:42,949][INFO ][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/home/ec2-user/logstash-8.6.1/logstash-generator.conf"], :thread=>"#<Thread:0x56582b65@/home/ec2-user/logstash-8.6.1/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-05-24T06:12:44,239][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.29}

[2023-05-24T06:12:45,458][ERROR][logstash.outputs.opensearch][main][94d7b43e9ddc82bd66b8069c6178c6e3c5f41ed993c59831bb213b4295ecaf15] Encountered a retryable error (will retry with exponential backoff) {:code=>403, :url=>"https://myendpoint.my-region-1.aoss.amazonaws.com:443/_bulk", :content_length=>87843}
2回答
0

Have you tried following this guide?

Hope this tutorial will help

profile pictureAWS
エキスパート
Roi
回答済み 1年前
0

I have tried that guide and have the same problem. After sometime the connection just seems to be dropped.

[logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://lv2g
[logstash.outputs.opensearch][main][37a5786b5056b771860f5f5d8b409ecca37dc54d771140a61d10c588f0e54c02] Marking url as dead. Last error: [LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError] OpenSearch Unreachable: [https://lv2g

This is OpenSearch Serverless.

acruz
回答済み 4日前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン