Hi All,
We are trying to send the some sample logs to OpenSearch Serverless collection using logstash from EC2 instance, we are getting this below error, can someone please help to advise how to troubleshoot this.
(From same EC2 instance we are able to send some logs via OpenSearch client for same collection endpoint, so i guess we have all the required access iam policies)
[2023-05-24T06:12:42,684][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://myendpoint.my-region-1.aoss.amazonaws.com:443/"}
[2023-05-24T06:12:42,704][INFO ][logstash.outputs.opensearch][main] Cluster version determined (2.0.0) {:version=>2}
[2023-05-24T06:12:42,713][WARN ][logstash.filters.grok ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-05-24T06:12:42,783][ERROR][logstash.outputs.opensearch][main] Unable to retrieve OpenSearch cluster uuid {:message=>"undefined method `[]' for nil:NilClass", :exception=>NoMethodError, :backtrace=>["/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/plugin_mixins/opensearch/common.rb:91:in `discover_cluster_uuid'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch.rb:253:in `finish_register'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch.rb:231:in `block in register'", "/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/plugin_mixins/opensearch/common.rb:83:in `block in after_successful_connection'"]}
[2023-05-24T06:12:42,792][INFO ][logstash.outputs.opensearch][main] Using a default mapping template {:version=>2, :ecs_compatibility=>:disabled}
[2023-05-24T06:12:42,867][ERROR][logstash.outputs.opensearch][main] Failed to install template {:message=>"Got response code '403' contacting OpenSearch at URL 'https://myendpoint.my-region-1.aoss.amazonaws.com:443/_index_template/logstash'", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/home/ec2-user/logstash-8.6.1/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:178:in `perform_request'",
[2023-05-24T06:12:42,949][INFO ][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/home/ec2-user/logstash-8.6.1/logstash-generator.conf"], :thread=>"#<Thread:0x56582b65@/home/ec2-user/logstash-8.6.1/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-05-24T06:12:44,239][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.29}
[2023-05-24T06:12:45,458][ERROR][logstash.outputs.opensearch][main][94d7b43e9ddc82bd66b8069c6178c6e3c5f41ed993c59831bb213b4295ecaf15] Encountered a retryable error (will retry with exponential backoff) {:code=>403, :url=>"https://myendpoint.my-region-1.aoss.amazonaws.com:443/_bulk", :content_length=>87843}
