How to resolve connection failures from Glue to Opensearch?

0

Trying to write to my OpenSearch cluster

"An error occurred while calling o265.pyWriteDynamicFrame. Connection error (check network and/or proxy settings)- all nodes failed; tried [[https://---.us-west-2.es.amazonaws.com:443]]"

Seemed to work with less data. I've set everything up according to the instructions.

ElasticsearchConnector7134forAWSGlue30_node1688409952934 = glueContext.write_dynamic_frame.from_options( frame=DropandRename_node1688409839967, connection_type="marketplace.spark", connection_options={ "path": "path-to-my-index", "es.nodes.wan.only": "true", "es.nodes": "https://---.us-west-2.es.amazonaws.com", "es.port": "443", "connectionName": "connection", }, transformation_ctx="ElasticsearchConnector7134forAWSGlue30_node1688409952934", )

1 Answer
0

Notice that error doesn't mean there is a connectivity issue (especially if you say it works with less data), it means it failed writing and that can be for a number of reasons, for instance the server nodes running out of disk space. Check the lines around that error in the log for some further indication on the issue, also you should check the server health status.

profile pictureAWS
EXPERT
answered a year ago
  • I tried enabling logging on my OSS domain, but I'm not seeing any obvious errors there.

    I looked through the job logs in glue and also didn't see anything. Any other suggestions?

  • Check in SparkUI, if it failed at the point when trying to store data, you should see failed tasks with the corresponding error

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions