AWS Glue & AWS Glue Connector for Elasticsearch)

Question

Hello,

I am currently trying to set up the [AWS Glue Connector for Elasticsearch](https://aws.amazon.com/marketplace/pp/prodview-v5ygernwn2gb6?sr=0-1&ref_=beagle&applicationId=GlueStudio)

I am having difficulty understanding the proper format for the output schema. There is no good documentation on setting up the output schema that I can find.

Can anyone help me with understand what format the output schema should be in?

Is the schema based on the full response from elasticsearch or just the data returned?

Currently I have the schema set up as the entire response from elasticsearch when you hit the _search endpoint as shown

![Elasticsearch Schema](/media/postImages/original/IMJrqgjKybS8y_0ULc-uVYaA)

However this is throwing long overflow errors as if the format is incorrect.

Any help with this matter would be greatly appreciated.

Answer

Hi,

I understand that you are using the AWS Glue Connector for Elasticsearch. After providing the input schema, you can view the resulting data schema for your data source by choosing the Output schema. The schema displayed on the Output schema tab is used by any child nodes that you add to the job graph. Here you can edit the schemas for a custom transform node. The output schema is going to validate the schema based on your data file.

I could also see you have mentioned that you are facing long overflow errors as if the format is incorrect. I am not sure what exactly the error here is but if its about mismatch between schema then please ensure that the output schema is matching up with the expected data value [from the data file].

If the issue still persist, then please open a support case with AWS providing the errors you are facing along with the Job Run Id and Region - https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-case

Thank you.

AWS Glue & AWS Glue Connector for Elasticsearch)

Relevant content