How to get glue to discover an eventbridge payload for use in CDK gluetable deployment?

0

New to glue and athena.

I have a great toy example by an AWS community builder working. But in my real use case, I want to capture all the fields from an eventbridge event 'detail' section and have columns created. This is nested multiple levels. I can't figure out the schema discovery process. Tried to post a text file to S3 and have a glue crawler work on it but no luck.

Thanks in advance.

1개 답변
1
수락된 답변

The crawler is for data, not likely to work well on metadata like that.
You could for instance run an AWS Lambda that reads the json event using pandas (which will infer the schema).

However, my advice is not to deploy tables (at least tables that evolve) using CDK, let the code/crawlers maintain it. What it sounds you are trying to do (coordinate CDK deployments with tables changes is going to be problematic)

profile pictureAWS
전문가
답변함 일 년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠