Use Glue schema registry when reading from Kinesis

0

I want to store the schema for Avro formatted messages in Glue schema registry, and I want to use this schema when reading records from Kinesis data stream. Currently, for reading records from the stream, I'm using something like: avro_schemas = { "record1": """ { "type": "record", "name": "record1", "fields": [ {"name": "intField", "type": "int"}, {"name": "strField", "type": "string"} ] } """ }

dataframe = glueContext.create_data_frame.from_options( connection_type="kinesis", connection_options={ "typeOfData": "kinesis", "streamARN": <stream_arn>, "startingPosition": "latest", "classification": "avro", "inferSchema": "false", "avroSchema": avro_schema }, transformation_ctx=f"kinesis_data_frame" )

How can I read the schema from the registry and use it to create the data frame?

YK
질문됨 5달 전664회 조회
1개 답변
0
수락된 답변

You can use boto3 to get the schema: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue/client/get_schema_version.html
Normally you don't have to do that, you create a table based on the schema and then you use it in the streaming job.
Check this: https://docs.aws.amazon.com/glue/latest/dg/add-job-streaming.html#create-table-streaming

profile pictureAWS
전문가
답변함 5달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인