By using AWS re:Post, you agree to the Terms of Use

Unanswered Questions tagged with Extract Transform & Load Data

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

How to reference a glue job python argument in CodeGenConfigurationNodes of create_job() function in boto3 Glue Client.

I would like to reference a python parameter of a glue job within the CodeGenConfigurationNodes of create_job() function in boto3 Glue Client. For instance: I have an argument `--s3_location` which has to be referred within S3CsvSource Node of my ETL job as given below ``` s3_source_node1 = glueContext.create_dynamic_frame.from_options( format_options={ "quoteChar": '"', "withHeader": True, "separator": ",", "optimizePerformance": False, }, connection_type="s3", format="csv", connection_options={"paths": [args["s3_location"]]}, transformation_ctx="s3_source_node1", ) ``` This has to be done via `create_job()` function available as part of GlueClient in boto3. However while defining `CodeGenConfigurationNodes` within `create_job()`, I was not able to mention `[args["s3_location"]` in `path` property of `S3CsvSource` node property. Current CodeGenConfigurationNodes for S3CsvSource is as below: ``` CodeGenConfigurationNodes = { 'node-1':{ 'S3CsvSource': { 'Name': 's3_source', 'Paths': [ 's3://my_bucket/sample_input.csv', ], 'Separator': 'comma', 'QuoteChar': 'quote', 'WithHeader': True, 'WriteHeader': True, }, }] ``` My expected result is something like ``` CodeGenConfigurationNodes = { 'node-1':{ 'S3CsvSource': { 'Name': 's3_source', 'Paths': [ args["s3_location"], ], 'Separator': 'comma', 'QuoteChar': 'quote', 'WithHeader': True, 'WriteHeader': True, }, }] ``` Where `args["s3_location"]` refers to glue job parameter `--s3_location`.
0
answers
0
votes
26
views
asked 25 days ago

HIVE_PARTITION_SCHEMA_MISMATCH

Error: HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced. The column 'isfallbackkey' in table 'analytics.events' is declared as type 'boolean', but partition 'dt=2022-08-22/appid=jovo-game-starter' declared column 'translationkey' as type 'string'. Data was passed as JSON in this format: ``` {"type":"device_capabilities","supportScreen":true,"supportAudio":true,"supportLongformAudio":false,"supportVideo":false,"eventId":"668c9479-9eee-4025-8b9a-1323db06b21f","appId":"jovo-game-starter","eventDate":"2022-08-22T17:26:36.376Z","timestamp":1661189196,"locale":"en","timeZone":"America/Phoenix","userId":"a8ad82ba-bfac-4f93-a46d-aae37e842a7b","sessionId":"139163c0-fcf2-4bcc-9ece-a8e6ab5c322e"} {"type":"session_start","userId":"a8ad82ba-bfac-4f93-a46d-aae37e842a7b","sessionId":"139163c0-fcf2-4bcc-9ece-a8e6ab5c322e","eventId":"411e3abf-07fc-453c-9edd-a0a84f29b75f","appId":"jovo-game-starter","eventDate":"2022-08-22T17:26:36.383Z","timestamp":1661189196,"locale":"en","timeZone":"America/Phoenix"} {"type":"intent","userId":"a8ad82ba-bfac-4f93-a46d-aae37e842a7b","sessionId":"139163c0-fcf2-4bcc-9ece-a8e6ab5c322e","intent":"LAUNCH","eventId":"09287f39-e487-474b-bafc-c0c1b9f59959","appId":"jovo-game-starter","eventDate":"2022-08-22T17:26:36.387Z","timestamp":1661189196,"locale":"en","timeZone":"America/Phoenix"} {"type":"translation","translationKey":"start","isFallbackKey":false,"translationLanguage":"en","translationPlatform":"core","eventId":"15b87be7-5349-4a9e-b950-76bd76b63972","appId":"jovo-game-starter","eventDate":"2022-08-22T17:26:37.889Z","timestamp":1661189198,"locale":"en","timeZone":"America/Phoenix","userId":"a8ad82ba-bfac-4f93-a46d-aae37e842a7b","sessionId":"139163c0-fcf2-4bcc-9ece-a8e6ab5c322e"} ``` Using Kinesis Firehose dynamic partitioning: events/dt=!{partitionKeyFromQuery:dt}/appid=!{partitionKeyFromQuery:appid}/
0
answers
0
votes
37
views
asked a month ago