Kinesis Firehose transformation json output

0

I am using a lambda to perform some transformation on records buffered into a kinesis stream.

This is my lambda (data is ingested in Gzip):

ef lambda_handler(event, context):
    output = []

    for record in event['records']:
        print(record['recordId'])
        data = base64.b64decode(record['data'])
        content = gzip.decompress(data).decode('utf-8')
        payload = json.loads(content)
        message = payload["logEvents"][0]["message"]
        print(message)
        output_record = {
            'recordId': record['recordId'],
            'result': 'Ok',
            'data': base64.b64encode(json.dumps(message).encode('utf-8')).decode('utf-8')
        }
        output.append(output_record)

    print('Successfully processed {} records.'.format(len(event['records'])))

    return {'records': output}

The issue is that Kinesis does not understand that the data sent back is in json and output as escaped strings:

"{\"id\": 123, \"name\" : \"Andrea\"}"

Any suggestion? I need the data back in json format to query that from Athena

profile picture
gefragt vor 4 Monaten621 Aufrufe
1 Antwort
0

Solved it. The message extracted from:

message = payload["logEvents"][0]["message"]

is actually a string. I did not notice this at first sight. If it contains a json you must load it before dump again.

profile picture
beantwortet vor 4 Monaten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen