Kinesis Firehose transformation json output

0

I am using a lambda to perform some transformation on records buffered into a kinesis stream.

This is my lambda (data is ingested in Gzip):

ef lambda_handler(event, context):
    output = []

    for record in event['records']:
        print(record['recordId'])
        data = base64.b64decode(record['data'])
        content = gzip.decompress(data).decode('utf-8')
        payload = json.loads(content)
        message = payload["logEvents"][0]["message"]
        print(message)
        output_record = {
            'recordId': record['recordId'],
            'result': 'Ok',
            'data': base64.b64encode(json.dumps(message).encode('utf-8')).decode('utf-8')
        }
        output.append(output_record)

    print('Successfully processed {} records.'.format(len(event['records'])))

    return {'records': output}

The issue is that Kinesis does not understand that the data sent back is in json and output as escaped strings:

"{\"id\": 123, \"name\" : \"Andrea\"}"

Any suggestion? I need the data back in json format to query that from Athena

profile picture
질문됨 4달 전621회 조회
1개 답변
0

Solved it. The message extracted from:

message = payload["logEvents"][0]["message"]

is actually a string. I did not notice this at first sight. If it contains a json you must load it before dump again.

profile picture
답변함 4달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠