Cloudwatch to Firehose problem

0

Hello!,

Im facing some issue while trying to stream data from cloudwatch to opensearch using firehose as middleware, the message that is giving me Opensearch, after running the transformation Lambda is:

Check your function and make sure the output is in required format. In addition to that, make sure the processed records contain valid result status of Dropped, Ok, or ProcessingFailed

but my lambda is returning correctly the expected structure for the transformation:

{'records': [{'recordId': '49636899004605693948429040836377972132019587527722663938000000', 'result': 'Ok', 'data': '...'}]}

Actually I'm using python for the transformation Lambda

 recordData = {
            'recordId': recordIdD,
            'result': 'Ok',
            'data': base64.b64encode(dataToSend).decode("utf8")
        }

Any advice about this issue to make the pipeline work?

Thanks in advance

1개 답변
0

I also same problem.

I used 'Process CloudWatch logs sent to Kinesis Firehose' Blue Print. It problem seems when record[n].data is multi JSON logs. BluePrint code joins multi JSON logs to one encoded data.

After that, the following error occurs. {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [3], expected one of [create, delete, index, update] but found [event_timestamp]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [3], expected one of [create, delete, index, update] but found [event_timestamp]"},"status":400}

Kinesis can't process multi json data in one record?

ssyun
답변함 일 년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠