Cloudwatch to Firehose problem

Question

Hello!,

Im facing some issue while trying to stream data from cloudwatch to opensearch using firehose as middleware, the message that is giving me Opensearch, after running the transformation Lambda is:

> Check your function and make sure the output is in required format. In addition to that, make sure the processed records contain valid result status of Dropped, Ok, or ProcessingFailed

but my lambda is returning correctly the expected structure for the transformation:

```
{'records': [{'recordId': '49636899004605693948429040836377972132019587527722663938000000', 'result': 'Ok', 'data': '...'}]}
```

Actually I'm using python for the transformation Lambda

```
 recordData = {
            'recordId': recordIdD,
            'result': 'Ok',
            'data': base64.b64encode(dataToSend).decode("utf8")
        }
```
Any advice about this issue to make the pipeline work?

Thanks in advance

Answer

I also same problem.

I used 'Process CloudWatch logs sent to Kinesis Firehose' Blue Print.
It problem seems when record[n].data is multi JSON logs.
BluePrint code joins multi JSON logs to one encoded data.

After that, the following error occurs.
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [3], expected one of [create, delete, index, update] but found [event_timestamp]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [3], expected one of [create, delete, index, update] but found [event_timestamp]"},"status":400}

Kinesis can't process multi json data in one record?

Cloudwatch to Firehose problem

関連するコンテンツ