I want to take advantage of AWS Kinesis Data Firehose with a destination of HTTP endpoint to stream text records. The destination expects for authentication an HTTP header of the form:
Authentication: Bearer: xxxxx-xxxxxx-xxxxxx
No other format is allowed, and I cannot do any processing on the receiving side.
After doing some testing, I couldn't find a way to have a successful setup. It seems there is no way for me to customize the access key header or to add another header that matches my requirement. This is because KDF sends the headers in the following format. I tried to include the key (10th header) as well as adding additional header (4th header) :
"headers": {
"content-length": "1286",
"x-amzn-tls-version": "TLSv1.2",
"x-forwarded-proto": "https",
#this "x-amz-firehose-common-attributes": "{\"commonAttributes\":{\"Authorization\":\"Bearer: xxxxxx-xxxxx-xxxxx-xxxxx-xxxxxxx\"}}",
"x-forwarded-port": "443",
"x-forwarded-for": "13.100.100.100",
"x-amzn-tls-cipher-suite": "ECDHE-RSA-AES128-GCM-SHA256",
"x-amz-firehose-request-id": "xxxxxx-ab4f-4771-a3bf-xxxxxxxx",
"x-amzn-trace-id": "Root=1-xxxx-5f4c2f236809bxxxxxxxx",
#this "x-amz-firehose-access-key": "xxxx-xxxx-xxx-xxxx-xxxxxxxxx",
....
}
Is there a way to make this work?
Thanks
Thanks for your reply. Well, in that case I prefer to move away from Kinesis into a Lambda function which will stream logs into the destination directly. Adding an API GW or a proxy will add to the complexity, cost, and points of failure.