AWS Kinesis Data Firehose: Authentication token using a custom HTTP header (or any workaround)

0

I want to take advantage of AWS Kinesis Data Firehose with a destination of HTTP endpoint to stream text records. The destination expects for authentication an HTTP header of the form:

Authentication: Bearer: xxxxx-xxxxxx-xxxxxx

No other format is allowed, and I cannot do any processing on the receiving side. After doing some testing, I couldn't find a way to have a successful setup. It seems there is no way for me to customize the access key header or to add another header that matches my requirement. This is because KDF sends the headers in the following format. I tried to include the key (10th header) as well as adding additional header (4th header) :

"headers": {
        "content-length": "1286",
        "x-amzn-tls-version": "TLSv1.2",
        "x-forwarded-proto": "https",
#this   "x-amz-firehose-common-attributes": "{\"commonAttributes\":{\"Authorization\":\"Bearer: xxxxxx-xxxxx-xxxxx-xxxxx-xxxxxxx\"}}",
        "x-forwarded-port": "443",
        "x-forwarded-for": "13.100.100.100",
        "x-amzn-tls-cipher-suite": "ECDHE-RSA-AES128-GCM-SHA256",
        "x-amz-firehose-request-id": "xxxxxx-ab4f-4771-a3bf-xxxxxxxx",
        "x-amzn-trace-id": "Root=1-xxxx-5f4c2f236809bxxxxxxxx",
#this   "x-amz-firehose-access-key": "xxxx-xxxx-xxx-xxxx-xxxxxxxxx",
        ....
}

Is there a way to make this work? Thanks

1 Answer
1

Hi,

As you mention and according to AWS documentation, Kinesis Data Firehose can only deliver data to endpoints that follow the Firehose request and response specifications, so your authentication token must travel in the X-Amz-Firehose-Access-Key header.

However, have you tried to modify the request between both systems through a proxy? For example, using a REST API Gateway with a custom mapping template or similar. Obviously this supposes an additional cost and latency.

profile picture
EXPERT
answered a year ago
profile picture
EXPERT
reviewed a year ago
  • Thanks for your reply. Well, in that case I prefer to move away from Kinesis into a Lambda function which will stream logs into the destination directly. Adding an API GW or a proxy will add to the complexity, cost, and points of failure.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions