How to stream a Buffer correctly using Streaming Node.js Lambda

Question

Trying to stream a pdf file from a Lambda using "streaming mode".

I am wrapping the handler in `awslambda.streamifyResponse` and then invoking the Lambda using `new InvokeWithResponseStreamCommand`. I tried wrapping the responseStream with `HttpResponseStream.from` and including `{ headers: { "Content-Type": "application/pdf" } }` as the metadata.

I use Puppeteer to render to a `Buffer` and then use `stream.pipeline(pdfBuffer, responseStream/wrappedResponseStream)` but it seems no matter what combination I use. The hex bytes in the pdf buffer end up converted to decimal and then converted to strings and then converted to binary data when crossing the wire, which greatly increases the amount of data and makes it very impractical to deal with the response data.

For example in my Lambda when printing the first few bytes of the pdfBuffer I see the following:

I added some debugging of the chunks that show up in the response.EventStream that I get when sending the `InvokeWithResponseStreamCommand`. I use a `const decoder = new TextDecoder('utf-8')` and call `console.log(decoder.decode(event.PayloadChunk.Payload)` which for the first chunk shows a string that starts with the following: "378068" this corresponds to stringified version of the decimal numbers of the the three first hex numbers in the pdf buffer which shows that something is taking the original buffer and converting the contents into a string in a very inefficient manner at some point in the Lambda.

Our Lambda is using a Docker image which uses `FROM public.ecr.aws/lambda/nodejs:18 AS base` as the base.

Sam Gronblom · Accepted Answer

Ok, I figured out what the issue was. You cannot pipeline the Buffer directly into the response stream. You have to wrap it in a Readable. I.e. `pipeline(Readable.from(pdfBufffer), responseStream)`. This usage was implicitly shown in the blog post that introduces the feature but there was no mention that you must pipeline buffers in this way. This might just be a node pitfall though.

re:Post Agent · Answer

To correctly stream a Buffer containing a PDF file from a Lambda function using the streaming mode, you need to make a few adjustments to your approach. Here's how you can modify your code to achieve this:

1. Wrap your handler with `awslambda.streamifyResponse` as you're already doing.

2. Set the correct content type for the PDF file:

```javascript
responseStream.setContentType("application/pdf");
```

3. Instead of using `stream.pipeline`, directly write the buffer to the responseStream:

```javascript
const pdfBuffer = await generatePdfBuffer(); // Your existing code to generate the PDF buffer
responseStream.write(pdfBuffer);
responseStream.end();
```

4. Make sure you're not converting the buffer to a string at any point in your code. Work with the raw buffer data throughout.

5. When invoking the Lambda function, use the `InvokeWithResponseStreamCommand` as you're already doing, but ensure you're handling the response correctly on the client side.

6. On the client side, when processing the response, treat the payload as binary data, not as text. Instead of using a TextDecoder, you should concatenate the raw buffer chunks:

```javascript
let pdfBuffer = Buffer.alloc(0);
for await (const event of response.EventStream) {
  if (event.PayloadChunk) {
    pdfBuffer = Buffer.concat([pdfBuffer, event.PayloadChunk.Payload]);
  }
}
```

7. After collecting all chunks, you can write the complete buffer to a file or process it as needed.

The issue you're experiencing with the hex bytes being converted to decimal and then to strings is likely happening because of incorrect handling of the binary data somewhere in your process. By ensuring you're working with raw buffer data throughout and not converting to strings, you should be able to stream the PDF file efficiently.

Remember to handle any errors that might occur during the streaming process, both in the Lambda function and on the client side. Also, make sure your Lambda function has the necessary permissions to stream responses.

By following these steps, you should be able to stream your PDF buffer correctly from your Lambda function without the unwanted conversions that are causing inefficiencies in your current setup.  
**Sources**  
[Writing response streaming-enabled Lambda functions - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/config-rs-write-functions.html)  
[Response streaming for Lambda functions - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/configuration-response-streaming.html)  
[Introducing AWS Lambda response streaming | AWS Compute Blog](https://aws.amazon.com/blogs/compute/introducing-aws-lambda-response-streaming/)

Sam Gronblom · Answer

Ok, I'll continue the discussion with the LLM.

1. Neither the responseStream or HttpResponseStream.from have a .setContentType method
2. 5-7 don't apply since I can confirm that the data is incorrect in the `event.PayloadChunk.Payload`

How to stream a Buffer correctly using Streaming Node.js Lambda

Add your answer

Relevant content