Corrupted data in Lambda function URL streaming response

0

I have a simple lambda function that streams a large response (around 500KB), and I occasionally observe that the response received by the client is corrupted (approx 1 out of 1500 response are corrupted).

Here is the function code:

const MAX_LINES = 30000

export const handler = awslambda.streamifyResponse(async (_event, responseStream, _context) => {
  await new Promise((resolve, _reject) => {
    let i = 1

    responseStream.on('error', (err) => {
      console.error('error:', err)
      _reject(err)
    })

    responseStream.on('close', () => {
      console.log('closed')
    })

    responseStream.on('finish', () => {
      console.log('finished')
      resolve()
    })

    write()

    function write() {
      let ok = true
      do {
        if (i === MAX_LINES) {
          responseStream.end(`This is line ${i}\n`)
        } else {
          ok = responseStream.write(`This is line ${i}\n`)
        }
        i++
      } while (i <= MAX_LINES && ok)

      if (i <= MAX_LINES) {
        console.log('buffer full - waiting for drain')
        responseStream.once('drain', write)
      }
    }
  })
})

The function is configured to use a function URL, with RESPONSE_STREAM, no auth. Memory is set to 1024MB, timeout 30s, unreserved concurrency, everything else as defaults.

The function ARN is arn:aws:lambda:us-west-2:947708160692:function:streaming-test

I used this Node.js script to make requests and detect corrupted responses:

import fs from 'fs'

const expected = fs.readFileSync('expected.txt', 'utf8') // you can download this once from the function URL using curl -o to use as a comparison
let corrupted = 0
let good = 0

const url = 'https://***redacted***' // replace with the function URL

for (;;) {
  const result = await fetch(url)
  const data = await result.text()

  if (data !== expected) {
    corrupted++
    console.log('Corrupted:', corrupted, 'good:', good)
    fs.writeFileSync(`corrupted-content-${corrupted.toString().padStart(4, '0')}.txt`, data)
  } else {
    good++
    process.stdout.write('.')
  }
}

Inspecting the corrupted responses, it looks like chunks are getting transposed, so chunk N is swapped with some chunk N+M. The total amount of data returned is the correct size.

Let me know if you are able to reproduce the issue, or if I can help clarify anything about my set up.

Russ
asked a year ago292 views
1 Answer
0

I can suggest a couple options that come to mind that may or may not work depending on your architecture. It sounds like the corruption is happening when the Lambda receives the data from the client so the issue is occurring on the client-side or during transport.

Retry Logic

  1. Create a unique id to each event.
  2. Update the Lambda function to update the client with the failed ids. This can be done in multiple ways. For example, the Lambda could write the failed ids to another source like a dead letter queue or a DynamoDB table. https://aws.amazon.com/about-aws/whats-new/2016/12/aws-lambda-supports-dead-letter-queues/
  3. Update the client to process the failed ids and re-send those specific events again. Set some maximum number of attempts to avoid processing a broken message endlessly.

Service Change Even if Lambda supports streaming it may not be the best option based on your use-case. AWS Kinesis is a very popular streaming service with a lot of built-in features. https://aws.amazon.com/kinesis/

I hope these suggestions help you. Please let me know if you would like additional suggestions or more details.

EXPERT
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions