API Gateway unable to stream back generated PDF due to encoding type
I have posted updates below as I have gone through troubleshooting and bug fixing in regards to this problem.
I have a simple application that I have built that runs perfectly locally. A React front-end that takes some text inputs, and an Express backend that generates a PDF with the information, and streams back a PDF that is generated. The whole application works fine, as does testing a locally running as well as specifically hitting the endpoints using Insomnia running locally.
The front-end is currently deployed in an S3 bucket being served through Cloudfront and is working fine. If I point an API request with Insomnia at my deployed API, the GET requests works fine, returning exactly what is expected, displaying the generated PDF in the content window.
However when I make my POST request, I get some mixed messages. If I hit the POST in Insomnia I get back a 200, but the PDF that is returns is blank even though I verified that it did receive the POST body.
If I use the deployed UI or my local UI pointing at the deployed API, I get a combination of errors.
- Access to fetch at 'API ENDPOINT URL' from origin 'MY UI ORIGIN' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.
- POST 'API ENDPOINT URL' net::ERR_FAILED
- Uncaught (in promise) TypeError: Failed to fetch
The second errors feel like false flags caused by possibly the first error. However I have CORS Enabled in my API Gateway and currently am allowing wildcarded access. And the GET request works fine. I also have enabled 'application/json' and 'application/pdf' in the Binary Media Types in the Gateway settings.
I saw some posts from a few years ago indicating that API Gateway doesn't support streaming back a file but I am unsure if that is correct. If so, that means I need to come up with a different solution for serving up generated files to end users. Or if that has been changed, there is another error in my AWS implementation.
UPDATE: I made some progress. For testing, I actually removed the OPTIONS method from my API Gateway. That actually removed the CORS errors, which was unexpected. The GET request stills functions as it did before and all responses come back as 200. However the PDF is malformed or corrupted. It still works locally, but the PDF coming back from the deployed API does not. I also get the same result from the Test function within API Gateway but can confirm it still works running locally.
UPDATED PROGRESS: After removing the rest of what appeared to be false flag errors, it has come down to API Gateway having challenges with encoding type. After making sure to have binary types being set within the API Gateway, within my code base, and having Accept headers set on the client, the returned document still appears to be encoded improperly. That is where this question is at now, as I continue to work on it.
- Newest
- Most votes
- Most comments
First step would be to make sure it is not a CORS issue. Please take a look at this document from AWS Premium Support on how to debug CORS issues - https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-cors-errors/
Also please take a look at the instructions on this document to ensure you are following all the steps involved in handling binary data correctly in your API Gateway assuming you are using REST APIs - https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings-configure-with-control-service-api.html
If you are using HTTP APIs instead of REST APIs in AWS API Gateway, please follow the instructions here - https://aws.amazon.com/blogs/compute/handling-binary-data-using-amazon-api-gateway-http-apis/
This article is also helpful - https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-binary-data-lambda/
I feel like I should come back and answer this question in case someone in the future (probably even me) comes across it. The underlying issue was problems with the encoding type that was being returned by the API Gateway. All the errors I listed were side errors having to do with OPTIONS requests that weren't needed but were automatically created when I set up CORS (my API and UI are on different subdomains, which I know if I had set it up on the first one, it would have never needed CORS to being with).
Within my API Gateway Settings, I did set Binary Media Types to include application/pdf and I did add headers to my initial API request from the UI with 'Content-Type': 'application/json', 'Accept': 'application/pdf'. The final issue had to do with how I was setting up the serverless deploy. My API was exporting module.exports.handler = serverless(app) but needed an additional configuration options that ended up looking like this:
module.exports.handler = serverless(app, {
binary: ['application/pdf', 'application/json']
});
After deploying that to my Lambda being served by API Gateway, this ended up resolving the issue and I was able to stream a generated PDF to the end-user.
Relevant content
- asked 2 years ago
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
I appreciate your response. I updated the status of my question. So I removed the OPTIONS method from my API Gateway which surprisingly all three of my listed errors, however it still returns what appears to be a blank or corrupted PDF. Though this functionality works fine locally and through a testing suite. I have application/pdf set as an a binary type in the API Gateway settings, and testing it through Insomnia, I set an Accept header with / for testing purposes. Unfortunately, the PDF still returns blank. I get the same result when I use the Test function within API Gateway but can confirm the PDFs generate properly locally. I have read over those articles and it looks like I have tried a lot of what they are suggesting.