A lambda of ours (which uses Puppeteer) has started crashing. The errors don't correlate to any code change, but do correlate to a bit of a memory spike. We got a few among them that indicate that there's no space left in the /tmp directory. The RESULT logs from the lambda show that Max Memory Usage is far below Memory Size, but it appears capped at an arbitrary number. It always reports Max Memory Usage of 580 MB, when it has 2048MB available to it.
I increased the amount of Ephemeral Storage available for the lambda, which didn't help. Additionally, I print out the size and contents of the /tmp directory using du whenever the lambda errors out, and the directory itself doesn't report being 100% full (often, it's around 25% full). I've pushed the current version of the code up to the lambda in an attempt to sort of force a refresh, which also didn't help.
Has anyone seen this before, or have an idea of what could solve it? Short of destroying the lambda and recreating it, I'm not sure how to figure out what memory issues are happening.
We do clean the /tmp directory, and as I mentioned it is far from full.