Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Illegal instruction in AWS Lambda

1

Hello,

We have started experiencing problems with our AWS Lambda function. Our machine learning inference code, a Dockerized Python application utilizing various ML libraries (numpy, scipy, onnxruntime, etc.), runs on AWS Lambda.

The code was running fine, but then suddenly, without any new deployment or configuration changes, the Lambda handler began crashing with the following error:

Error: Runtime exited with error: signal: illegal instruction Runtime.ExitError

Although our application is correctly restarted, the runtime that executed the failing request is corrupted, causing subsequent requests to this container to fail randomly.

We've attempted to replicate the problem, but it doesn't appear to be strictly related to the input data. Running the same code with the same configuration on the same input completes successfully.

We first experienced these issues in the us-east-1 region on 2024-05-31T16:01:32.320Z and then in the eu-west-1 region on 2024-06-03T15:53:50.707Z.

Has anyone else encountered similar problems, or is there any AWS-related change that might explain this behavior?

  • Zipora
    2 years ago

    we have the same exact issue with very similar lambda dependencies and usage (machine learning inference code, a Dockerized Python application utilizing various ML libraries (numpy, scipy, etc.), runs on AWS Lambda.) and it started to fail also 2 on 2024-05-31...

Topics
ServerlessDevOpsMachine Learning & AICompute
Tags
ServerlessDevOpsMachine Learning & AIAWS Lambda
Language
English
asked 2 years ago841 views
3 Answers
0

It could be any number of things, but it sounds like your function might simply be running out of memory. That also wouldn't exclusively be related to the input it's given or require anything to have changed significantly at/by AWS for the amount of memory available for a runtime environment to vary enough to trigger that situation.

You could try doubling the memory for your function and testing if it makes a difference. Support might also suggest trying that.

EXPERT
answered 2 years ago
  • Sfinx
    2 years ago

    We are currently at max 10GB allowed by lambda, plus from the report log it does not seems to be problem with memory: REPORT RequestId: 268d45e4-30d7-4bc1-b14d-bf9ccd57b0cd Duration: 62508.76 ms Billed Duration: 62509 ms Memory Size: 10240 MB Max Memory Used: 4897 MB

0

I would recommend that you open a support case for that level of troubleshooting. No one on the re:Post forum can access backend information regarding changes to the service.

AWS
EXPERT
answered 2 years ago
0

This can happen if there are binary packages included in your Lambda deployment which were not created on a platform that is the same as the underlying OS that Lambda uses. Make sure that your packages (especially those with binary dependencies) are created on Amazon Linux.

AWS
EXPERT
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content