What could be the cause of randomly appearing SnapStartTimeoutException
0
We have a lambda function invoked via api-gateway. Occasionally we see SnapStartTimeoutExceptions in the api-gateway logs and the client gets an http 500 error. We do not see any errors in the logs of the lambda function. What are the potential causes of a SnapStartTimeoutException?
Please see the end of the logs below.
(b25eda70-670f-4a34-ad5f-d36a36459563) HTTP Method: POST, Resource Path: /v1/issue
(b25eda70-670f-4a34-ad5f-d36a36459563) Method request path: {proxy=v1/issue}
(b25eda70-670f-4a34-ad5f-d36a36459563) Method request query string:
{}
(b25eda70-670f-4a34-ad5f-d36a36459563) Method request headers: {Authorization=**************************************************************************************************************************************************************************************************************************************************************************************************************************************************************D58YW4, Accept=application/json, User-Agent=libwww-perl/6.08, X-Forwarded-Proto=https, X-Forwarded-For=180.181.128.16, Host=xxx.xxxx.com, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=1-63e75267-28dfdf0352bb73e061ba500a, Content-Type=application/json}
(b25eda70-670f-4a34-ad5f-d36a36459563) Method request body before transformations:
<request payload>
(b25eda70-670f-4a34-ad5f-d36a36459563) Endpoint request URI: https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:payment-orders:PROD/invocations
(b25eda70-670f-4a34-ad5f-d36a36459563) Endpoint request headers: {X-Amz-Date=20230211T083135Z, x-amzn-apigateway-api-id=hrus1phje2, Accept=application/json, User-Agent=AmazonAPIGateway_hrus1phje2, Host=lambda.ap-southeast-2.amazonaws.com, X-Amz-Content-Sha256=898ccbf8d65637ee8124239956f8e80312149ea37ea820c68c598476abe3ca01, X-Amzn-Trace-Id=Root=1-63e75267-28dfdf0352bb73e061ba500a;Sampled=0, x-amzn-lambda-integration-tag=b25eda70-670f-4a34-ad5f-d36a36459563, Authorization=**************************************************************************************************************************************************************************************************************************************************************************************************************************************************b231f3, X-Amz-Source-Arn=arn:aws:execute-api:ap-southeast-2:222861176820:hrus1phje2/v1/ANY/{proxy+}, X-Amz-Security-Token= [TRUNCATED]
(b25eda70-670f-4a34-ad5f-d36a36459563) Endpoint request body after transformations: {"resource":"/{proxy+}","path":"/v1/issue","httpMethod":"POST","headers":{"Accept":"application/json","Authorization":"Bearer xxxxxxxxx","Content-Type":"application/json","Host":"xxx.xxxx.com","User-Agent":"libwww-perl/6.08","X-Amzn-Trace-Id":"Root=1-63e75267-28dfdf0352bb73e061ba500a","X-Forwarded-For":"180.181.128.16","X-Forwarded-Port":"443","X-Forwarded-Proto":"https"},"multiValueHeaders":{"Accept":["application/json"],"Authorization":["Bearer xxxxxxxxx [TRUNCATED]
(b25eda70-670f-4a34-ad5f-d36a36459563) Sending request to https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:orders:PROD/invocations
(b25eda70-670f-4a34-ad5f-d36a36459563) Received response. Status: 200, Integration latency: 85 ms
(b25eda70-670f-4a34-ad5f-d36a36459563) Endpoint response headers: {Date=Sat, 11 Feb 2023 08:31:35 GMT, Content-Type=application/json, Content-Length=422, Connection=keep-alive, x-amzn-RequestId=8caf7aa5-f5b5-48c3-a9db-60007eae7966, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=9, X-Amzn-Trace-Id=root=1-63e75267-28dfdf0352bb73e061ba500a;sampled=0}
(b25eda70-670f-4a34-ad5f-d36a36459563) Endpoint response body before transformations:
{
"statusCode": 200,
"multiValueHeaders": {
"Content-Type": [
"application/json"
],
"Vary": [
"Origin",
"Access-Control-Request-Method",
"Access-Control-Request-Headers"
]
},
"body": "<response payload>",
"isBase64Encoded": false
}
(b25eda70-670f-4a34-ad5f-d36a36459563) Method response body after transformations:
<response payload>
(b25eda70-670f-4a34-ad5f-d36a36459563) Method response headers: {Content-Type=application/json, Vary=Origin,Access-Control-Request-Method,Access-Control-Request-Headers, X-Amzn-Trace-Id=Root=1-63e75267-28dfdf0352bb73e061ba500a;Sampled=0}
(b25eda70-670f-4a34-ad5f-d36a36459563) Successfully completed execution
(b25eda70-670f-4a34-ad5f-d36a36459563) Method completed with status: 200
(b25eda70-670f-4a34-ad5f-d36a36459563) AWS Integration Endpoint RequestId : 8caf7aa5-f5b5-48c3-a9db-60007eae7966
(a08f7adb-75d4-48ac-9150-c71d800f1630) Extended Request Id: APwyIGGAywMFjqg=
(a08f7adb-75d4-48ac-9150-c71d800f1630) Starting authorizer: vug364 for request: a08f7adb-75d4-48ac-9150-c71d800f1630
(a08f7adb-75d4-48ac-9150-c71d800f1630) Incoming identity: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************aVuL_0
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request URI: https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:security-access-token-validator/invocations
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request headers: {x-amzn-lambda-integration-tag=a08f7adb-75d4-48ac-9150-c71d800f1630, Authorization=************************************************************************************************************************************************************************************************************************************************************************************************************8248da, X-Amz-Date=20230212T220110Z, x-amzn-apigateway-api-id=hrus1phje2, Accept=application/json, User-Agent=AmazonAPIGateway_hrus1phje2, X-Amz-Security-Token=[TRUNCATED]
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request body after transformations:
{
"type": "TOKEN",
"methodArn": "arn:aws:execute-api:ap-southeast-2:222861176820:hrus1phje2/v1/POST/v1/issue",
"authorizationToken": "Bearer XXXX"
}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Sending request to https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:security-access-token-validator/invocations
(a08f7adb-75d4-48ac-9150-c71d800f1630) Authorizer result body before parsing:
{
"principalId": "user",
"policyDocument": {
"Version": "2012-10-17",
"Statement": {
"Action": "execute-api:Invoke",
"Resource": "arn:aws:execute-api:ap-southeast-2:222861176820:hrus1phje2/v1/POST/v1/issue",
"Effect": "Allow"
}
}
}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Using valid authorizer policy for principal: **er
(a08f7adb-75d4-48ac-9150-c71d800f1630) Successfully completed authorizer execution
(a08f7adb-75d4-48ac-9150-c71d800f1630) Usage Plan check succeeded for API Key and API Stage hrus1phje2/v1
(a08f7adb-75d4-48ac-9150-c71d800f1630) Starting execution for request: a08f7adb-75d4-48ac-9150-c71d800f1630
(a08f7adb-75d4-48ac-9150-c71d800f1630) HTTP Method: POST, Resource Path: /v1/issue
(a08f7adb-75d4-48ac-9150-c71d800f1630) Method request path: {proxy=v1/issue}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Method request query string:
{}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Method request headers: {Authorization=**************************************************************************************************************************************************************************************************************************************************************************************************************************************************************aVuL_0, Accept=application/json, X-Forwarded-Proto=https, X-Forwarded-For=13.211.239.135, Host=xxx.xxxxxx.com, Accept-Encoding=gzip, deflate, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=1-63e961a6-407ccedc709a85302228a1b9, Content-Type=application/json; charset=utf-8}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Method request body before transformations:
{
<request payload>
}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request URI: https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:orders:PROD/invocations
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request headers: {X-Amz-Date=20230212T220111Z, x-amzn-apigateway-api-id=hrus1phje2, Accept=application/json, User-Agent=AmazonAPIGateway_hrus1phje2, Host=lambda.ap-southeast-2.amazonaws.com, X-Amz-Content-Sha256=59e81e2b4e6f0aa684118f5d602d69054aa7fa2b0f95383446dfbca3713b50d6, X-Amzn-Trace-Id=Root=1-63e961a6-407ccedc709a85302228a1b9;Sampled=0, x-amzn-lambda-integration-tag=a08f7adb-75d4-48ac-9150-c71d800f1630, Authorization=**************************************************************************************************************************************************************************************************************************************************************************************************************************************************e978db, X-Amz-Source-Arn=arn:aws:execute-api:ap-southeast-2:222861176820:hrus1phje2/v1/ANY/{proxy+}, X-Amz-Security-Token= [TRUNCATED]
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request body after transformations: {"resource":"/{proxy+}","path":"/v1/issue","httpMethod":"POST","headers":{"Accept":"application/json","Accept-Encoding":"gzip, deflate","Authorization":"Bearer xxxxxxxxxxxx","Content-Type":"application/json; charset=utf-8","Host":"xxx.xxxxxxs.com","X-Amzn-Trace-Id":"Root=1-63e961a6-407ccedc709a85302228a1b9","X-Forwarded-For":"13.211.239.135","X-Forwarded-Port":"443","X-Forwarded-Proto":"https"},"multiValueHeaders":{"Accept":["application/json"],"Accept-Encoding":["gzip, deflate"],"Authorization":["Bearer xxxxxxxxxx [TRUNCATED]
(a08f7adb-75d4-48ac-9150-c71d800f1630) Sending request to https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:payment-orders:PROD/invocations
(a08f7adb-75d4-48ac-9150-c71d800f1630) Received response. Status: 408, Integration latency: 7061 ms
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint response headers: {Date=Sun, 12 Feb 2023 22:01:18 GMT, Content-Type=application/json, Content-Length=90, Connection=keep-alive, x-amzn-RequestId=a2d71fa8-296e-4c16-b3ca-04e158297833, x-amzn-ErrorType=SnapStartTimeoutException}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint response body before transformations:
{
"Message": "Lambda couldn't restore the snapshot within the timeout limit.",
"Type": "User"
}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Lambda invocation failed with status: 408. Lambda request id: a2d71fa8-296e-4c16-b3ca-04e158297833
(a08f7adb-75d4-48ac-9150-c71d800f1630) Execution failed due to configuration error: Lambda couldn't restore the snapshot within the timeout limit.
(a08f7adb-75d4-48ac-9150-c71d800f1630) Method completed with status: 500
(a08f7adb-75d4-48ac-9150-c71d800f1630) AWS Integration Endpoint RequestId : a2d71fa8-296e-4c16-b3ca-04e158297833
(2aec1649-3068-4ff4-858c-ef17a5acbb2b) Extended Request Id: AQRoaESvywMFjqg=
(2aec1649-3068-4ff4-858c-ef17a5acbb2b) Starting authorizer: vug364 for request: 2aec1649-3068-4ff4-858c-ef17a5acbb2b
(2aec1649-3068-4ff4-858c-ef17a5acbb2b) Incoming identity: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************aVuL_0```
Language
English
asked a year ago526 viewslg...
1 Answer
- Newest
- Most votes
- Most comments
Are these answers helpful? Upvote the correct answer to help the community benefit from your knowledge.
0
Hi Nalaka,
According to below article:
Restore phase (Lambda SnapStart only) When you first invoke a SnapStart function and as the function scales up, Lambda resumes new execution environments from the persisted snapshot instead of initializing the function from scratch. If you have an afterRestore() runtime hook, the code runs at the end of the Restore phase. You are charged for the duration of afterRestore() runtime hooks. The runtime (JVM) must load and afterRestore() runtime hooks must complete within the timeout limit (2 seconds). Otherwise, you'll get a SnapStartTimeoutException. When the Restore phase completes, Lambda invokes the function handler (the Invoke phase).
https://docs.amazonaws.cn/en_us/lambda/latest/dg/lambda-runtime-environment.html
Relevant content
- asked 4 years agolg...
- asked 2 years agolg...
- Accepted Answerasked 9 months agolg...
- asked 18 days agolg...
AWS OFFICIALUpdated 3 years ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 10 months ago
Thanks @alatech. However, we do not have any code in an afterRestore() hook. Our lambda function is running with Java 11 and Spring. Do you know of any code that gets called automatically in the as a part of an afterRestore() hook?
Hi, honestly, no. afterRestore is just a hook that allows you to have some custom logic in case you wish to add some code after a "lambda snapshot" is restored.
I would suggest to look in your logs and see if there is some pattern when the exception happens. I suspect could be related to some network connectivity issue that usually resolves itself.