What could be the cause of randomly appearing SnapStartTimeoutException

0

We have a lambda function invoked via api-gateway. Occasionally we see SnapStartTimeoutExceptions in the api-gateway logs and the client gets an http 500 error. We do not see any errors in the logs of the lambda function. What are the potential causes of a SnapStartTimeoutException?

Please see the end of the logs below.

(b25eda70-670f-4a34-ad5f-d36a36459563) HTTP Method: POST, Resource Path: /v1/issue
(b25eda70-670f-4a34-ad5f-d36a36459563) Method request path: {proxy=v1/issue}
(b25eda70-670f-4a34-ad5f-d36a36459563) Method request query string: 
{}
(b25eda70-670f-4a34-ad5f-d36a36459563) Method request headers: {Authorization=**************************************************************************************************************************************************************************************************************************************************************************************************************************************************************D58YW4, Accept=application/json, User-Agent=libwww-perl/6.08, X-Forwarded-Proto=https, X-Forwarded-For=180.181.128.16, Host=xxx.xxxx.com, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=1-63e75267-28dfdf0352bb73e061ba500a, Content-Type=application/json}
(b25eda70-670f-4a34-ad5f-d36a36459563) Method request body before transformations: 
<request payload>
(b25eda70-670f-4a34-ad5f-d36a36459563) Endpoint request URI: https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:payment-orders:PROD/invocations
(b25eda70-670f-4a34-ad5f-d36a36459563) Endpoint request headers: {X-Amz-Date=20230211T083135Z, x-amzn-apigateway-api-id=hrus1phje2, Accept=application/json, User-Agent=AmazonAPIGateway_hrus1phje2, Host=lambda.ap-southeast-2.amazonaws.com, X-Amz-Content-Sha256=898ccbf8d65637ee8124239956f8e80312149ea37ea820c68c598476abe3ca01, X-Amzn-Trace-Id=Root=1-63e75267-28dfdf0352bb73e061ba500a;Sampled=0, x-amzn-lambda-integration-tag=b25eda70-670f-4a34-ad5f-d36a36459563, Authorization=**************************************************************************************************************************************************************************************************************************************************************************************************************************************************b231f3, X-Amz-Source-Arn=arn:aws:execute-api:ap-southeast-2:222861176820:hrus1phje2/v1/ANY/{proxy+}, X-Amz-Security-Token= [TRUNCATED]
(b25eda70-670f-4a34-ad5f-d36a36459563) Endpoint request body after transformations: {"resource":"/{proxy+}","path":"/v1/issue","httpMethod":"POST","headers":{"Accept":"application/json","Authorization":"Bearer xxxxxxxxx","Content-Type":"application/json","Host":"xxx.xxxx.com","User-Agent":"libwww-perl/6.08","X-Amzn-Trace-Id":"Root=1-63e75267-28dfdf0352bb73e061ba500a","X-Forwarded-For":"180.181.128.16","X-Forwarded-Port":"443","X-Forwarded-Proto":"https"},"multiValueHeaders":{"Accept":["application/json"],"Authorization":["Bearer xxxxxxxxx [TRUNCATED]
(b25eda70-670f-4a34-ad5f-d36a36459563) Sending request to https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:orders:PROD/invocations
(b25eda70-670f-4a34-ad5f-d36a36459563) Received response. Status: 200, Integration latency: 85 ms
(b25eda70-670f-4a34-ad5f-d36a36459563) Endpoint response headers: {Date=Sat, 11 Feb 2023 08:31:35 GMT, Content-Type=application/json, Content-Length=422, Connection=keep-alive, x-amzn-RequestId=8caf7aa5-f5b5-48c3-a9db-60007eae7966, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=9, X-Amzn-Trace-Id=root=1-63e75267-28dfdf0352bb73e061ba500a;sampled=0}
(b25eda70-670f-4a34-ad5f-d36a36459563) Endpoint response body before transformations: 
{
    "statusCode": 200,
    "multiValueHeaders": {
        "Content-Type": [
            "application/json"
        ],
        "Vary": [
            "Origin",
            "Access-Control-Request-Method",
            "Access-Control-Request-Headers"
        ]
    },
    "body": "<response payload>",
    "isBase64Encoded": false
}
(b25eda70-670f-4a34-ad5f-d36a36459563) Method response body after transformations: 
<response payload>
(b25eda70-670f-4a34-ad5f-d36a36459563) Method response headers: {Content-Type=application/json, Vary=Origin,Access-Control-Request-Method,Access-Control-Request-Headers, X-Amzn-Trace-Id=Root=1-63e75267-28dfdf0352bb73e061ba500a;Sampled=0}
(b25eda70-670f-4a34-ad5f-d36a36459563) Successfully completed execution
(b25eda70-670f-4a34-ad5f-d36a36459563) Method completed with status: 200
(b25eda70-670f-4a34-ad5f-d36a36459563) AWS Integration Endpoint RequestId : 8caf7aa5-f5b5-48c3-a9db-60007eae7966
(a08f7adb-75d4-48ac-9150-c71d800f1630) Extended Request Id: APwyIGGAywMFjqg=
(a08f7adb-75d4-48ac-9150-c71d800f1630) Starting authorizer: vug364 for request: a08f7adb-75d4-48ac-9150-c71d800f1630
(a08f7adb-75d4-48ac-9150-c71d800f1630) Incoming identity: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************aVuL_0
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request URI: https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:security-access-token-validator/invocations
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request headers: {x-amzn-lambda-integration-tag=a08f7adb-75d4-48ac-9150-c71d800f1630, Authorization=************************************************************************************************************************************************************************************************************************************************************************************************************8248da, X-Amz-Date=20230212T220110Z, x-amzn-apigateway-api-id=hrus1phje2, Accept=application/json, User-Agent=AmazonAPIGateway_hrus1phje2, X-Amz-Security-Token=[TRUNCATED]
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request body after transformations: 
{
    "type": "TOKEN",
    "methodArn": "arn:aws:execute-api:ap-southeast-2:222861176820:hrus1phje2/v1/POST/v1/issue",
    "authorizationToken": "Bearer XXXX"
}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Sending request to https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:security-access-token-validator/invocations
(a08f7adb-75d4-48ac-9150-c71d800f1630) Authorizer result body before parsing: 
{
    "principalId": "user",
    "policyDocument": {
        "Version": "2012-10-17",
        "Statement": {
            "Action": "execute-api:Invoke",
            "Resource": "arn:aws:execute-api:ap-southeast-2:222861176820:hrus1phje2/v1/POST/v1/issue",
            "Effect": "Allow"
        }
    }
}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Using valid authorizer policy for principal: **er
(a08f7adb-75d4-48ac-9150-c71d800f1630) Successfully completed authorizer execution
(a08f7adb-75d4-48ac-9150-c71d800f1630) Usage Plan check succeeded for API Key  and API Stage hrus1phje2/v1
(a08f7adb-75d4-48ac-9150-c71d800f1630) Starting execution for request: a08f7adb-75d4-48ac-9150-c71d800f1630
(a08f7adb-75d4-48ac-9150-c71d800f1630) HTTP Method: POST, Resource Path: /v1/issue
(a08f7adb-75d4-48ac-9150-c71d800f1630) Method request path: {proxy=v1/issue}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Method request query string: 
{}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Method request headers: {Authorization=**************************************************************************************************************************************************************************************************************************************************************************************************************************************************************aVuL_0, Accept=application/json, X-Forwarded-Proto=https, X-Forwarded-For=13.211.239.135, Host=xxx.xxxxxx.com, Accept-Encoding=gzip, deflate, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=1-63e961a6-407ccedc709a85302228a1b9, Content-Type=application/json; charset=utf-8}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Method request body before transformations: 
{
   <request payload>
}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request URI: https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:orders:PROD/invocations
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request headers: {X-Amz-Date=20230212T220111Z, x-amzn-apigateway-api-id=hrus1phje2, Accept=application/json, User-Agent=AmazonAPIGateway_hrus1phje2, Host=lambda.ap-southeast-2.amazonaws.com, X-Amz-Content-Sha256=59e81e2b4e6f0aa684118f5d602d69054aa7fa2b0f95383446dfbca3713b50d6, X-Amzn-Trace-Id=Root=1-63e961a6-407ccedc709a85302228a1b9;Sampled=0, x-amzn-lambda-integration-tag=a08f7adb-75d4-48ac-9150-c71d800f1630, Authorization=**************************************************************************************************************************************************************************************************************************************************************************************************************************************************e978db, X-Amz-Source-Arn=arn:aws:execute-api:ap-southeast-2:222861176820:hrus1phje2/v1/ANY/{proxy+}, X-Amz-Security-Token= [TRUNCATED]
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint request body after transformations: {"resource":"/{proxy+}","path":"/v1/issue","httpMethod":"POST","headers":{"Accept":"application/json","Accept-Encoding":"gzip, deflate","Authorization":"Bearer xxxxxxxxxxxx","Content-Type":"application/json; charset=utf-8","Host":"xxx.xxxxxxs.com","X-Amzn-Trace-Id":"Root=1-63e961a6-407ccedc709a85302228a1b9","X-Forwarded-For":"13.211.239.135","X-Forwarded-Port":"443","X-Forwarded-Proto":"https"},"multiValueHeaders":{"Accept":["application/json"],"Accept-Encoding":["gzip, deflate"],"Authorization":["Bearer xxxxxxxxxx [TRUNCATED]
(a08f7adb-75d4-48ac-9150-c71d800f1630) Sending request to https://lambda.ap-southeast-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-southeast-2:222861176820:function:payment-orders:PROD/invocations
(a08f7adb-75d4-48ac-9150-c71d800f1630) Received response. Status: 408, Integration latency: 7061 ms
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint response headers: {Date=Sun, 12 Feb 2023 22:01:18 GMT, Content-Type=application/json, Content-Length=90, Connection=keep-alive, x-amzn-RequestId=a2d71fa8-296e-4c16-b3ca-04e158297833, x-amzn-ErrorType=SnapStartTimeoutException}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Endpoint response body before transformations: 
{
    "Message": "Lambda couldn't restore the snapshot within the timeout limit.",
    "Type": "User"
}
(a08f7adb-75d4-48ac-9150-c71d800f1630) Lambda invocation failed with status: 408. Lambda request id: a2d71fa8-296e-4c16-b3ca-04e158297833
(a08f7adb-75d4-48ac-9150-c71d800f1630) Execution failed due to configuration error: Lambda couldn't restore the snapshot within the timeout limit.
(a08f7adb-75d4-48ac-9150-c71d800f1630) Method completed with status: 500
(a08f7adb-75d4-48ac-9150-c71d800f1630) AWS Integration Endpoint RequestId : a2d71fa8-296e-4c16-b3ca-04e158297833
(2aec1649-3068-4ff4-858c-ef17a5acbb2b) Extended Request Id: AQRoaESvywMFjqg=
(2aec1649-3068-4ff4-858c-ef17a5acbb2b) Starting authorizer: vug364 for request: 2aec1649-3068-4ff4-858c-ef17a5acbb2b
(2aec1649-3068-4ff4-858c-ef17a5acbb2b) Incoming identity: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************aVuL_0```
1개 답변
0

Hi Nalaka,

According to below article:

Restore phase (Lambda SnapStart only) When you first invoke a SnapStart function and as the function scales up, Lambda resumes new execution environments from the persisted snapshot instead of initializing the function from scratch. If you have an afterRestore() runtime hook, the code runs at the end of the Restore phase. You are charged for the duration of afterRestore() runtime hooks. The runtime (JVM) must load and afterRestore() runtime hooks must complete within the timeout limit (2 seconds). Otherwise, you'll get a SnapStartTimeoutException. When the Restore phase completes, Lambda invokes the function handler (the Invoke phase).

https://docs.amazonaws.cn/en_us/lambda/latest/dg/lambda-runtime-environment.html

profile picture
전문가
답변함 일 년 전
profile picture
전문가
검토됨 2달 전
  • Thanks @alatech. However, we do not have any code in an afterRestore() hook. Our lambda function is running with Java 11 and Spring. Do you know of any code that gets called automatically in the as a part of an afterRestore() hook?

  • Hi, honestly, no. afterRestore is just a hook that allows you to have some custom logic in case you wish to add some code after a "lambda snapshot" is restored.

    I would suggest to look in your logs and see if there is some pattern when the exception happens. I suspect could be related to some network connectivity issue that usually resolves itself.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠