Skip to content

Greengrass lambdas are not starting due to log pipe blocked errors

0

Same question as this one: https://repost.aws/questions/QUjDFdSS7NSAe6Q3C9w7ZJyw/greengrass-lambda-failing-to-start-due-to-log-pipe-blocked

But since that one never received a proper answer, I am posting another one. I am having the exact same issue. The core device filled up the hard drive space, and after that lambdas are refusing to start again with these log pipe blocked errors. I cleared hard drive space so that is no longer a problem. Rebooting the device, restarting the greengrass service, redeploying components does not help.

Rewriting all 10 lambdas we have in our project to use generic components is not an option at this time. Yes, all lambdas are affected and having the same problem with the same error logs. One of the users on the original question asked for full log snippet, here it is:

2024-07-22T13:37:24.448Z [INFO] (pool-2-thread-56) dev.development.StateOperationalData: Lambda log pipe /opt/greengrass/work/dev.development.StateOperationalData/logs/0/trace is blocked, attempting to unblock it. {serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=STOPPING}
2024-07-22T13:37:24.504Z [INFO] (pool-2-thread-56) dev.development.StateOperationalData: shell-runner-start. {scriptName=services.dev.development.StateOperationalData.lifecycle.startup.script, serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=STARTING, command=["/opt/greengrass/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.13/lambda..."]}
2024-07-22T13:37:33.885Z [WARN] (Copier) dev.development.StateOperationalData: stderr. Killed. {scriptName=services.dev.development.StateOperationalData.lifecycle.startup.script, serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=STARTING}
2024-07-22T13:37:33.886Z [INFO] (Copier) dev.development.StateOperationalData: Startup script exited. {exitCode=137, serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=STARTING}
2024-07-22T13:37:33.888Z [INFO] (pool-2-thread-56) dev.development.StateOperationalData: shell-runner-start. {scriptName=services.dev.development.StateOperationalData.lifecycle.shutdown.script, serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=BROKEN, command=["/opt/greengrass/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.13/lambda..."]}
2024-07-22T13:37:47.221Z [WARN] (Copier) dev.development.StateOperationalData: stderr. Killed. {scriptName=services.dev.development.StateOperationalData.lifecycle.shutdown.script, serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=BROKEN}
2024-07-22T13:37:59.603Z [WARN] (Copier) dev.development.StateOperationalData: stderr. Killed. {scriptName=services.dev.development.StateOperationalData.lifecycle.shutdown.script, serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=BROKEN}
2024-07-22T13:37:59.604Z [WARN] (pool-2-thread-56) dev.development.StateOperationalData: shell-runner-error. {scriptName=services.dev.development.StateOperationalData.lifecycle.shutdown.script, serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=BROKEN, command=["/opt/greengrass/packages/artifacts/aws.greengrass.LambdaLauncher/2.0.13/lambda..."]}
2024-07-22T13:37:59.604Z [INFO] (pool-2-thread-56) dev.development.StateOperationalData: Lambda log pipe /opt/greengrass/work/dev.development.StateOperationalData/logs/0/warn is blocked, attempting to unblock it. {serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=BROKEN}
2024-07-22T13:37:59.604Z [INFO] (pool-2-thread-56) dev.development.StateOperationalData: Lambda log pipe /opt/greengrass/work/dev.development.StateOperationalData/logs/0/fatal is blocked, attempting to unblock it. {serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=BROKEN}
2024-07-22T13:37:59.604Z [INFO] (pool-2-thread-56) dev.development.StateOperationalData: Lambda log pipe /opt/greengrass/work/dev.development.StateOperationalData/logs/0/info is blocked, attempting to unblock it. {serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=BROKEN}
2024-07-22T13:37:59.604Z [INFO] (pool-2-thread-56) dev.development.StateOperationalData: Lambda log pipe /opt/greengrass/work/dev.development.StateOperationalData/logs/0/debug is blocked, attempting to unblock it. {serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=BROKEN}
2024-07-22T13:37:59.604Z [INFO] (pool-2-thread-56) dev.development.StateOperationalData: Lambda log pipe /opt/greengrass/work/dev.development.StateOperationalData/logs/0/error is blocked, attempting to unblock it. {serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=BROKEN}
2024-07-22T13:37:59.605Z [INFO] (pool-2-thread-56) dev.development.StateOperationalData: Lambda log pipe /opt/greengrass/work/dev.development.StateOperationalData/logs/0/trace is blocked, attempting to unblock it. {serviceInstance=0, serviceName=dev.development.StateOperationalData, currentState=BROKEN} 

Plz help!

asked 2 years ago246 views
1 Answer
0
Accepted Answer

Updating my old answers - this issue was due to broken lambda manager. It was resolved by clearing the deployment (deploying only Nucleus), deleting the working files of the lambda manager component and then deploying the full deployment with components again.

answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.