Why is shutdown script on component in greengrass deployment not completing before new components are starting?

Grengrass V2 component recipes have no way of specifying that a component conflicts with another (i.e. that one component cannot be started while the specified other is running). The easiest mitigation without any changes to either recipe is to deploy the removal of the old component, and then in a different, subsequent deployment add the new, conflicting component. This guarantees the shutdown of the old component before the run step of the new component.

In AWS IoT Greengrass v2, the issue you're experiencing is likely related to the shutdown script timing out before it can fully complete its task. When a component is being replaced or updated, Greengrass has a default timeout period for the shutdown script to complete. If your shutdown script doesn't finish within this configured timeout, Greengrass will proceed with starting the new component, even if the old container is still running.

There are a few ways to address this issue:

1. Increase the timeout period for your shutdown script in the component recipe. You can specify a longer Timeout value in the recipe's shutdown section to give your container more time to stop properly.

2. Modify your shutdown script to be more effective. Instead of just using docker stop, which sends a SIGTERM and waits for the container to gracefully shut down, you might need a more forceful approach. Consider using:

docker stop <container name> || docker rm -f <container name> || true

This will attempt to stop the container gracefully first, then forcibly remove it if needed.

3. Use docker compose down instead of just stopping the container if you're using Docker Compose. This ensures containers are properly removed:

docker compose -f {artifacts:path}/docker-compose.yml down

This stops and removes containers but keeps the images intact.

4. For database containers specifically, you might need to add additional commands to ensure proper shutdown of the database process before stopping the container, as databases often need time to flush data to disk and release locks.

Remember that when a component's shutdown script fails to complete within the timeout period, Greengrass logs will show a SHUTDOWN_TIMEOUT error. Checking these logs can help confirm if this is indeed the issue you're facing.
Sources
Error response from daemon: Conflict. The container name "/<container_name>" is already in use by container "xxxx". You have to remove (or rename) that container to be able to reuse that name! | AWS re:Post
Detailed component status codes - AWS IoT Greengrass