Hi, this is really unusual for us. We've been running ECS in anger for the last 8 years with very few issues.
Today, the ecs-agents and all docker tasks on all our test instances stopped working overnight. The AWS console didn't show pretty much any errors except for our dev instance of redis failing to run, but when we logged onto the actual (3) instances no ecs agents were running. No errors that we could see in any CloudWatch logs, just nothing working and errors on all our builds. We resolved the issue by stopping then starting all 3 instances. The "restart" command did not help.
This was bad enough, but then this afternoon the same thing happened in PRODUCTION to our 6 prod instances. The same lack of clues. Running "docker ps" showed no running tasks, although again the AWS console seemed to be stuck in the past.
This is so weird we really don't know where to go with this next. Thoughts anyone? All help appreciated!
Thanks, that's a good clue but unfortunately in order to recover our system we had to stop and start the instance. Just a simple restart did not work. This sadly means that there are no old logs on what is in this case a new instance.
If it happens again, please check the logs. Currently, if there are no old instances left, I don't have any information to troubleshoot, so I think it will be difficult to further investigate the cause.
Thanks, we just had to prioritise getting production running above troubleshooting. Can you think of anything that would be likely to kill the ecs-agent on multiple instances across multiple AZs simultaneously? This happened to us in our test env yesterday, followed by production a few hours later.