CodeDeploy Windows Agent unexpected restart

Question

Hi

I'm having an issue deploying my app using CodeDeploy on Windows Server 2022. It only happens when the deployment is triggered by an Auto Scaling Group lifecycle hook. Triggering the deployment through a Code Pipeline works as expected. From the application logs, it appears as if my PowerShell script is never called since the first line of my script writes a debug message to the log. This is all I see:

```
2023-09-19 18:20:23 LifecycleEvent - AfterInstall
2023-09-19 18:20:23 Script - after_bootstrap.ps1
```
Looking into the agent logs, it looks like the agent is being restarted for some reason:

```
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Command Status = InProgress
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Command Status = InProgress
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling #.execute_command
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling #.execute_command
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::DeploymentSpecification: **REDACTED**
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::DeploymentSpecification: **REDACTED**
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Command AfterInstall maps to method after_install
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Command AfterInstall maps to method after_install
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::DeploymentSpecification: **REDACTED**
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::DeploymentSpecification: Parse: **REDACTED**
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Successfully parsed the deployment spec
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Successfully parsed the deployment spec
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Creating deployment root directory C:\ProgramData/Amazon/CodeDeploy/ba94a190-9986-4481-967f-158ee7a
86bf4/d-SJCEE6VR1
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Creating deployment root directory C:\ProgramData/Amazon/CodeDeploy/ba94a190-9986-4481-967f-158ee7a
86bf4/d-SJCEE6VR1
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::HookExecutor: Checking for app spec in C:\ProgramData/Amazon/CodeDeploy/ba94a190-9986-4481-967f-158ee7a86bf4/d-SJCEE
6VR1/deployment-archive/appspec.yml
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::HookExecutor: Checking for app spec in C:\ProgramData/Amazon/CodeDeploy/ba94a190-9986-4481-967f-158ee7a86bf4/d-SJCEE
6VR1/deployment-archive/appspec.yml
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2023-09-19T18:20:23 DEBUG [codedeploy-agent(1636)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2023-09-19T18:20:23 INFO  [codedeploy-agent(1636)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.6.0.49_msi.
2023-09-19T18:20:23 INFO  [codedeploy-agent(1636)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.6.0.49_msi.
2023-09-19T18:20:24 INFO  [codedeploy-agent(1636)]: CodeDeploy Instance Agent Service: stopping the agent
2023-09-19T18:20:24 INFO  [codedeploy-agent(1636)]: CodeDeploy Instance Agent Service: stopping the agent
2023-09-19T18:23:42 INFO  [codedeploy-agent(2104)]: CodeDeploy Instance Agent Service: started
2023-09-19T18:23:42 INFO  [codedeploy-agent(2104)]: CodeDeploy Instance Agent Service: started
2023-09-19T18:23:42 INFO  [codedeploy-agent(2104)]: On Premises config file does not exist or not readable
2023-09-19T18:23:42 INFO  [codedeploy-agent(2104)]: On Premises config file does not exist or not readable
2023-09-19T18:23:42 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Configuring deploy control client: Region="us-east-1"
2023-09-19T18:23:42 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Configuring deploy control client: Region="us-east-1"
2023-09-19T18:23:42 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Deploy control endpoint override=
2023-09-19T18:23:42 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Deploy control endpoint override=
2023-09-19T18:23:42 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Enable auth policy = false
2023-09-19T18:23:42 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Enable auth policy = false
2023-09-19T18:23:42 INFO  [codedeploy-agent(2104)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2023-09-19T18:23:42 INFO  [codedeploy-agent(2104)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2023-09-19T18:23:42 INFO  [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Archives to retain is: 5}
2023-09-19T18:23:42 INFO  [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Archives to retain is: 5}
2023-09-19T18:23:42 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Initializing Host Agent: Host Identifier = **REDACTED**
2023-09-19T18:23:42 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Initializing Host Agent: Host Identifier = **REDACTED**
2023-09-19T18:23:42 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Validating CodeDeploy Plugin Configuration
2023-09-19T18:23:42 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Validating CodeDeploy Plugin Configuration
2023-09-19T18:23:43 INFO  [codedeploy-agent(2104)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2023-09-19T18:23:43 INFO  [codedeploy-agent(2104)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2023-09-19T18:23:43 INFO  [codedeploy-agent(2104)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2023-09-19T18:23:43 INFO  [codedeploy-agent(2104)]: CodeDeploy endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2023-09-19T18:23:43 DEBUG [codedeploy-agent(2104)]: Current deploy control endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2023-09-19T18:23:43 DEBUG [codedeploy-agent(2104)]: Current deploy control endpoint: https://codedeploy-commands.us-east-1.amazonaws.com
2023-09-19T18:23:43 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: CodeDeploy Plugin Configuration is valid
2023-09-19T18:23:43 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: CodeDeploy Plugin Configuration is valid
2023-09-19T18:23:43 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2023-09-19T18:23:43 DEBUG [codedeploy-agent(2104)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PollHostCommand:
2023-09-19T18:23:43 INFO  [codedeploy-agent(2104)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.6.0.49_msi.
2023-09-19T18:23:43 INFO  [codedeploy-agent(2104)]: Version file found in C:/ProgramData/Amazon/CodeDeploy/.version with agent version OFFICIAL_1.6.0.49_msi.
2023-09-19T18:24:29 INFO  [codedeploy-agent(2104)]: [Aws::CodeDeployCommand::Client 200 45.238725 0 retries] poll_host_command(host_identifier:"**REDACTED**")
```

The deployment eventually times out after an hour and tries again. Sometimes it is successful, but more often than not it fails in this manner. This is new behavior for us since we moved from Windows 2012 R2 to Windows Server 2022. If it never succeeded I would suspect a permissions issue, but since it sometimes works from a lifecycle hook and always works using CodePipeline I'm not really sure what to try.

Thanks for your help

Accepted Answer

The issue turned out to be that the CodeDeploy agent service was set to the "Automatic" start up type. I'm guessing that when a new EC2 instance was started, the CodeDeploy agent would begin the application install and was then interrupted by either sysprep or another startup/configure service. Once I changed the agent service to "Manual" start up type and then started the service with a PowerShell user data script, everything started working as expected.

Answer

Hello there!

From the details you shared, I observed that the issue is occurring only when you are using Windows Server 2022. You didn't notice the same behavior where auto Scaling runs all the lifecycle hooks attached to the Auto Scaling group, including the lifecycle hook created by CodeDeploy on Windows 2012 R2. Is that my correct understanding?

From the log shared, I can see that you are using latest codedeploy version 1.6.0. However the agent restarts after 'PollHostCommand' was called and eventually times out after multiple retries.

I understand that the deployment gets succeeded sometime but fails most of the time. From the logs provided it is not very clear what caused the agent to restart or why the deployment is failing intermittently.

For further investigation I would recommend reaching out directly to AWS Code Deploy Support team so that we can review the resources and troubleshoot the issue.

CodeDeploy Windows Agent unexpected restart

Relevant content