Elastic Beanstalk Deployments Periodically Failing | /var/app/current: directory not empty | Amazon Linux 2/3.4.1

Question

When deploying on Elastic Beanstalk (Amazon Linux 2), deployment periodically fails because the `/var/app/current` directory isn't empty. There is little consistency in when a deployment fails or succeeds. With the exact same code base, we can have multiple successes in a row, multiple failures in a row, or any combination thereof. This issue does not seem specific to one platform either based on this [old AWS Developer Forum post](https://forums.aws.amazon.com/thread.jspa?threadID=335593) that experienced it on PHP, while we're experiencing it on Ruby. Here are the more detailed specifics which are a combination of comments from the now archived thread, and my updated experiences since it was archived.

--

Error on `Platform PHP 7.4, Amazon Linux 2/3.1.5`
```
2021/02/12 19:16:54.204134 Executing instruction: FlipApplication
2021/02/12 19:16:55.339260 An error occurred during execution of command https://forums.aws.amazon.com/ - https://forums.aws.amazon.com/. Stop running the command. Error: remove current dir failed: unlinkat /var/app/current: directory not empty
2021/02/12 19:16:55.339271 Executing cleanup logic
2021/02/12 19:16:55.339344 CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":}
```

Error on `Platform Ruby 3.0, Amazon Linux 2/3.4.1`
```
2022/02/01 22:36:13.837697 [ERROR] An error occurred during execution of command [app-deploy] - [flip ruby application]. Stop running the command. Error: remove current dir failed: unlinkat /var/app/current/tmp/cache/bootsnap/compile-cache-iseq/c0: directory not empty 
2022/02/01 22:36:13.838041 [INFO] Executing cleanup logic
2022/02/01 22:36:13.838350 [INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[{"msg":"Instance deployment failed. For details, see 'eb-engine.log'.","timestamp":1643754973,"severity":"ERROR"}]}]}
```

I also experienced this same issue on `Platform Ruby 2.7, Amazon Linux 2/3.2.4`

--

I am able to get around this issue by removing the ruby gem `bootsnap` which caches some components of the Rails app, but as this issue seems to affect other language platforms as well I can't really blame `bootsnap` directly, and removing it does detrimentally affect our app. I'm not sure if the PHP users also use some sort of caching process.

I've also attempted changing file permissions via deployment hooks to force delete the contents of `/var/app/current` but the same issue still is presented.

Any ideas would be very welcomed!

Answer

Hello,

It might be quite challenging to analyze this issue because a [support case](https://console.aws.amazon.com/support/home#/case/create) would helpful so as to get logs and more details. However at a high level, I would say the common cause I have seen is similar to your scenario. In this case, it was discovered that a gem or script was modifying the directory.

Most of the time that's the root cause, which is where a custom script assumes the directory exists or assumes it is empty and tries to alter the directory.
Elastic Beanstalk (EB)  doesn't want scripts altering this directory as EB prepares everything usually into /var/app/staging and copies over into /var/app/current.

It is recommended to not modify the /var/app/current so it does not interfere with how EB maintains it. If any scripts needs to write/modify files of a new application version, using something like [container commands](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#linux-container-commands) is better. Container commands are commands ran in the /var/app/staging directory before files are copied to /var/app/current.

As for why it happens sometimes and not others, deployment scripts that try to modify the /var/app/current directory can behave very differently on first time deployment vs subsequent deployments to the same EC2 instance. It is much safer to manipulate files in /var/app/staging and allow EB to move the final files to /var/app/current for you.

An additional helpful link is [Platform hooks](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/platforms-linux-extend.html). Note that using different ebextension blocks (ex. container commands instead of commands) or using the newer platform hooks such as post deploy hooks will have different behavior and run in different order and you would have to experiment or consider what one best fits your needs.

Elastic Beanstalk Deployments Periodically Failing | /var/app/current: directory not empty | Amazon Linux 2/3.4.1

Relevant content