Elastic Beanstalk deployments timing out

Question

I'm currently unable to deploy my app to the production environment, deployments time out but I'm struggling to work out why.

It's a php web app (Moodle), I can deploy to my staging environment without issue but it fails on production.

The application maintains a number of different types of caches, some are on the local file system and some are on the EFS. Others I've managed to move off to Elasticache.

I also regularly hit an issue where the server maxes out the number of allowed inodes on the Linux filesystem which I think is related.

My suspicion is that the deployment is doing some kind of check or update against all files, maybe updating permissions? And this is taking too long.

I don't really know where to find logs that will tell me this, when I grab the full logs from an instance, none of them have any detailed info on the deployment process or what step is causing the delay.

Does anyone have any pointers on where I should be looking?

Answer

Has the system always been like this? When did this issue start? What changes did you make recently?

> The application maintains a number of different types of caches, some are on the local file system and some are on the EFS. Others I've managed to move off to Elasticache.
> I also regularly hit an issue where the server maxes out the number of allowed inodes on the Linux filesystem which I think is related.

- How are you monitoring inode usage?
- Do you you see gradually/steeply rising usage, or peaks and troughs?
- Is there a particular file system where inodes are concentrated?
  - run `df -i` whether one FS has disproportionately higher IUse% than the rest
- Determine where on your server inodes are concentrated. I would start with `/tmp` and `/var`
  - du --inodes /tmp 2>/dev/null -s # note that you'll likely need sudo to access directories that require administrative privileges

> My suspicion is that the deployment is doing some kind of check or update against all files, maybe updating permissions? And this is taking too long.
Why do you have this suspicison?

Does your application create a lot of on-the-fly tmp files? Did you try running your application in DEBUG logging mode? I would try this in staging to determine from logs whether there could be a proliferation of tmp files.

Elastic Beanstalk deployments timing out

Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Licensed under the Amazon Software License (the "License"). You may not use

this file except in compliance with the License. A copy of the License is

located at

https://aws.amazon.com/asl/

or in the "license" file accompanying this file. This file is distributed on

an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or

implied. See the License for the specific language governing permissions

and limitations under the License.

Relevant content