- Newest
- Most votes
- Most comments
Has the system always been like this? When did this issue start? What changes did you make recently?
The application maintains a number of different types of caches, some are on the local file system and some are on the EFS. Others I've managed to move off to Elasticache. I also regularly hit an issue where the server maxes out the number of allowed inodes on the Linux filesystem which I think is related.
- How are you monitoring inode usage?
- Do you you see gradually/steeply rising usage, or peaks and troughs?
- Is there a particular file system where inodes are concentrated?
- run
df -i
whether one FS has disproportionately higher IUse% than the rest
- run
- Determine where on your server inodes are concentrated. I would start with
/tmp
and/var
- du --inodes /tmp 2>/dev/null -s # note that you'll likely need sudo to access directories that require administrative privileges
My suspicion is that the deployment is doing some kind of check or update against all files, maybe updating permissions? And this is taking too long. Why do you have this suspicison?
Does your application create a lot of on-the-fly tmp files? Did you try running your application in DEBUG logging mode? I would try this in staging to determine from logs whether there could be a proliferation of tmp files.
Relevant content
- asked 2 years ago
- asked 2 years ago
- AWS OFFICIALUpdated 3 months ago
- AWS OFFICIALUpdated a month ago
- AWS OFFICIALUpdated 3 months ago
Thanks for the genuine response (everything else here seems to be copied from ChatGPT)
I've managed to reduce the inode usage by clearing out my application caches. Unfortunately deployments are still timing out, so I think it's doing some across the EFS which still has a lot of required files.
In the eb-activity.log on the instance I can see it stops at this line: [Application update 20240416.2-640@315/AppDeployStage0/AppDeployPreHook/12_update_permissions.sh] : Starting activity...
I think the shell script is this one, my EFS is mounted at /var/app/moodledata/ so I don't think it would be trying to reset those permissions? /opt/elasticbeanstalk/hooks/appdeploy/pre/12_update_permissions.sh #!/usr/bin/env bash #==============================================================================
Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
Licensed under the Amazon Software License (the "License"). You may not use
this file except in compliance with the License. A copy of the License is
located at
https://aws.amazon.com/asl/
or in the "license" file accompanying this file. This file is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or
implied. See the License for the specific language governing permissions
and limitations under the License.
#==============================================================================
set -xe
EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config
@JessShannon, I am not entirely sure 12_update_permissions.sh is something that's injected by beanstalk. Are permissions updates something your absolutely need to do? Is it possible for you to keep the permissions update separate from your Beanstlak deployment? Laterally, you could also try updating the value of the "Timeout" option setting listed here to 3600s if you have not already? https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options-general.html#command-options-general-autoscalingscheduledaction