Elastic beanstalk Enhanced health not generating healthd/application.log files
I have Enhanced health reporting turned on for my Elastic beanstalk environment. The environment is:
- Multicontainer docker setup running in “Amazon Linux 2”
- It has an nginx proxy (Configuration > Software shows: Log streaming: disabled / Proxy server: nginx / Rotate logs: disabled / X-Ray daemon: disabled)
- Enhanced monitoring is on (Configuration > Monitoring shows: CloudWatch Custom Metrics-Environment: CloudWatch Custom Metrics-Instance: / Health event log streaming: disabled / Ignore HTTP 4xx: enabled / Ignore load balancer 4xx: disabled System: Enhanced)
However, on the Health page, none of the requests, response, or latency fields are populating, while load & CPU utilization are populating. It is my understanding that this data is populated from a log file that is written to
/var/log/nginx/healthd/, but that directory is empty. It seems like this is a bug or some sort of misconfiguration. Does anyone know why this might be happening?
I included some relevant info from the machine below.
The healthd config file (I commented out the
group_id, which is a uuid in the actual file):
$ cat /etc/healthd/config.yaml group_id: XXXX log_to_file: true endpoint: https://elasticbeanstalk-health.us-east-2.amazonaws.com appstat_log_path: /var/log/nginx/healthd/application.log appstat_unit: sec appstat_timestamp_on: completion
The output of the healthd daemon log—showing warnings for not finding previous application.log.YYYY-MM-DD-HH files:
$ head /var/log/healthd/daemon.log # Logfile created on 2022-04-02 21:02:22 +0000 by logger.rb/66358 A, [2022-04-02T21:02:24.123304 #4122] ANY -- : healthd daemon 1.0.6 initialized W, [2022-04-02T21:02:24.266469 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:29.266806 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:34.404332 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:39.406846 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:44.410108 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:49.410342 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:54.410611 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist W, [2022-04-02T21:02:59.410860 #4122] WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
The /var/logs/nginx/ directory with perms and ownership. Is
nginx supposed to own healthd?
$ ls -l /var/log/nginx/ total 12 -rw-r--r-- 1 root root 11493 Apr 4 21:15 access.log drwxr-xr-x 2 nginx nginx 6 Apr 2 21:01 healthd drwxr-xr-x 2 root root 6 Apr 2 21:02 rotated
The empty /var/logs/nginx/healthd/ directory:
$ ls /var/log/nginx/healthd/ # this directory is empty
It turns out that even though the Elastic Beanstalk web console is showing my setup as having an nginx proxy, for multicontainer docker + docker compose, it doesn’t use an Elastic Beanstalk proxy, so I needed to have my own nginx container report this information.
There is helpful info in these:
- How to generally have container logs mounted properly: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.container.console.html#docker-env-cfg.healthd-logging
- How to set up your own nginx to log in the right format: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/health-enhanced-serverlogs.html#health-enhanced-serverlogs.configure
After all that I had my own nginx logging healthd info to
NGINXDIR="/var/log/eb-docker/containers/nginx/healthd" and an empty directory
HEALTHDDIR="/var/log/nginx/healthd", so I created an .ebextensions script that:
mkdir -p "$NGINXDIR" && chmod 777 "$NGINXDIR"(would love to hear advice on a tighter perms and/or user/group owners for that directory)
- Sets up a symlinks in the HEALTHDDIR pointing to my NGINXDIR:
if [ -L "$HEALTHDDIR" ]; then echo "is symlink: $HEALTHDDIR" else mv "$HEALTHDDIR" "$BACKUPDIR" echo "moved $HEALTHDDIR to $BACKUPDIR" ln -s "$NGINXDIR" "$HEALTHDDIR" echo "linked $NGINXDIR to $HEALTHDDIR" fi
Seems to be working now and I’m now seeing data in the “Nxx Responses” and “PNN Latency” columns, which are reading from the last 10 seconds and updating regularly.
Unable to delete my Elastic Beanstalk environmentasked 3 years ago
When I go to my site no error but my bean stalk environment has errorsasked 6 months ago
Elastic beanstalk Enhanced health not generating healthd/application.log filesAccepted Answerasked 3 months ago
elastic beanstalkasked 3 months ago
Elastic beanstalk next.js application's health status become warningasked 2 months ago
Elastic Beanstalk health is in Severeasked 5 months ago
AWS Elastic Beanstalk Running in Private VPC without internet accessasked 3 months ago
Elastic Beanstalk environment does not terminateasked 3 years ago
Elastic Beanstalk Error deploy python flask application sqlite3asked 18 days ago
Elastic BeanStalk shows unhealthy; health check not workingasked 15 days ago