By using AWS re:Post, you agree to the Terms of Use
/Elastic beanstalk Enhanced health not generating healthd/application.log files/

Elastic beanstalk Enhanced health not generating healthd/application.log files

3

I have Enhanced health reporting turned on for my Elastic beanstalk environment. The environment is:

  1. Multicontainer docker setup running in “Amazon Linux 2”
  2. It has an nginx proxy (Configuration > Software shows: Log streaming: disabled / Proxy server: nginx / Rotate logs: disabled / X-Ray daemon: disabled)
  3. Enhanced monitoring is on (Configuration > Monitoring shows: CloudWatch Custom Metrics-Environment: CloudWatch Custom Metrics-Instance: / Health event log streaming: disabled / Ignore HTTP 4xx: enabled / Ignore load balancer 4xx: disabled System: Enhanced)

However, on the Health page, none of the requests, response, or latency fields are populating, while load & CPU utilization are populating. It is my understanding that this data is populated from a log file that is written to /var/log/nginx/healthd/, but that directory is empty. It seems like this is a bug or some sort of misconfiguration. Does anyone know why this might be happening?

I included some relevant info from the machine below.


The healthd config file (I commented out the group_id, which is a uuid in the actual file):

$ cat /etc/healthd/config.yaml
group_id: XXXX
log_to_file: true
endpoint: https://elasticbeanstalk-health.us-east-2.amazonaws.com
appstat_log_path: /var/log/nginx/healthd/application.log
appstat_unit: sec
appstat_timestamp_on: completion

The output of the healthd daemon log—showing warnings for not finding previous application.log.YYYY-MM-DD-HH files:

$ head /var/log/healthd/daemon.log
# Logfile created on 2022-04-02 21:02:22 +0000 by logger.rb/66358
A, [2022-04-02T21:02:24.123304 #4122]   ANY -- : healthd daemon 1.0.6 initialized
W, [2022-04-02T21:02:24.266469 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:29.266806 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:34.404332 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:39.406846 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:44.410108 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:49.410342 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:54.410611 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:59.410860 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist

The /var/logs/nginx/ directory with perms and ownership. Is nginx supposed to own healthd?

$ ls -l /var/log/nginx/
total 12
-rw-r--r-- 1 root  root  11493 Apr  4 21:15 access.log
drwxr-xr-x 2 nginx nginx     6 Apr  2 21:01 healthd
drwxr-xr-x 2 root  root      6 Apr  2 21:02 rotated

The empty /var/logs/nginx/healthd/ directory:

$ ls /var/log/nginx/healthd/
# this directory is empty
asked 3 months ago74 views
1 Answers
0
Accepted Answer

It turns out that even though the Elastic Beanstalk web console is showing my setup as having an nginx proxy, for multicontainer docker + docker compose, it doesn’t use an Elastic Beanstalk proxy, so I needed to have my own nginx container report this information.

There is helpful info in these:

After all that I had my own nginx logging healthd info to NGINXDIR="/var/log/eb-docker/containers/nginx/healthd" and an empty directory HEALTHDDIR="/var/log/nginx/healthd", so I created an .ebextensions script that:

  1. Does mkdir -p "$NGINXDIR" && chmod 777 "$NGINXDIR" (would love to hear advice on a tighter perms and/or user/group owners for that directory)
  2. Sets up a symlinks in the HEALTHDDIR pointing to my NGINXDIR:
      if [ -L "$HEALTHDDIR" ]; then
        echo "is symlink: $HEALTHDDIR"
      else
        mv "$HEALTHDDIR" "$BACKUPDIR"
        echo "moved $HEALTHDDIR to $BACKUPDIR"
        ln -s "$NGINXDIR" "$HEALTHDDIR"
        echo "linked $NGINXDIR to $HEALTHDDIR"
      fi

Seems to be working now and I’m now seeing data in the “Nxx Responses” and “PNN Latency” columns, which are reading from the last 10 seconds and updating regularly.

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions