Elastic beanstalk Enhanced health not generating healthd/application.log files

3

I have Enhanced health reporting turned on for my Elastic beanstalk environment. The environment is:

  1. Multicontainer docker setup running in “Amazon Linux 2”
  2. It has an nginx proxy (Configuration > Software shows: Log streaming: disabled / Proxy server: nginx / Rotate logs: disabled / X-Ray daemon: disabled)
  3. Enhanced monitoring is on (Configuration > Monitoring shows: CloudWatch Custom Metrics-Environment: CloudWatch Custom Metrics-Instance: / Health event log streaming: disabled / Ignore HTTP 4xx: enabled / Ignore load balancer 4xx: disabled System: Enhanced)

However, on the Health page, none of the requests, response, or latency fields are populating, while load & CPU utilization are populating. It is my understanding that this data is populated from a log file that is written to /var/log/nginx/healthd/, but that directory is empty. It seems like this is a bug or some sort of misconfiguration. Does anyone know why this might be happening?

I included some relevant info from the machine below.


The healthd config file (I commented out the group_id, which is a uuid in the actual file):

$ cat /etc/healthd/config.yaml
group_id: XXXX
log_to_file: true
endpoint: https://elasticbeanstalk-health.us-east-2.amazonaws.com
appstat_log_path: /var/log/nginx/healthd/application.log
appstat_unit: sec
appstat_timestamp_on: completion

The output of the healthd daemon log—showing warnings for not finding previous application.log.YYYY-MM-DD-HH files:

$ head /var/log/healthd/daemon.log
# Logfile created on 2022-04-02 21:02:22 +0000 by logger.rb/66358
A, [2022-04-02T21:02:24.123304 #4122]   ANY -- : healthd daemon 1.0.6 initialized
W, [2022-04-02T21:02:24.266469 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:29.266806 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:34.404332 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:39.406846 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:44.410108 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:49.410342 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:54.410611 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:59.410860 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist

The /var/logs/nginx/ directory with perms and ownership. Is nginx supposed to own healthd?

$ ls -l /var/log/nginx/
total 12
-rw-r--r-- 1 root  root  11493 Apr  4 21:15 access.log
drwxr-xr-x 2 nginx nginx     6 Apr  2 21:01 healthd
drwxr-xr-x 2 root  root      6 Apr  2 21:02 rotated

The empty /var/logs/nginx/healthd/ directory:

$ ls /var/log/nginx/healthd/
# this directory is empty
preguntada hace 3 años2 mil visualizaciones
1 Respuesta
0
Respuesta aceptada

It turns out that even though the Elastic Beanstalk web console is showing my setup as having an nginx proxy, for multicontainer docker + docker compose, it doesn’t use an Elastic Beanstalk proxy, so I needed to have my own nginx container report this information.

There is helpful info in these:

After all that I had my own nginx logging healthd info to NGINXDIR="/var/log/eb-docker/containers/nginx/healthd" and an empty directory HEALTHDDIR="/var/log/nginx/healthd", so I created an .ebextensions script that:

  1. Does mkdir -p "$NGINXDIR" && chmod 777 "$NGINXDIR" (would love to hear advice on a tighter perms and/or user/group owners for that directory)
  2. Sets up a symlinks in the HEALTHDDIR pointing to my NGINXDIR:
      if [ -L "$HEALTHDDIR" ]; then
        echo "is symlink: $HEALTHDDIR"
      else
        mv "$HEALTHDDIR" "$BACKUPDIR"
        echo "moved $HEALTHDDIR to $BACKUPDIR"
        ln -s "$NGINXDIR" "$HEALTHDDIR"
        echo "linked $NGINXDIR to $HEALTHDDIR"
      fi

Seems to be working now and I’m now seeing data in the “Nxx Responses” and “PNN Latency” columns, which are reading from the last 10 seconds and updating regularly.

respondido hace 3 años
profile picture
EXPERTO
revisado hace 7 meses

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas