Elastic beanstalk Enhanced health not generating healthd/application.log files

Question

I have Enhanced health reporting turned on for my Elastic beanstalk environment. The environment  is:

1. Multicontainer docker setup running in “Amazon Linux 2”
2. It has an nginx proxy (Configuration > Software shows: Log streaming: disabled / Proxy server: nginx / Rotate logs: disabled / X-Ray daemon: disabled)
3. Enhanced monitoring is on (Configuration > Monitoring shows: CloudWatch Custom Metrics-Environment: CloudWatch Custom Metrics-Instance: / Health event log streaming: disabled / Ignore HTTP 4xx: enabled / Ignore load balancer 4xx: disabled System: Enhanced)

However, on the Health page, none of the requests, response, or latency fields are populating, while load & CPU utilization are populating. It is my understanding that this data is populated from a log file that is written to `/var/log/nginx/healthd/`, but that directory is empty. It seems like this is a bug or some sort of misconfiguration. Does anyone know why this might be happening?

I included some relevant info from the machine below.

---

The healthd config file (I commented out the `group_id`, which is a uuid in the actual file):

```
$ cat /etc/healthd/config.yaml
group_id: XXXX
log_to_file: true
endpoint: https://elasticbeanstalk-health.us-east-2.amazonaws.com
appstat_log_path: /var/log/nginx/healthd/application.log
appstat_unit: sec
appstat_timestamp_on: completion
```

The output of the healthd daemon log—showing warnings for not finding previous application.log.YYYY-MM-DD-HH files:

```
$ head /var/log/healthd/daemon.log
# Logfile created on 2022-04-02 21:02:22 +0000 by logger.rb/66358
A, [2022-04-02T21:02:24.123304 #4122]   ANY -- : healthd daemon 1.0.6 initialized
W, [2022-04-02T21:02:24.266469 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:29.266806 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:34.404332 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:39.406846 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:44.410108 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:49.410342 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:54.410611 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
W, [2022-04-02T21:02:59.410860 #4122]  WARN -- : log file "/var/log/nginx/healthd/application.log.2022-04-02-21" does not exist
```

The /var/logs/nginx/ directory with perms and ownership. Is `nginx` supposed to own healthd?

```
$ ls -l /var/log/nginx/
total 12
-rw-r--r-- 1 root  root  11493 Apr  4 21:15 access.log
drwxr-xr-x 2 nginx nginx     6 Apr  2 21:01 healthd
drwxr-xr-x 2 root  root      6 Apr  2 21:02 rotated
```

The empty /var/logs/nginx/healthd/ directory:

```
$ ls /var/log/nginx/healthd/
# this directory is empty
```

Accepted Answer

It turns out that even though the Elastic Beanstalk web console is showing my setup as having an nginx proxy, for multicontainer docker + docker compose, it doesn’t use an Elastic Beanstalk proxy, so I needed to have my own nginx container report this information.

There is helpful info in these:
* How to generally have container logs mounted properly: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.container.console.html#docker-env-cfg.healthd-logging
* How to set up your own nginx to log in the right format: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/health-enhanced-serverlogs.html#health-enhanced-serverlogs.configure

After all that I had my own nginx logging healthd info to `NGINXDIR="/var/log/eb-docker/containers/nginx/healthd"` and an empty directory `HEALTHDDIR="/var/log/nginx/healthd"`, so I created an .ebextensions script that:

1. Does `mkdir -p "$NGINXDIR" && chmod 777 "$NGINXDIR"` (would love to hear advice on a tighter perms and/or user/group owners for that directory)
2. Sets up a symlinks in the HEALTHDDIR pointing to my NGINXDIR:

```
      if [ -L "$HEALTHDDIR" ]; then
        echo "is symlink: $HEALTHDDIR"
      else
        mv "$HEALTHDDIR" "$BACKUPDIR"
        echo "moved $HEALTHDDIR to $BACKUPDIR"
        ln -s "$NGINXDIR" "$HEALTHDDIR"
        echo "linked $NGINXDIR to $HEALTHDDIR"
      fi
```

Seems to be working now and I’m now seeing data in the “Nxx Responses” and “PNN Latency” columns, which are reading from the last 10 seconds and updating regularly.

Elastic beanstalk Enhanced health not generating healthd/application.log files

Relevant content