Exclude nginx logs from Cloudwatch log group

Question

Hello,

I have an Elastic Beanstalk Java application and configured application logs to be sent to Cloudwatch using log groups. Now I have eb-engine.log, eb-hooks.log, access.log, error.log, web.stdout.log log groups.

Basically, I only need web.stdout.log to be streamed to Cloudwatch, I do not need others. Thus, I want to exclude them.

What I tried:
1. Adding custom log config to .ebextensions (here https://repost.aws/knowledge-center/elastic-beanstalk-customized-log-files);
2. Installing and configuring cloudwatch agent at /etc/awslogs/awslogs.conf

None of that actually helped, all logs still are being streamed to Cloudwatch. What solution can be here?

Accepted Answer

Hi,

Elastic Beanstalk uses the awslogs agent to stream logs to CloudWatch. If you want to customize which logs are streamed, you'll need to modify the awslogs configuration.

Here are the steps to achieve this:

**1. Understand the Default Configuration:**
Elastic Beanstalk's default setup has a set of predefined logs it streams to CloudWatch. These configurations are usually stored in the `/etc/awslogs/config/` directory. There might be a file named `elasticbeanstalk.conf` or similar that contains the log stream configurations.

**2. Create a Custom Configuration in `.ebextensions`:**
You can override the default configuration using `.ebextensions`. Here's a sample configuration that only streams `web.stdout.log`:
```
files:
  "/etc/awslogs/config/web-stdout.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      [/var/log/nginx/web.stdout.log]
      datetime_format = %Y-%m-%dT%H:%M:%S.%fZ
      file = /var/log/nginx/web.stdout.log
      buffer_duration = 5000
      log_stream_name = {instance_id}
      initial_position = start_of_file
      log_group_name = your-log-group-name
```
Replace `your-log-group-name` with the desired CloudWatch log group name.

**3. Remove Default Configuration:**
You can also add commands in your `.ebextensions` to delete the default configuration files to ensure they don't interfere:
```
commands:
  01_remove_default_log_configs:
    command: "rm -f /etc/awslogs/config/*.conf"
```

**4. Restart the` awslogs` Agent:**
After modifying the configuration, you should restart the `awslogs` agent to pick up the changes. You can add this to your `.ebextensions`:

**5. Combine Everything:**
Combine all of the above configurations into a single `.config` file in the `.ebextensions` directory of your Elastic Beanstalk application source bundle.
```
files:
  "/etc/awslogs/config/web-stdout.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      [/var/log/nginx/web.stdout.log]
      datetime_format = %Y-%m-%dT%H:%M:%S.%fZ
      file = /var/log/nginx/web.stdout.log
      buffer_duration = 5000
      log_stream_name = {instance_id}
      initial_position = start_of_file
      log_group_name = your-log-group-name

commands:
  01_remove_default_log_configs:
    command: "rm -f /etc/awslogs/config/*.conf"
  02_restart_awslogs:
    command: "service awslogs restart"
```

**6. Deploy the Updated Configuration:**
Once you've added the `.config` file to your `.ebextensions` directory, re-deploy your Elastic Beanstalk application. This will apply the changes and only web.stdout.log should be streamed to CloudWatch.

Remember to monitor your application closely after making these changes to ensure everything is working as expected.

Exclude nginx logs from Cloudwatch log group

Relevant content