Exclude nginx logs from Cloudwatch log group

0

Hello,

I have an Elastic Beanstalk Java application and configured application logs to be sent to Cloudwatch using log groups. Now I have eb-engine.log, eb-hooks.log, access.log, error.log, web.stdout.log log groups.

Basically, I only need web.stdout.log to be streamed to Cloudwatch, I do not need others. Thus, I want to exclude them.

What I tried:

  1. Adding custom log config to .ebextensions (here https://repost.aws/knowledge-center/elastic-beanstalk-customized-log-files);
  2. Installing and configuring cloudwatch agent at /etc/awslogs/awslogs.conf

None of that actually helped, all logs still are being streamed to Cloudwatch. What solution can be here?

1 Answer
0
Accepted Answer

Hi,

Elastic Beanstalk uses the awslogs agent to stream logs to CloudWatch. If you want to customize which logs are streamed, you'll need to modify the awslogs configuration.

Here are the steps to achieve this:

1. Understand the Default Configuration: Elastic Beanstalk's default setup has a set of predefined logs it streams to CloudWatch. These configurations are usually stored in the /etc/awslogs/config/ directory. There might be a file named elasticbeanstalk.conf or similar that contains the log stream configurations.

2. Create a Custom Configuration in .ebextensions: You can override the default configuration using .ebextensions. Here's a sample configuration that only streams web.stdout.log:

files:
  "/etc/awslogs/config/web-stdout.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      [/var/log/nginx/web.stdout.log]
      datetime_format = %Y-%m-%dT%H:%M:%S.%fZ
      file = /var/log/nginx/web.stdout.log
      buffer_duration = 5000
      log_stream_name = {instance_id}
      initial_position = start_of_file
      log_group_name = your-log-group-name

Replace your-log-group-name with the desired CloudWatch log group name.

3. Remove Default Configuration: You can also add commands in your .ebextensions to delete the default configuration files to ensure they don't interfere:

commands:
  01_remove_default_log_configs:
    command: "rm -f /etc/awslogs/config/*.conf"

4. Restart the awslogs Agent: After modifying the configuration, you should restart the awslogs agent to pick up the changes. You can add this to your .ebextensions:

5. Combine Everything: Combine all of the above configurations into a single .config file in the .ebextensions directory of your Elastic Beanstalk application source bundle.

files:
  "/etc/awslogs/config/web-stdout.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      [/var/log/nginx/web.stdout.log]
      datetime_format = %Y-%m-%dT%H:%M:%S.%fZ
      file = /var/log/nginx/web.stdout.log
      buffer_duration = 5000
      log_stream_name = {instance_id}
      initial_position = start_of_file
      log_group_name = your-log-group-name

commands:
  01_remove_default_log_configs:
    command: "rm -f /etc/awslogs/config/*.conf"
  02_restart_awslogs:
    command: "service awslogs restart"

6. Deploy the Updated Configuration: Once you've added the .config file to your .ebextensions directory, re-deploy your Elastic Beanstalk application. This will apply the changes and only web.stdout.log should be streamed to CloudWatch.

Remember to monitor your application closely after making these changes to ensure everything is working as expected.

profile picture
answered 9 months ago
profile pictureAWS
EXPERT
reviewed 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions