[BUG] Possible bug - not all logs of the AWS EB application appear in the CW web.stdout.log

Question

Bug Report:

**Description**: Not all logs of the deployed AWS Beanstalk application appear in the corresponding log group (`/var/log/web.stdout.log`) in AWS CloudWatch (nor in `/var/log/web.stdout.log` on the EC2 instance).

**Steps to Reproduce**:
1. Deploy the demo application to AWS Beanstalk.
2. Check the corresponding log group (`/var/log/web.stdout.log`) in AWS CloudWatch.
3. Some logs are missing.

**Expected Result**: All logs of the deployed application should appear in the corresponding log group (`/var/log/web.stdout.log`) in AWS CloudWatch.

**Actual Result**: Not all logs of the deployed application appear in the corresponding log group (`/var/log/web.stdout.log`) in AWS CloudWatch.

**AWS Beanstalk Environment Configuration**:
- Environment type: Single instance (also tried with multiple instances)
- Platform: Corretto 8 running on 64bit Amazon Linux 2/3.4.6 (also tested with a couple of earlier versions)
- Instance type: t3.micro (also tested on t2.medium and t2.small)
- Log streaming to CloudWatch enabled

**AWS CloudWatch Log Group**:
- Log group name: `/var/log/web.stdout.log`

**Used workaround**: None.

**Severity**: High

**Additional Details (bug reproduction)**: The demo application is a simple java application for writing 100000 log lines to stdout from 10 threads (10000 logs per thread). When the application is deployed to AWS Beanstalk and the logs are checked in AWS CloudWatch, not all logs appear in the corresponding log group (`/var/log/web.stdout.log`).

**Demo Application**:
- Repository link: https://github.com/grigoart/aws-eb-log-bug/
- Build jar by running `gradle jar` or use `application.jar` from repository (java version "1.8.0_151")
- Running locally:
  - `java -jar application.jar > log.txt`
  - wait for ~1 min and stop the application
  - `wc -l log.txt`
  - output: "100000 log.txt" (as expected)
- Running in AWS EB Environment:
  - deploy jar
  - wait for ~1 min
  - verify that AWS CW log group `/var/log/web.stdout.log` does not have all the logs (e.g. all numbers >1000 are missing). According to AWS CW log insights there are only 1000 log entries (see the image below)
  - alternatively:
    - connect to EC2 instance using SSH
	- `wc -l /var/log/web.stdout.log`
	- output: "2360 /var/log/web.stdout.log" (unexpected result, should be >= 100000)

![AWS CW Logs Insights](/media/postImages/original/IMt0LymCNEQ92-hAj-deaJKg)

Accepted Answer

Got the answer from tech support:

> The rate-limiting errors were observed in “/var/log/messages” file.

> ... the rate-limiting signals were being sent by “journald” service.

Solved by configuring "rsyslog” and “journald” services through `.ebextensions` (workaround solution provided by tech support):

```yaml
files:
    /tmp/Test.txt:
        mode: "000644"
        owner: root
        group: root
        content: |
            $SystemLogRateLimitInterval 0
            $SystemLogRateLimitBurst    0
            $ImjournalRateLimitInterval 0

/tmp/script.sh:
        mode: "000755"
        owner: root
        group: root
        content: |
            #!/bin/bash
            #Fetching grep values for if condition to check whether the update has been performed or not.
            value=$(grep -f /tmp/Test.txt /etc/rsyslog.conf)
            value1=$(grep -w 'RateLimitInterval=0' /etc/systemd/journald.conf)
            value2=$(grep -w 'RateLimitBurst=0' /etc/systemd/journald.conf)

# Updating /etc/rsyslog.conf file
            if [ ! -z "$value" ]
            then
            echo "Match Found"
            else
            echo "Modifying /etc/rsyslog.conf"
            echo -e '$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst    0
$ImjournalRateLimitInterval 0' >> /etc/rsyslog.conf
            fi

# Updating /etc/systemd/journald.conf file
            if [ ! -z $value1 ]
            then
            echo "Match Found for RateLimitInterval"
            else
            echo "Adding RateLimitInterval=0 to /etc/systemd/journald.conf"
            echo -e "RateLimitInterval=0" >> /etc/systemd/journald.conf
            fi

if [ ! -z $value2 ]
            then
            echo "Match Found for RateLimitBurst"
            else
            echo "Adding RateLimitBurst=0 to /etc/systemd/journald.conf"
            echo -e "RateLimitBurst=0" >> /etc/systemd/journald.conf
            fi

commands:
    01_run_script.sh: 
        command: ./tmp/script.sh 
    02_restart_journald:
        command: systemctl restart systemd-journald
    03_restart_rsyslog:
        command: systemctl restart rsyslog
```

Answer

Did you enable the streaming environment's logs to the CloudWath Logs ?  You can do it using Console (for detailed steps how to achieve this please check [2]), using EB CLI [3] or using configuration files [4]. After completing these steps you should be able to see these CloudWatch LogGroups and the environment will start streaming appropriate logs into them.

[1] Enabling Elastic Beanstalk enhanced health reporting https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/health-enhanced-enable.html 
[2] Using Elastic Beanstalk with Amazon CloudWatch Logs https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html 
[3] Instance log streaming using the Elastic Beanstalk console
 https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html#AWSHowTo.cloudwatchlogs.streaming.console 
[4] Instance log streaming using the EB CLI https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html#AWSHowTo.cloudwatchlogs.streaming.ebcli 
[5] Instance log streaming using configuration files
 https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html#AWSHowTo.cloudwatchlogs.files

[BUG] Possible bug - not all logs of the AWS EB application appear in the CW web.stdout.log

Relevant content