Logging EC2 Instances Bash Commands with CloudWatch Logs

In today's increasingly complex threat landscape, effective security monitoring of your EC2 instances is not just beneficial—it's essential. Security teams need real-time visibility into system activities, unauthorized access attempts, and potential security incidents. Amazon CloudWatch provides robust capabilities to collect security-relevant logs and metrics, monitor for suspicious patterns, and trigger automated responses to security events. This article demonstrates how to configure the CloudWatch agent to strengthen your EC2 security posture through logging of bash commands.

Prerequisites

• An AWS account with appropriate permissions
• One or more running EC2 instances
• SSH access to your instances

Installing and Configuring the CloudWatch Agent

Step 1: Install the CloudWatch Agent

First, connect to your EC2 instance via SSH and install the CloudWatch agent:

sudo yum install amazon-cloudwatch-agent

This command installs the CloudWatch agent package on your Amazon Linux instance.

Step 2: Install collectd for Enhanced Metrics (Optional)

For enhanced metric collection, install the collectd package:

sudo yum install collectd

The collectd daemon collects system and application metrics that the CloudWatch agent can then send to CloudWatch.

Step 3: Configure the CloudWatch Agent

Run the configuration wizard to set up the agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

The wizard will guide you through a series of questions to create a configuration file tailored to your monitoring needs. You can specify:

• Which metrics to collect
• Which log files to monitor
• How frequently to collect data
• Which AWS region to use

To export Bash commands for all users to CloudWatch logs, use the file path /home/*/.bash_history

After completing the wizard, the configuration file will be saved at:

/opt/aws/amazon-cloudwatch-agent/bin/config.json

Step 4: Start and Enable the CloudWatch Agent

Start the CloudWatch agent with your configuration:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json

This command:

• -a fetch-config: Specifies the action to fetch the configuration
• -m ec2: Indicates the agent is running on an EC2 instance
• -s: Starts the agent after fetching the configuration
• -c file:/path/to/config.json: Specifies the configuration file location

To ensure the agent starts automatically on system boot:

sudo systemctl enable amazon-cloudwatch-agent

You can verify the agent is running with:

sudo systemctl status amazon-cloudwatch-agent

Troubleshooting

If you encounter issues with the CloudWatch agent, check these log files:

cat /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

This log contains detailed information about the agent's operation, including any errors encountered during startup or while collecting metrics.

For configuration details, examine:

cat /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml

This file shows the active configuration in TOML format, which is converted from your JSON configuration.

Best Practices

1. Filters: Use 'filters' field to exclude sensitive log entries.. Refer to [1] for config file fields options.
2. Use IAM Roles: Attach an IAM role with appropriate CloudWatch permissions to your EC2 instances.
3. Log Rotation: Implement log rotation to prevent the CloudWatch agent logs from consuming too much disk space.

Conclusion

Setting up the CloudWatch agent on your EC2 instances provides valuable insights into system activiies and performance. By following this guide, you can export custom log files such as bash commands logs.

The CloudWatch agent's flexibility allows you to customize monitoring to your specific needs, whether you're running a small development environment or a large-scale production system.

[1] https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html

[2] Sample config file:

[ec2-user]$ cat /opt/aws/amazon-cloudwatch-agent/bin/config.json
{
        "agent": {
                "metrics_collection_interval": 60,
                "run_as_user": "root"
        },
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/home/*/.bash_history",
                                                "log_group_class": "STANDARD",
                                                "log_group_name": ".bash_history",
                                                "log_stream_name": "{instance_id}",
                                                "retention_in_days": -1
                                        }
                                ]
                        }
                }
        },
        "metrics": {
                "aggregation_dimensions": [
                        [
                                "InstanceId"
                        ]
                ],
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "ImageId": "${aws:ImageId}",
                        "InstanceId": "${aws:InstanceId}",
                        "InstanceType": "${aws:InstanceType}"
                },
                "metrics_collected": {
                        "collectd": {
                                "metrics_aggregation_interval": 60
                        },
                        "disk": {
                                "measurement": [
                                        "used_percent"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        },
                        "statsd": {
                                "metrics_aggregation_interval": 60,
                                "metrics_collection_interval": 10,
                                "service_address": ":8125"
                        }
                }
        },
        "traces": {
                "buffer_size_mb": 3,
                "concurrency": 8,
                "insecure": false,
                "traces_collected": {
                        "xray": {
                                "bind_address": "127.0.0.1:2000",
                                "tcp_proxy": {
                                        "bind_address": "127.0.0.1:2000"
                                }
                        }
                }
        }
}