Automating CloudWatch Alarms

Question

I'm working on automating CloudWatch alarms. I got most of it to work, but now the logs aren't showing up on my CloudWatch dashboard. How do I get it to show the logs in my dashboard and CloudWatch Insights? I also need the templates to deploy to my dashboard instead of creating one every time an instance is spun up. Would this also fix my logging problem? The article I'm using to guide me through the process is below.

https://aws.amazon.com/blogs/mt/automate-creation-of-amazon-cloudwatch-alarms-and-dashboards-with-aws-systems-manager-and-ansible/

re:Post Agent · Accepted Answer

To get your logs showing up in your CloudWatch dashboard, you need to ensure that you've properly configured log output to CloudWatch Logs. For AWS Systems Manager Automation actions, you can enable this by:

1. Opening the AWS Systems Manager console
2. Navigating to the Automation section
3. Selecting the Preferences tab and clicking Edit
4. Checking the "Send output to CloudWatch Logs" option
5. Optionally enabling encryption for your log data

For your dashboard issue, CloudWatch dashboards can be created and customized to include both metrics and logs. You can use CloudWatch Logs Insights to create dashboard widgets that monitor specific log patterns and display them alongside your metrics.

Regarding templates for your dashboards, you have a couple of options:

1. Create a template dashboard that includes all the widgets and configurations you need
2. Use infrastructure as code tools (like CloudFormation or Terraform) to define your dashboards
3. Programmatically create and update dashboards using the AWS SDK or CLI

Using templates would help with your logging problem if the issue is related to dashboard configuration, but it wouldn't fix underlying logging configuration issues. Make sure your resources are properly configured to send logs to CloudWatch Logs first, then focus on how those logs are displayed in your dashboards.

For automated alarm creation, you can use CloudWatch's built-in alarm recommendations or create custom alarms based on specific metrics, logs, or metric math expressions. These can be templated and deployed automatically as part of your instance provisioning process.  
**Sources**  
[Logging Automation action output with CloudWatch Logs - AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-action-logging.html)  
[CloudWatch alarms and dashboards - AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/amazon-rds-monitoring-alerting/cloudwatch-dashboards.html)  
[Elevating Your AWS Observability: Unlocking the Power of Amazon CloudWatch Alarms | AWS Cloud Operations Blog](https://aws.amazon.com/blogs/mt/elevating-your-aws-observability-unlocking-the-power-of-amazon-cloudwatch-alarms/)

Nihar · Answer

Hello Eric,

To use multiple log groups in CloudWatch, here's how you can set it up:

1. First, define multiple log groups in your configuration:
	```yaml
	# In your CloudFormation template or similar IaC
	LogGroups:
	  - LogGroupName: /aws/application1
	    RetentionInDays: 30
	  - LogGroupName: /aws/application2
	    RetentionInDays: 14
	```

2. Configure the CloudWatch agent to send logs to different groups:
	```json
	{
	  "agent": {
	    "run_as_user": "root"
	  },
	  "logs": {
	    "logs_collected": {
	      "files": {
	        "collect_list": [
	          {
	            "file_path": "/var/log/application1/*.log",
	            "log_group_name": "/aws/application1",
	            "log_stream_name": "{instance_id}-application1"
	          },
	          {
	            "file_path": "/var/log/application2/*.log",
	            "log_group_name": "/aws/application2",
	            "log_stream_name": "{instance_id}-application2"
	          }
	        ]
	      }
	    }
	  }
	}
	```

3. Store this configuration in Systems Manager Parameter Store:
	```bash
	aws ssm put-parameter \
	    --name "AmazonCloudWatch-Config" \
	    --type "String" \
	    --value file://config.json
	```

4. Install and configure the CloudWatch agent on your instances:
	```bash
	# Install the agent
	yum install -y amazon-cloudwatch-agent

# Configure using the SSM parameter
	/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
	    -a fetch-config \
	    -m ec2 \
	    -s \
	    -c ssm:AmazonCloudWatch-Config
	```

5. Update your dashboard to include widgets from multiple log groups:
	```json
	{
	    "widgets": [
	        {
	            "type": "log",
	            "properties": {
	                "query": "SOURCE '/aws/application1' | fields @timestamp, @message",
	                "region": "us-east-1",
	                "title": "Application 1 Logs"
	            }
	        },
	        {
	            "type": "log",
	            "properties": {
	                "query": "SOURCE '/aws/application2' | fields @timestamp, @message",
	                "region": "us-east-1",
	                "title": "Application 2 Logs"
	            }
	        }
	    ]
	}
	```

**Remember to:**
- Ensure proper IAM permissions for accessing multiple log groups
- Update retention policies as needed
- Consider costs when collecting logs from multiple sources
- Use log stream names that make sense for your use case

---

*I hope this is helpful. If yes, please upvote/accept my answer. Thank you.*

Automating CloudWatch Alarms

Add your answer

Relevant content