How to get cloudwatch agent working on ECS with terraform

0

I'm doing my best to follow these instructions. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Generation_CloudWatch_Agent.html

I've added a container definition to my terraform

resource "aws_ecs_task_definition" "taskdef" {
    family = "my_service${var.suffix}_service"
    container_definitions = jsonencode([
        {
            assign_public_ip = true
            name = "cloudwatch-agent"
            image = "public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest"
            memory = 256
            cpu = 256
            portMappings = [
                {
                    protocol = "tcp"
                    containerPort = 25888
                },
                {
                    protocol = "udp"
                    containerPort = 25888
                }
            ]
            logConfiguration = {
                logDriver = "awslogs"
                options = {
                    awslogs-create-group = "true"
                    awslogs-group = "/aws/ecs/my-log-group"
                    awslogs-region = "us-west-2"
                    awslogs-stream-prefix = "ecs"
                }
            }
            Environment = [
                {"name": "CW_CONFIG_CONTENT", "value": "{\"logs\": { \"metrics_collected\": { \"emf\": { }}}}"}
            ]
        },
        {
            assign_public_ip = true
            name = "my-service${var.suffix}container"
            cpu = 1024
            memory = 2048
            image = "${var.image-name}@${var.image-hash}"
            portMappings = [
                {
                    name = "http"
                    containerPort = 80
                    hostPort = 80
                    protocol = "http"
                }
            ]
            logConfiguration = {
                logDriver = "awslogs"
                options = {
                    awslogs-create-group = "true"
                    awslogs-group = "/aws/ecs/my-log-group"
                    awslogs-region = "us-west-2"
                    awslogs-stream-prefix = "ecs"
                }
            }
            health_check = {

            }
            Environment = [
                {"name": "CLOUDWATCH_LOG_GROUP", "value": aws_cloudwatch_log_group.metrics.name},   
                {"name": "AWS_EMF_AGENT_ENDPOINT", "value": "tcp://127.0.0.1:25888"},
                {"name": "METRICS_NAMESPACE", "value": "athena-scanrunner${var.suffix}-metrics"}
            ]
        }
    ])
    cpu = 2048
    execution_role_arn = var.service_role
    task_role_arn = var.service_role
    memory = 4096
    network_mode = "awsvpc"
    requires_compatibilities = ["FARGATE"]
    runtime_platform {
        cpu_architecture = "ARM64"
        operating_system_family = "LINUX"
    }
}

I couldn't get the valueFrom field to work so I used JSON directly for the config.

In my code I've added the config

Amazon.CloudWatch.EMF.Config.EnvironmentConfigurationProvider.Config =
        new Amazon.CloudWatch.EMF.Config.Configuration
        {
            ServiceName = "Athena-ScanRunner",
            ServiceType = "WebApi",
            LogGroupName = Environment.GetEnvironmentVariable("CLOUDWATCH_LOG_GROUP"),
            AgentEndPoint = "tcp://127.0.0.1:25888",
            EnvironmentOverride = Amazon.CloudWatch.EMF.Environment.Environments.ECS
        };


    var builder = WebApplication.CreateBuilder(args);
    builder.Configuration.GetAWSOptions();
    builder.Services.AddControllers().AddJsonOptions(options =>
    {
        options.JsonSerializerOptions.Converters.Add(new JsonStringEnumConverter());

        options.JsonSerializerOptions.DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull;
    });
    builder.Services.AddHealthChecks();
    builder.Services.AddAWSService<IAmazonDynamoDB>();
    builder.Services.AddAWSService<IAmazonECS>();
    builder.Services.AddAWSService<IAmazonS3>();
    builder.Services.AddAWSService<IAmazonSQS>();
    builder.Services.AddEmf();
    builder.Services.AddSingleton<IScanTaskDb, ScanTaskDynamoDb>();
    builder.Services.AddSingleton<ITaskQueue, SqsTaskQueue>();
    builder.Services.AddSingleton<IFileStore, S3FileStore>();


    var app = builder.Build();

    app.UseHealthChecks("/");
    app.UseEmfMiddleware();

and I've added the metrics to each api endpoint.

I get no metrics flowing. How do I make this setup work?

1 Answer
1

Unfortunately you haven’t shared the full VPC setup or task role definition. However.

Do you have the ecs task running in a public subnet?

Does the task have the correct IAM permissions to create log groups and streams?

Also you should place your CW Agent Config in an SSM parameter and let the container read it in as an environment variable.

profile picture
EXPERT
answered 16 days ago
profile picture
EXPERT
reviewed 16 days ago
  • Yeah public subnet, you can also see the assigned public IP.

    Permissions look correct, I'm getting no errors and normal logs work correctly.

    Also you should place your CW Agent Config in an SSM parameter and let the container read it in as an environment variable.

    valueFrom doesn't work with terraform or entering the JSON in the task definition editor. I can't do this if the mechanism specified in the docs just silently fails.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions