Cloudwatch log_stream_name {instance_id} is sending over hostname instead

0

Hi,

I am trying to set up CloudWatch log streaming for custom logs for a multi-container application running on elastic beanstalk.

The issue that I am facing is when the log stream is sent over from an instance instead of being the instance id such as i-123456789a1234567 it is sending the hostname which is like ip-00-0-000-000.eu-west-1.compute.internal

Following the documentation here https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html#AWSHowTo.cloudwatchlogs.loggroups under Custom log file streaming

and the following configuration block (found on the elastic beanstalk samples - https://github.com/awsdocs/elastic-beanstalk-samples/blob/master/configuration-files/aws-provided/instance-configuration/logs-streamtocloudwatch-linux.config)

packages:
  yum:
    awslogs: []

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`

  "/etc/awslogs/awslogs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [general]
      state_file = /var/lib/awslogs/agent-state

  "/etc/awslogs/config/logs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/log/messages]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/messages"]]}`
      log_stream_name = {instance_id}
      file = /var/log/messages

      [/var/log/dmesg]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/dmesg"]]}`
      log_stream_name = {instance_id}
      file = /var/log/dmesg

commands:
  "01":
    command: systemctl enable awslogsd.service
  "02":
    command: systemctl restart awslogsd

Any ideas on how I can get this to stream through the reference of the EC2 instance rather than the hostname?

Thanks

1 Answer
1

Hi,

My guess is cloudinit is installing old version of Cloudwatch Agent

packages:
  yum:
    awslogs: []

According to the documentation, the older CloudWatch Logs agent doesn't support Instance Metadata Service Version 2 (IMDSv2). https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html

The name of the new package should be amazon-cloudwatch-agent. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-EC2-Instance.html

I'm not using EB, so I don't know how it's work but to install the new CW Agent on EC2 instance, I usually follow these steps https://repost.aws/knowledge-center/cloudwatch-push-metrics-unified-agent

profile picture
Donov
answered a year ago
  • Thanks very much for your response!

    I think you are right that for some reason it is trying to use the old CW Logs Agent, I tried updating the configuration of the yum install to match the newer package name.

    packages:
      yum:
       amazon-cloudwatch-agent: []
    

    This didn't work however but it did match the latest release on the Github release for the package https://github.com/aws/amazon-cloudwatch-agent/releases/

    I then followed this example from the AWSDocs Github

    which used a container command to start the Cloudwatch agent in a different way

      "02_start_cloudwatch_agent": 
        command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
    

    I think the next steps are to try following these instructions from the link you provided https://repost.aws/knowledge-center/cloudwatch-push-metrics-unified-agent on a single EC2 instance running inside Elastic Beanstalk, if it works then try and automate it into the .ebextensions

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions