Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

You are not onboarded to Container Insights with enhanced observability for Amazon ECS

0

Hello, I have an ECS cluster & I have enabled "Container Insights with enhanced observability" on a cluster level. I didn't enable it at an account level as described here. I don't think that enabling it at account level is necessary.

enabled_cluster

My task execution role has necessary permissions, more than what is required.

role_permission

I can see that my tasks are failing & the reason for their failure too.

task_fails task_fail_reason

Yet, I can't see any logs in Container Insights dashboard. I have failed tasks but there are no logs in the performance dashboard & I see this message on top.

not_onboarded

Does this mean that ECS cluster is not being observed by Container Insights? I am relying on Container Insights enhanced observability to give me logs when containers fail to start or fail for any reason like memory, cpu constraint etc.

Thanks for your help in advance.

Topics
Management & GovernanceNetworking & Content DeliveryContainers
Tags
Amazon CloudWatchAmazon Elastic Container ServiceContainers
Language
English
asked 9 months ago833 views
2 Answers
2

Hello.

I think that no logs are recorded unless the tasks in the ECS cluster are started successfully.
Therefore, in order to resolve the error that the log group does not exist, please manually create the log group specified in the task definition and then try starting the task.
If the container starts successfully, I think Container Insights will start recording.
https://repost.aws/ja/knowledge-center/ecs-resource-initialization-error

EXPERT
answered 9 months ago
EXPERT
reviewed 9 months ago
  • rePost-User
    9 months ago

    Thanks, I have got the logs sorted which brings me to the question, how to capture events where task failed to start or the container didn't start for some reason, example resource constraint? I can't seem to find anything in logs but under Events in ECS console, I do see "service XYZ deployment failed: tasks failed to start."

  • Riku_Kobayashi EXPERT
    9 months ago

    If you are using an ECS service, you can create an EventBridge rule as shown below to notify the cause when a task fails to start. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_service_events.html

    {
    "source": [
        "aws.ecs"
    ],
    "detail-type": [
        "ECS Service Action"
    ],
    "detail": {
        "eventType": [
            "WARN",
            "ERROR"
        ]
    }
}

    In addition, there is a setting that outputs the cause to CloudWatch Logs when an ECS task provided by AWS stops, so you may want to check that. https://github.com/aws-samples/amazon-ecs-stopped-tasks-cwlogs

1

Based on the information provided, it appears that your ECS cluster is not fully onboarded to Container Insights with enhanced observability, despite enabling it at the cluster level. Here are a few points to consider:

  1. While enabling Container Insights with enhanced observability at the cluster level is a valid approach, it's possible that not all components are properly configured for full functionality.

  2. The message "You are not onboarded to Container Insights with enhanced observability for Amazon ECS" suggests that there might be additional steps required to fully activate the feature.

  3. Container Insights with enhanced observability is designed to provide detailed metrics and logs, including information about failed tasks. The fact that you're not seeing logs in the Container Insights dashboard indicates that the feature is not functioning as expected.

  4. To troubleshoot this issue, you might want to try the following:

    a. Double-check that your Amazon ECS agent is version 1.29 or later, which is required for Container Insights.

    b. Verify that your AWS KMS key is properly configured to work with CloudWatch if you're using encryption.

    c. Consider enabling Container Insights with enhanced observability at the account level. While not strictly necessary, this ensures that all new clusters automatically use the feature and might resolve any configuration issues.

    d. Review your CloudWatch log groups to ensure that logs are being sent correctly from your ECS tasks.

  5. If these steps don't resolve the issue, you may need to contact AWS support for further assistance, as there might be an underlying problem with how Container Insights is interacting with your specific ECS setup.

Remember, Container Insights with enhanced observability should provide visibility into task and container-level metrics, as well as logs for issues like container startup failures or resource constraints. If you're not seeing this information, it indicates that the feature is not fully operational for your cluster.
Sources
Container Insights with enhanced observability now available in Amazon ECS | AWS News Blog
Setting up Container Insights on Amazon ECS - Amazon CloudWatch

answered 9 months ago
EXPERT
reviewed 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content