By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

FluentBit Cloudwatch plugin not able to use credentials from Token Exchange Service component

0

After deploying the public.ecr.aws/aws-observability/aws-for-fluent-bit:latest container as a greengrass component, I am unable to have it successfully send logs to cloudwatch, because the FluentBit executable does not seem to want to get its credentials from the Token Exchange Service.

I have tried passing on the TES specific environment variables to the container, (AWS_CONTAINER_CREDENTIALS_FULL_URI, AWS_CONTAINER_AUTHORIZATION_TOKEN, AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT and AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH), and have tried passing the value of $AWS_CONTAINER_CREDENTIALS_FULL_URI to the sts_endpoint parameter documented here: https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch, but to no effect: the plugin seems to reach out to to ECS to get credentials regardless.

Can anyone tell me what I might be doing wrong, or is this a known limitation of the fluentbit cloudwatch plugin?

Topics
Internet of Things (IoT)Management & Governance
Tags
AWS IoT GreengrassAmazon CloudWatch
Language
English
Eric Casteleijn
asked 10 days ago130 views
2 Answers
0

Hello,

Make sure you're running the container as explained here: https://docs.aws.amazon.com/greengrass/v2/developerguide/run-docker-container.html#docker-container-token-exchange-service.

If that does not work, then that software may not support getting AWS credentials properly and you will need to reach out to that vendor for support.

Cheers,

Michael

AWS
EXPERT
MichaelDombrowski-AWS
answered 10 days ago
  • Eric Casteleijn
    10 days ago

    Thanks, that is what I'm doing, I believe. Making a boto call from within the container works, but not the plugin. The vendor in this case would be Amazon itself, since they are providing the container image and (I believe) maintain the fluentbit plugin that talks to Cloudwatch.

  • MichaelDombrowski-AWS EXPERT
    10 days ago

    Do you have any logs from the plugin which would help explain what the problem is?

  • Eric Casteleijn
    10 days ago

    I appreciate you taking a look. it doesn't appear I can paste more than a couple of lines of logs in this comment, will try in a separate answer.

0

what I believe are relevant part of the logs:

bash-4.2# export AWS_ROLE_ARN=REDACTED
bash-4.2# fluent-bit/bin/fluent-bit -i cpu -o cloudwatch_logs -p region=us-east-1 -p log_group_name=ksdd-2842 -p log_stream_prefix=$AWS_IOT_THING_NAME -p role_arn=REDACTED -vvv
...
[2024/04/24 22:13:28] [debug] [aws_credentials] Initialized Env Provider in standard chain
[2024/04/24 22:13:28] [debug] [aws_credentials] Initialized AWS Profile Provider in standard chain
[2024/04/24 22:13:28] [debug] [aws_credentials] Not initializing EKS provider because AWS_WEB_IDENTITY_TOKEN_FILE was not set
[2024/04/24 22:13:28] [debug] [aws_credentials] Not initializing ECS Provider because AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is not set
[2024/04/24 22:13:28] [debug] [aws_credentials] Initialized EC2 Provider in standard chain
[2024/04/24 22:13:28] [debug] [aws_credentials] Sync called on the STS provider
[2024/04/24 22:13:28] [debug] [aws_credentials] Sync called on the EC2 provider
[2024/04/24 22:13:28] [debug] [aws_credentials] Init called on the STS provider
[2024/04/24 22:13:28] [debug] [aws_credentials] Init called on the env provider
[2024/04/24 22:13:28] [debug] [aws_credentials] Init called on the profile provider
[2024/04/24 22:13:28] [debug] [aws_credentials] Reading shared config file.
[2024/04/24 22:13:28] [debug] [aws_credentials] Shared config file /root/.aws/config does not exist
[2024/04/24 22:13:28] [debug] [aws_credentials] Reading shared credentials file.
[2024/04/24 22:13:28] [debug] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist
[2024/04/24 22:13:28] [debug] [aws_credentials] Init called on the EC2 IMDS provider
[2024/04/24 22:13:28] [debug] [aws_credentials] requesting credentials from EC2 IMDS
[2024/04/24 22:13:29] [error] [net] connection #25 timeout after 1 seconds to: 169.254.169.254:80
[2024/04/24 22:13:29] [debug] [net] socket #25 could not connect to 169.254.169.254:80
[2024/04/24 22:13:29] [debug] [net] could not connect to 169.254.169.254:80
[2024/04/24 22:13:29] [debug] [upstream] connection #-1 failed to 169.254.169.254:80
[2024/04/24 22:13:29] [debug] [aws_client] connection initialization error
[2024/04/24 22:13:29] [debug] [imds] imds endpoint unavailable
[2024/04/24 22:13:29] [ warn] [imds] unable to evaluate IMDS version
[2024/04/24 22:13:29] [debug] [aws_credentials] Calling STS..
[2024/04/24 22:13:29] [debug] [http_client] not using http_proxy for header
[2024/04/24 22:13:29] [debug] [aws_credentials] Requesting credentials from the env provider..
[2024/04/24 22:13:29] [debug] [aws_credentials] Retrieving credentials for AWS Profile default
[2024/04/24 22:13:29] [debug] [aws_credentials] Reading shared config file.
[2024/04/24 22:13:29] [debug] [aws_credentials] Shared config file /root/.aws/config does not exist
[2024/04/24 22:13:29] [debug] [aws_credentials] Reading shared credentials file.
[2024/04/24 22:13:29] [error] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist
[2024/04/24 22:13:29] [error] [aws_credentials] Failed to retrieve credentials for AWS Profile default
[2024/04/24 22:13:29] [debug] [aws_credentials] Requesting credentials from the EC2 provider..
[2024/04/24 22:13:29] [debug] [aws_credentials] requesting credentials from EC2 IMDS
[2024/04/24 22:13:30] [error] [net] connection #26 timeout after 1 seconds to: 169.254.169.254:80
[2024/04/24 22:13:30] [debug] [net] socket #26 could not connect to 169.254.169.254:80
[2024/04/24 22:13:30] [debug] [net] could not connect to 169.254.169.254:80
[2024/04/24 22:13:30] [debug] [upstream] connection #-1 failed to 169.254.169.254:80
[2024/04/24 22:13:30] [error] [aws_client] connection initialization error
[2024/04/24 22:13:30] [debug] [imds] imds endpoint unavailable
[2024/04/24 22:13:30] [ warn] [imds] unable to evaluate IMDS version
[2024/04/24 22:13:30] [ warn] [aws_credentials] No cached credentials are available and a credential refresh is already in progress. The current co-routine will retry.
[2024/04/24 22:13:30] [error] [signv4] Provider returned no credentials, service=sts
[2024/04/24 22:13:30] [debug] [aws_client] could not sign request
...
Eric Casteleijn
answered 10 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content