Greengrass autoprovision with temp credentials is broken?

0

I've been trying to autoprovision greengrass devices using temporary credentials. It always fails at the same spot. I've tried on different types of devices.

Attaching TES role policy to IoT thing... Exiting due to unexpected error while looking up managed policy - The security token included in the request is invalid (Service: Iam, Status Code: 403, Request ID: 18d4f9c5-3f9a-49dc-YYYY-XXXXXXXXXXXXX, Extended Request ID: null) Error while trying to setup Greengrass Nucleus software.amazon.awssdk.services.iam.model.IamException: The security token included in the request is invalid (Service: Iam, Status Code: 403, Request ID: 18d4f9c5-3f9a-49dc-YYYY-XXXXXXXXXXX, Extended Request ID: null)

The creds are exported as environmental vars correctly.** I confirm the temp credentials work on the devices using other AWS CLI commands. I also confirm its calling the original users permissions with : aws sts get-caller-identity**

If I use the original users access key and secret, the device will autoprovision correctly. But never with temp credentials. Nucleus version is 2.8.1

Temp creds WILL WORK if you don't have '--provision true' set.

I don't believe any special permissions are ever needed for TES. So I think there may be some endpoint issue in the Nucleus code. Someone please prove me wrong, its driving me crazy.

  • Hey AWS-Frenzy - to help us try to reproduce your issue, can you provide some the exact steps you're following, as well as the provisioning command you're using? Thanks

  • For JoeAtAWS:

    I am using the standard command that creating a new Greengrass device creates:

    sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassInstaller/lib/Greengrass.jar --aws-region us-east-1 --thing-name Thing1 --thing-group-name Goup1 --component-default-user ggc_user:ggc_group --provision true --setup-system-service true --deploy-dev-tools true

  • Sorry Joe, you asked for exact steps. I'll list them out.

    1. In console, I create a new GG device. This gives 2 commands to be run on the device. I pause here, and don't run those commands yet.
    2. As a user with full admin privileges I create temp creds: aws sts get-session-token --region us-east-1
    3. I then export those temp creds to environmental variables on the device: export AWS_ACCESS_KEY_ID=XXX export AWS_SECRET_ACCESS_KEY=YYY export AWS_SESSION_TOKEN=AQoDYXdzEJr...<remainder of session token>
    4. I confirm the device can use those temp creds: aws s3 ls and aws sts get-caller-identity
    5. I run the first command on the device to get the software needed: curl -s https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-nucleus-latest.zip > greengrass-nucleus-latest.zip && unzip greengrass-nucleus-latest.zip -d GreengrassInstaller
    6. I run the second command to install the software and autoprovision the device sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassInstaller/lib/Greengrass.jar --aws-region us-east-1 --thing-name Thing1 --thing-group-name Goup1 --component-default-user ggc_user:ggc_group --provision true --setup-system-service true --deploy-dev-tools true

    That is when I get the: Attaching TES role policy to IoT thing... Exiting due to unexpected error while looking up managed policy - The security token included in the request is invalid (Service: Iam, Status Code: 403

  • Do you have credentials on the device in some other way in addition to the environment variables? For example, a profile file in ~/.aws/credentials, EC2 instance credentials, or SSM credentials?

  • Negative. There are no other credentials on these devices other then what we added to the exported vars.

    I ask this kindly to anyone at AWS, can you simply try to provision your own device using temp credentials and see if it works with the current Nucleus code?

2 Answers
3
Accepted Answer

Hi,

The problem is that get-session-token does not allow any calls to IAM unless MFA authentication information is provided as part of the request as described in the documentation here: https://docs.aws.amazon.com/cli/latest/reference/sts/get-session-token.html.

You can use long lived credentials, pass MFA authentication to STS, or use one of the other supported provisioning methods such as fleet provisioning https://docs.aws.amazon.com/greengrass/v2/developerguide/fleet-provisioning.html or JITP https://docs.aws.amazon.com/iot/latest/developerguide/jit-provisioning.html

You can also create and later deactivate long term credentials as another possibility.

AWS
EXPERT
answered a year ago
  • Confirmed solution. When using the MFA token in getting temp creds, I am able to provision. (I think GG docs should be updated to say this is required.) I did however have to pass the creds using the system properties method. Environmental vars still didn't work. I can live with that.

  • I wanted to add a Thank you Michael for working through this with me. I really appreciate it.

  • You're very welcome, sorry it took so long to get to the bottom of it, but I'm glad that we found a solution for you (MFA). I'll also check with our docs team about adding this information.

0

Few pointes for you to consider:

1/ I am assuming you are following the guidelines as in this documentation, but re-checking. https://docs.aws.amazon.com/greengrass/v2/developerguide/quick-installation.html#provide-installer-aws-credentials

2/ Can you try provisioning with an IAM user's set of credentials temporarily with the AdministratorAccess IAM policy (you can remove after Greengrass has successfully provisioned).

3/ Follow the Troubleshooting AWS IoT Greengrass guide. It has specific section for the temporary credentials fail https://docs.aws.amazon.com/greengrass/v2/developerguide/troubleshooting.html

AWS
answered a year ago
  • Yes on all 3.

    On #2, see my comment that I confirm the devices are provisioning when using the long term creds of the user. But not when using temporary. On #3, see my comment that the temp credentials are in fact being applied correctly. The devices can use those temp credentials to do other things, like list S3 buckets, etc. They just can't autoprovision.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions