- Newest
- Most votes
- Most comments
I patched my GitLab instance with this patch: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71657/diffs?commit_id=fdb7ffdc17b40ee69687aa618ac3bc201c3c8257,
Now the error message is: `An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements``
I can't figure out what happens now...
Any idea welcome
Here is the debug:
$ aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name test --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE --debug
2021-10-05 14:44:37,307 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.2.43 Python/3.8.8 Linux/4.14.243-185.433.amzn2.x86_64 docker/x86_64.amzn.2
2021-10-05 14:44:37,308 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: \['sts', 'assume-role-with-web-identity', '--role-arn', 'arn:aws:iam::<REDACTED>', '--role-session-name', 'test', '--web-identity-token', 'file:///tmp/aws_web_identity_token_file', '--debug']
2021-10-05 14:44:37,324 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7ff686c1cca0>
2021-10-05 14:44:37,325 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7ff686d76700>
2021-10-05 14:44:37,325 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2021-10-05 14:44:37,325 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7ff686e220d0>
2021-10-05 14:44:37,325 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7ff686e22ee0>
2021-10-05 14:44:37,326 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7ff686c2d700>
2021-10-05 14:44:37,326 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7ff686d41550>
2021-10-05 14:44:37,326 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2021-10-05 14:44:37,326 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7ff686c63940>
2021-10-05 14:44:37,327 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.43/dist/awscli/data/cli.json
2021-10-05 14:44:37,331 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7ff686c76790>
2021-10-05 14:44:37,331 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7ff686c77310>
2021-10-05 14:44:37,331 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7ff686c77280>
2021-10-05 14:44:37,331 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7ff686c77430>
2021-10-05 14:44:37,332 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7ff686c773a0>
2021-10-05 14:44:37,332 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7ff686b49400>
2021-10-05 14:44:37,333 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.2.43 Python/3.8.8 Linux/4.14.243-185.433.amzn2.x86_64 docker/x86_64.amzn.2 prompt/off
2021-10-05 14:44:37,333 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: \['sts', 'assume-role-with-web-identity', '--role-arn', 'arn:aws:iam::<REDACTED>', '--role-session-name', 'test', '--web-identity-token', 'file:///tmp/aws_web_identity_token_file', '--debug']
2021-10-05 14:44:37,333 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7ff686c20310>
2021-10-05 14:44:37,334 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7ff68766f310>
2021-10-05 14:44:37,334 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7ff686b8c160>
2021-10-05 14:44:37,334 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7ff687667790>
2021-10-05 14:44:37,335 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7ff6875c0f70>
2021-10-05 14:44:37,336 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2021-10-05 14:44:37,339 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7ff686d41430>
2021-10-05 14:44:37,339 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7ff686d72550>
2021-10-05 14:44:37,352 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.43/dist/botocore/data/sts/2011-06-15/service-2.json
2021-10-05 14:44:37,355 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sts: calling handler <function add_waiters at 0x7ff686c63940>
2021-10-05 14:44:37,372 - MainThread - awscli.clidriver - DEBUG - OrderedDict(\[('role-arn', <awscli.arguments.CLIArgument object at 0x7ff6861110a0>), ('role-session-name', <awscli.arguments.CLIArgument object at 0x7ff686111190>), ('web-identity-token', <awscli.arguments.CLIArgument object at 0x7ff686111160>), ('provider-id', <awscli.arguments.CLIArgument object at 0x7ff686111130>), ('policy-arns', <awscli.arguments.ListArgument object at 0x7ff6861114c0>), ('policy', <awscli.arguments.CLIArgument object at 0x7ff686111580>), ('duration-seconds', <awscli.arguments.CLIArgument object at 0x7ff6861117c0>)])
2021-10-05 14:44:37,372 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.assume-role-with-web-identity: calling handler <function add_streaming_output_arg at 0x7ff686c208b0>
2021-10-05 14:44:37,373 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.assume-role-with-web-identity: calling handler <function add_cli_input_json at 0x7ff6875cb820>
2021-10-05 14:44:37,374 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.assume-role-with-web-identity: calling handler <function add_cli_input_yaml at 0x7ff6875cbaf0>
2021-10-05 14:44:37,374 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.assume-role-with-web-identity: calling handler <function unify_paging_params at 0x7ff686d76d30>
2021-10-05 14:44:37,388 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.43/dist/botocore/data/sts/2011-06-15/paginators-1.json
2021-10-05 14:44:37,389 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sts.assume-role-with-web-identity: calling handler <function add_generate_skeleton at 0x7ff686cead30>
2021-10-05 14:44:37,389 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.assume-role-with-web-identity: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7ff6861113d0>>
2021-10-05 14:44:37,390 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.assume-role-with-web-identity: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7ff686111310>>
2021-10-05 14:44:37,390 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sts.assume-role-with-web-identity: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7ff6861119d0>>
2021-10-05 14:44:37,391 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role-with-web-identity.role-arn: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff68610df70>
2021-10-05 14:44:37,392 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.sts.assume-role-with-web-identity: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ff68a9d3820>
2021-10-05 14:44:37,392 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'arn:aws:iam::<REDACTED>' for parameter "role_arn": 'arn:aws:iam::<REDACTED>'
2021-10-05 14:44:37,392 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role-with-web-identity.role-session-name: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff68610df70>
2021-10-05 14:44:37,392 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.sts.assume-role-with-web-identity: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ff68a9d3820>
2021-10-05 14:44:37,393 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'test' for parameter "role_session_name": 'test'
2021-10-05 14:44:37,393 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role-with-web-identity.web-identity-token: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff68610df70>
2021-10-05 14:44:37,393 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.sts.assume-role-with-web-identity: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ff68a9d3820>
2021-10-05 14:44:37,394 - MainThread - awscli.arguments - DEBUG - Unpacked value of '\[MASKED]\n' for parameter "web_identity_token": '\[MASKED]\n'
2021-10-05 14:44:37,394 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role-with-web-identity.provider-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff68610df70>
2021-10-05 14:44:37,394 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role-with-web-identity.policy-arns: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff68610df70>
2021-10-05 14:44:37,395 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role-with-web-identity.policy: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff68610df70>
2021-10-05 14:44:37,395 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role-with-web-identity.duration-seconds: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff68610df70>
2021-10-05 14:44:37,395 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role-with-web-identity.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff68610df70>
2021-10-05 14:44:37,395 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role-with-web-identity.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff68610df70>
2021-10-05 14:44:37,396 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sts.assume-role-with-web-identity.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff68610df70>
2021-10-05 14:44:37,396 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.assume-role-with-web-identity: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7ff6861113d0>>
2021-10-05 14:44:37,396 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.assume-role-with-web-identity: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7ff686111310>>
2021-10-05 14:44:37,396 - MainThread - botocore.hooks - DEBUG - Event calling-command.sts.assume-role-with-web-identity: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7ff6861119d0>>
2021-10-05 14:44:37,397 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2021-10-05 14:44:37,397 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2021-10-05 14:44:37,398 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2021-10-05 14:44:37,399 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.43/dist/botocore/data/endpoints.json
2021-10-05 14:44:37,408 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ff6889fb4c0>
2021-10-05 14:44:37,409 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.sts: calling handler <function add_generate_presigned_url at 0x7ff688a2ca60>
2021-10-05 14:44:37,412 - MainThread - botocore.endpoint - DEBUG - Setting sts timeout as (60, 60)
2021-10-05 14:44:37,414 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.sts.AssumeRoleWithWebIdentity: calling handler <function base64_decode_input_blobs at 0x7ff686b8c8b0>
2021-10-05 14:44:37,414 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.sts.AssumeRoleWithWebIdentity: calling handler <function generate_idempotent_uuid at 0x7ff68899d550>
2021-10-05 14:44:37,415 - MainThread - botocore.hooks - DEBUG - Event before-call.sts.AssumeRoleWithWebIdentity: calling handler <function inject_api_version_header_if_needed at 0x7ff6889a1dc0>
2021-10-05 14:44:37,415 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=AssumeRoleWithWebIdentity) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/2.2.43 Python/3.8.8 Linux/4.14.243-185.433.amzn2.x86_64 docker/x86_64.amzn.2 prompt/off command/sts.assume-role-with-web-identity'}, 'body': {'Action': 'AssumeRoleWithWebIdentity', 'Version': '2011-06-15', 'RoleArn': 'arn:aws:iam::<REDACTED>', 'RoleSessionName': 'test', 'WebIdentityToken': '\[MASKED]\n'}, 'url': 'https://sts.eu-west-3.amazonaws.com/', 'context': {'client_region': 'eu-west-3', 'client_config': <botocore.config.Config object at 0x7ff68608e700>, 'has_streaming_input': False, 'auth_type': None}}
2021-10-05 14:44:37,415 - MainThread - botocore.hooks - DEBUG - Event request-created.sts.AssumeRoleWithWebIdentity: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7ff68608e670>>
2021-10-05 14:44:37,416 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sts.AssumeRoleWithWebIdentity: calling handler <function disable_signing at 0x7ff68899d9d0>
2021-10-05 14:44:37,416 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.eu-west-3.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.2.43 Python/3.8.8 Linux/4.14.243-185.433.amzn2.x86_64 docker/x86_64.amzn.2 prompt/off command/sts.assume-role-with-web-identity', 'Content-Length': '1209'}>
2021-10-05 14:44:37,417 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.2.43/dist/botocore/cacert.pem
2021-10-05 14:44:37,418 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): sts.eu-west-3.amazonaws.com:443
2021-10-05 14:44:38,108 - MainThread - urllib3.connectionpool - DEBUG - https://sts.eu-west-3.amazonaws.com:443 "POST / HTTP/1.1" 400 390
2021-10-05 14:44:38,109 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '33c809b7-3786-4d94-ae82-b2c3d023dc8b', 'Content-Type': 'text/xml', 'Content-Length': '390', 'Date': 'Tue, 05 Oct 2021 14:44:37 GMT', 'Connection': 'close'}
2021-10-05 14:44:38,109 - MainThread - botocore.parsers - DEBUG - Response body:
b'<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">\n <Error>\n <Type>Sender</Type>\n <Code>InvalidIdentityToken</Code>\n <Message>Couldn\'t retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\n </Error>\n <RequestId>33c809b7-3786-4d94-ae82-b2c3d023dc8b</RequestId>\n</ErrorResponse>\n'
2021-10-05 14:44:38,110 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '33c809b7-3786-4d94-ae82-b2c3d023dc8b', 'Content-Type': 'text/xml', 'Content-Length': '390', 'Date': 'Tue, 05 Oct 2021 14:44:37 GMT', 'Connection': 'close'}
2021-10-05 14:44:38,111 - MainThread - botocore.parsers - DEBUG - Response body:
b'<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">\n <Error>\n <Type>Sender</Type>\n <Code>InvalidIdentityToken</Code>\n <Message>Couldn\'t retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements</Message>\n </Error>\n <RequestId>33c809b7-3786-4d94-ae82-b2c3d023dc8b</RequestId>\n</ErrorResponse>\n'
2021-10-05 14:44:38,111 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sts.AssumeRoleWithWebIdentity: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7ff685de80a0>>
2021-10-05 14:44:38,111 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2021-10-05 14:44:38,112 - MainThread - botocore.hooks - DEBUG - Event after-call.sts.AssumeRoleWithWebIdentity: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7ff68608eca0>>
2021-10-05 14:44:38,112 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "awscli/clidriver.py", line 459, in main
File "awscli/clidriver.py", line 594, in __call__
File "awscli/clidriver.py", line 770, in __call__
File "awscli/clidriver.py", line 901, in invoke
File "awscli/clidriver.py", line 913, in _make_client_call
File "botocore/client.py", line 278, in _api_call
File "botocore/client.py", line 597, in _make_api_call
botocore.errorfactory.InvalidIdentityTokenException: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements
An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements
Any ideas how this issue was resolved.Please post solution as we are also encountering same issue.Thanks
When trying to understand how IAM Role for ServiceAccount works, I decoded a ServiceAccount token, here is what it looks like:
{
"aud": \[
"https://kubernetes.default.svc.cluster.local"
],
"exp": 1664781927,
"iat": 1633245927,
"iss": "https://kubernetes.default.svc.cluster.local",
"kubernetes.io": {
"namespace": "kube-system",
"pod": {
"name": "kindnet-xrlgj",
"uid": "314bd99e-c985-48f6-94c0-e19273dc75f8"
},
"serviceaccount": {
"name": "kindnet",
"uid": "dc206c6d-334f-4994-a1f0-6ab50c20dc1b"
},
"warnafter": 1633249534
},
"nbf": 1633245927,
"sub": "system:serviceaccount:kube-system:kindnet"
}
While here is what a GitLab job token looks like:
{
"namespace_id": "123456",
"namespace_path": "<username>",
"project_id": "67890",
"project_path": "<username>/<project>",
"user_id": "87398753",
"user_login": "<username>",
"user_email": "<email>",
"pipeline_id": "98080980",
"pipeline_source": "merge_request_event",
"job_id": "1387684868",
"ref": "<branch>",
"ref_type": "branch",
"ref_protected": "false",
"jti": "7f945784-8081-4909-bd97-eb0f7df6dddf",
"iss": "gitlab.com",
"iat": 1633184303,
"nbf": 1633184298,
"exp": 1633187903,
"sub": "job_1387684868"
}
So JWT generated by Kubernetes contains aud
claim, but the one generated by GitLab does not contain it.
That's why it does not work...
Edited by: mcanevet on Oct 5, 2021 6:28 AM
Relevant content
- asked 2 years ago
- asked 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 7 months ago
- AWS OFFICIALUpdated 2 years ago
Any ideas how this issue was resolved.Please post solution as we are also encountering same issue.