By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Bug when combining SCP policies? S3 rb claims to work, but the bucket is not deleted.

0

I created an SCP (service control policy) in my AWS organization restricting resource write-access to four regions (us-east-1, eu-west-1, eu-central-1, eu-central-2) but with an exception for a particular role: Johns-Admin-Role.

The SCP is made by combining these templates:

https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps_examples_general.html#example-scp-deny-region 

      "Statement": [
        {
            "StringNotEquals": {
              "aws:RequestedRegion": [
                "eu-central-1",
                "eu-central-2",
                "eu-west-1",
                "us-east-1"
              ]
            }
          },
          "Effect": "Deny",
          "NotAction": [
            "a4b:*",
            "acm:*",
          ...

https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps_examples_general.html#example-scp-restricts-with-exception

            "ArnNotLike": {
              "aws:PrincipalARN": [
               "arn:aws:iam::924188600000:role/Johns-Admin-Role"
              ]
            },

The use-case is deleting an S3 bucket in region ap-south-1 which was created by a member account before this SCP policy was enabled.

Prior to adding the "role exception" above (ArnNotLike), attempts to remove the bucket fail as expected, using the CLI with a user in the management account, using a "CLI config profile" to assume an admin-permitted role in the member account owning the bucket:

$ aws s3 ls --profile umi2a
    2023-10-18 18:16:44 umi-ww-temp-3
    2023-10-17 08:25:15 umi2-stdout

$ aws s3 rb s3://umi-ww-temp-3  --profile umi2a  --endpoint-url https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com
    remove_bucket failed: s3://umi-ww-temp-3
    An error occurred (AccessDenied)
    when calling the DeleteBucket operation:
    Access Denied

With the "role exception" condition in the SCP, deletion claims to work, but the bucket is not removed:

$ aws s3api delete-bucket  --bucket umi-ww-temp-3  --profile umi2a  --endpoint-url https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com
$ 

$ aws s3 rb s3://umi-ww-temp-3  --profile umi2a  --endpoint-url https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com
    remove_bucket: umi-ww-temp-3

$ aws s3 ls --profile umi2a
    2023-10-18 18:16:44 umi-ww-temp-3
    2023-10-17 08:25:15 umi2-stdout

Shouldn't these CLI commands either fail with an error message, or work as expected, but NOT claim to succeed at something which didn't happen?

And - any tips how to get this to work?

More details:

$ aws iam get-role --role-name Johns-Admin-Role  --profile umi2a
{
    "Role": {
        "Path": "/",
        "RoleName": "Johns-Admin-Role",
        "RoleId": "AROA5OLPPJTLCC5K7XXXX",
        "Arn": "arn:aws:iam::924188600000:role/Johns-Admin-Role",
        "CreateDate": "2023-10-16T17:08:23+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "arn:aws:iam::924188600000:root"
                    },
                    "Action": "sts:AssumeRole",
                    "Condition": {}
                }
            ]
        },
        "Description": "",
        "MaxSessionDuration": 3600,
        "RoleLastUsed": {
            "LastUsedDate": "2023-10-19T18:43:24+00:00",
            "Region": "ap-south-1"
        }
    }
}

$ aws iam list-attached-role-policies --role-name Johns-Admin-Role  --profile umi2a
{
    "AttachedPolicies": [
        {
            "PolicyName": "AdministratorAccess",
            "PolicyArn": "arn:aws:iam::aws:policy/AdministratorAccess"
        }
    ]
}

$ cat ~/.aws/credentials
...
[umi2-admin]
aws_access_key_id = AK...
aws_secret_access_key = IoD...
region = eu-west-1

$ cat ~/.aws/config
...
[profile umi2a]
role_arn = arn:aws:iam::924188600000:role/Johns-Admin-Role
source_profile = umi2-admin
role_session_name = John
region = eu-west-1
services = s3-umi2

[services s3-umi2]
s3 = 
   endpoint_url = https://s3.eu-west-1.amazonaws.com
  • ben-from-aws EXPERT
    6 months ago

    Hi, have you checked if a bucket policy exists that prevents deletion?

  • Marcus Rosen
    6 months ago

    Are you able to share the full SCP policy?

Topics
StorageManagement & Governance
Tags
Amazon Simple Storage ServiceAWS Account ManagementService Control Policy
Language
English
John DB
asked 6 months ago245 views
7 Answers
0

Hi Marcus,

> Are you able to share the full SCP policy?

With some convoluted grepping, here it is:

$ aws organizations describe-policy --policy-id p-0zb7uhp6 --output yaml | \
   grep Content | \
   cut -f2 -d"'" |  \
   jq -S -M
{
  "Statement": [
    {
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalARN": [
            "arn:aws:iam::*:role/Johns-UMI2-Admin-Role",
            "arn:aws:iam::*:role/Johns-UMI3-Admin-Role",
            "arn:aws:iam::*:role/Johns-UMI4-Admin-Role",
            "arn:aws:iam::*:role/Johns-UMI5-Admin-Role"
          ]
        },
        "StringNotEquals": {
          "aws:RequestedRegion": [
            "eu-central-1",
            "eu-central-2",
            "eu-west-1",
            "us-east-1"
          ]
        }
      },
      "Effect": "Deny",
      "NotAction": [
        "a4b:*",
        "acm:*",
        "aws-marketplace-management:*",
        "aws-marketplace:*",
        "aws-portal:*",
        "budgets:*",
        "ce:*",
        "chime:*",
        "cloudfront:*",
        "config:*",
        "cur:*",
        "directconnect:*",
        "ec2:DescribeRegions",
        "ec2:DescribeTransitGateways",
        "ec2:DescribeVpnGateways",
        "fms:*",
        "globalaccelerator:*",
        "health:*",
        "iam:*",
        "importexport:*",
        "kms:*",
        "mobileanalytics:*",
        "networkmanager:*",
        "organizations:*",
        "pricing:*",
        "ram:*",
        "resource-explorer-2:DeleteIndex*",
        "resource-explorer-2:GetIndex*",
        "resource-explorer-2:ListViews*",
        "route53:*",
        "route53domains:*",
        "route53-recovery-cluster:*",
        "route53-recovery-control-config:*",
        "route53-recovery-readiness:*",
        "s3:Get*",
        "s3:List*",
        "s3:PutAccountPublic*",
        "shield:*",
        "sts:*",
        "support:*",
        "trustedadvisor:*",
        "waf-regional:*",
        "waf:*",
        "wafv2:*",
        "wellarchitected:*"
      ],
      "Resource": "*",
      "Sid": "DenyAllOutsideJohnsStuff"
    }
  ],
  "Version": "2012-10-17"
}
John DB
answered 6 months ago
  • Marcus Rosen
    6 months ago

    Regarding this "Shouldn't these CLI commands either fail with an error message, or work as expected, but NOT claim to succeed at something which didn't happen?"

    Are you potentially getting a timeout when calling the regional s3 endpoint? Are you running the command from within an AWS account VPC?

    Can you run the command with a --debug flag?

0

Hi Ben,

> Hi, have you checked if a bucket policy exists that prevents deletion?

Good point. This bucket has no attached policy, so I assume that a delete-bucket ability is provided with the AWS "AdministratorAccess" policy associated with the role, as works with other S3 buckets.

$ aws s3api get-bucket-policy \
   --bucket umi-ww-temp-3 \
   --profile umi2a  \
   --endpoint https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation:
  The bucket policy does not exist
John DB
answered 6 months ago
0

Hi Marcus,

> Are you potentially getting a timeout when calling the regional s3 endpoint?
> Are you running the command from within an AWS account VPC?
> Can you run the command with a --debug flag?

No evidence of timeouts, the commands return quickly with output, or quickly with no output.
I run these command from my client MacOS (home office) via the AWS CLI.

$ date; aws s3 rb s3://umi-ww-temp-3  --profile umi2a  --endpoint-url https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com  --debug
Sun Oct 22 17:17:38 CEST 2023

2023-10-22 17:17:38,738 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64
2023-10-22 17:17:38,739 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'rb', 's3://umi-ww-temp-3', '--profile', 'umi2a', '--endpoint-url', 'https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com', '--debug']
2023-10-22 17:17:38,765 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x10fe642c0>
2023-10-22 17:17:38,765 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x10fc8dc60>
2023-10-22 17:17:38,765 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2023-10-22 17:17:38,765 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x10fbf7ce0>
2023-10-22 17:17:38,765 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x10fc0d300>
2023-10-22 17:17:38,765 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x10fe66d40>
2023-10-22 17:17:38,765 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x10fcf0680>
2023-10-22 17:17:38,765 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2023-10-22 17:17:38,765 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x10fe66c00>
2023-10-22 17:17:38,765 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x10ff1f650>>
2023-10-22 17:17:38,765 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/data/cli.json
2023-10-22 17:17:38,766 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x10fd99f80>
2023-10-22 17:17:38,768 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x10fd9a2a0>
2023-10-22 17:17:38,768 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x10fd9a200>
2023-10-22 17:17:38,768 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x10fd9a3e0>
2023-10-22 17:17:38,768 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x10fd9a340>
2023-10-22 17:17:38,768 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x10ff1c7c0>
2023-10-22 17:17:38,768 - MainThread - botocore.session - DEBUG - Setting config variable for profile to 'umi2a'
2023-10-22 17:17:38,768 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off
2023-10-22 17:17:38,768 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'rb', 's3://umi-ww-temp-3', '--profile', 'umi2a', '--endpoint-url', 'https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com', '--debug']
2023-10-22 17:17:38,768 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x10fe64c20>
2023-10-22 17:17:38,768 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x10f506840>
2023-10-22 17:17:38,768 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x10fee8180>
2023-10-22 17:17:38,768 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x10f3f8b80>
2023-10-22 17:17:38,768 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x10f51e160>
2023-10-22 17:17:38,772 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-10-22 17:17:38,777 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2023-10-22 17:17:38,777 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x10fcc4e00>
2023-10-22 17:17:38,777 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x10fc6dee0>
2023-10-22 17:17:38,777 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x10fe66c00>
2023-10-22 17:17:38,777 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x10ff1f650>>
2023-10-22 17:17:38,777 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_rb: calling handler <function add_waiters at 0x10fe66c00>
2023-10-22 17:17:38,777 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_rb: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x10ff1f650>>
2023-10-22 17:17:38,777 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.rb.path: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10ff57e50>
2023-10-22 17:17:38,777 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.rb: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10f4296d0>
2023-10-22 17:17:38,777 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.rb.force: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10ff57e50>
2023-10-22 17:17:38,777 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.rb: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10f4296d0>
2023-10-22 17:17:38,780 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2023-10-22 17:17:38,780 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2023-10-22 17:17:38,780 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2023-10-22 17:17:38,780 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2023-10-22 17:17:38,781 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2023-10-22 17:17:38,781 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/endpoints.json
2023-10-22 17:17:38,789 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x10e1525c0>
2023-10-22 17:17:38,799 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/s3/2006-03-01/service-2.json
2023-10-22 17:17:38,811 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json
2023-10-22 17:17:38,813 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/partitions.json
2023-10-22 17:17:38,815 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x10e0a2d40>
2023-10-22 17:17:38,815 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x10e0a2ac0>
2023-10-22 17:17:38,840 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2023-10-22 17:17:38,841 - MainThread - botocore.utils - DEBUG - Registering S3 region redirector handler
2023-10-22 17:17:38,841 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x10e176ca0>
2023-10-22 17:17:38,841 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x10ff75990>>
2023-10-22 17:17:38,841 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Bucket': 'umi-ww-temp-3', 'Region': 'eu-west-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2023-10-22 17:17:38,842 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com/umi-ww-temp-3
2023-10-22 17:17:38,844 - MainThread - botocore.regions - DEBUG - Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2023-10-22 17:17:38,844 - MainThread - botocore.regions - DEBUG - Selected auth type "v4" as "v4" with signing context params: {'region': 'eu-west-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2023-10-22 17:17:38,844 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.DeleteBucket: calling handler <function base64_decode_input_blobs at 0x10fee8220>
2023-10-22 17:17:38,844 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.DeleteBucket: calling handler <function validate_bucket_name at 0x10e174a40>
2023-10-22 17:17:38,844 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.DeleteBucket: calling handler <function remove_bucket_from_url_paths_from_model at 0x10e176ac0>
2023-10-22 17:17:38,844 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.DeleteBucket: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x10ff75990>>
2023-10-22 17:17:38,844 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.DeleteBucket: calling handler <function generate_idempotent_uuid at 0x10e174860>
2023-10-22 17:17:38,845 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.DeleteBucket: calling handler <function add_expect_header at 0x10e174e00>
2023-10-22 17:17:38,845 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.DeleteBucket: calling handler <function inject_api_version_header_if_needed at 0x10e176340>
2023-10-22 17:17:38,845 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=DeleteBucket) with params: {'url_path': '', 'query_string': {}, 'method': 'DELETE', 'headers': {'User-Agent': 'aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off command/s3.rb'}, 'body': b'', 'auth_path': '/umi-ww-temp-3/', 'url': 'https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com/umi-ww-temp-3', 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x10f494ad0>, 'has_streaming_input': False, 'auth_type': 'v4', 'signing': {'region': 'eu-west-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 's3_redirect': {'redirected': False, 'bucket': 'umi-ww-temp-3', 'params': {'Bucket': 'umi-ww-temp-3'}}}}
2023-10-22 17:17:38,849 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.DeleteBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x110649dd0>>
2023-10-22 17:17:38,849 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.DeleteBucket: calling handler <function set_operation_specific_signer at 0x10e174720>
2023-10-22 17:17:38,849 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.DeleteBucket: calling handler <function remove_arn_from_signing_path at 0x10e176c00>
2023-10-22 17:17:38,850 - MainThread - botocore.credentials - DEBUG - Credentials for role retrieved from cache.
2023-10-22 17:17:38,850 - MainThread - botocore.credentials - DEBUG - Retrieved credentials will expire at: 2023-10-22 15:43:03+00:00
2023-10-22 17:17:38,850 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2023-10-22 17:17:38,850 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
DELETE
/umi-ww-temp-3

host:umi-ww-temp-3.s3.ap-south-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20231022T151738Z
x-amz-security-token:IQoJb3JpZ2luX2VjEO///////////wEaCWV1LXdlc3QtMSJHMEUCIEwZ57Y8J2GZqE+ZzSVFbybiKDiJrx3Qe5Rog/oRC5MZAiEA0KIajShoVcI9CjNf4PpQaHtAZfSRml2D3q13t0CdYrsqkQIIGBAAGgw5MjQxODg2MjYxMzQiDH39zS0PWLVSnHf3ISruAUPrJvUOEnFClHMWwkn+ELUvyEY4HKXNXNU9nOVx8mJC9lGguY2Oatvw9PnDp4RnkzB4ORePCArfcSjtyzWTy8TtdpvHfUxRzI41xaM1M/wNjTTHiBd4WserRyQTOKIs86pI5VT36OPA2nI93JVIEVFP7tmUp7npVJM6EFjN33a/TXKnPwgv5L5Xstmv8o9zY96n8b7uFK1ikR3DZAwlzbdrN3KPSer4Aq8gus68Aq0Zr5GfkLNGeKHZ+bjVXyHt2WSpPHp1dqG5vqeTc42sQHlslbIkwLkaQpRCKBb1PLQ+h9BjpCrJRtyAzSFL54gw9+nUqQY6nQHat+BvdAdYsix6heBMVi/k6wSV+7sOcfT1NENblABF52BEJXKSvUXbhHtUcmfdx5n/u5/KcMARAM9moOxLBuY9LjZ4w9xXevlaMgf6FMlOfkc3hniYxj+A0q/DiPL24jWjmWTVV8qBkHuPVJU2eZT53BrEvxGicMqT1Zpz3EKC8Nog63H0ZIeTyE5eEeIBZdSUvQe1XvG5fa0DSdbK
...
John DB
answered 6 months ago
0

More debug log, the above was truncated due to limits.

host:umi-ww-temp-3.s3.ap-south-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20231022T151739Z
x-amz-security-token:IQoJb3JpZ2luX2VjEO///////////wEaCWV1LXdlc3QtMSJHMEUCIEwZ57Y8J2GZqE+ZzSVFbybiKDiJrx3Qe5Rog/oRC5MZAiEA0KIajShoVcI9CjNf4PpQaHtAZfSRml2D3q13t0CdYrsqkQIIGBAAGgw5MjQxODg2MjYxMzQiDH39zS0PWLVSnHf3ISruAUPrJvUOEnFClHMWwkn+ELUvyEY4HKXNXNU9nOVx8mJC9lGguY2Oatvw9PnDp4RnkzB4ORePCArfcSjtyzWTy8TtdpvHfUxRzI41xaM1M/wNjTTHiBd4WserRyQTOKIs86pI5VT36OPA2nI93JVIEVFP7tmUp7npVJM6EFjN33a/TXKnPwgv5L5Xstmv8o9zY96n8b7uFK1ikR3DZAwlzbdrN3KPSer4Aq8gus68Aq0Zr5GfkLNGeKHZ+bjVXyHt2WSpPHp1dqG5vqeTc42sQHlslbIkwLkaQpRCKBb1PLQ+h9BjpCrJRtyAzSFL54gw9+nUqQY6nQHat+BvdAdYsix6heBMVi/k6wSV+7sOcfT1NENblABF52BEJXKSvUXbhHtUcmfdx5n/u5/KcMARAM9moOxLBuY9LjZ4w9xXevlaMgf6FMlOfkc3hniYxj+A0q/DiPL24jWjmWTVV8qBkHuPVJU2eZT53BrEvxGicMqT1Zpz3EKC8Nog63H0ZIeTyE5eEeIBZdSUvQe1XvG5fa0DSdbK

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-10-22 17:17:39,659 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20231022T151739Z
20231022/ap-south-1/s3/aws4_request
f60bee2e5fedc47ab203abde86130fe5a8381f2b9f9b6c0d31bf3e583d2aa380
2023-10-22 17:17:39,659 - MainThread - botocore.auth - DEBUG - Signature:
56c73f5cf3502b5e61d4d17ea72f67f2b0448e78cfb5841dab20ef7c41be1028
2023-10-22 17:17:39,659 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=DELETE, url=https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com/umi-ww-temp-3, headers={'User-Agent': b'aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off command/s3.rb', 'X-Amz-Date': b'20231022T151739Z', 'X-Amz-Security-Token': b'IQoJb3JpZ2luX2VjEO///////////wEaCWV1LXdlc3QtMSJHMEUCIEwZ57Y8J2GZqE+ZzSVFbybiKDiJrx3Qe5Rog/oRC5MZAiEA0KIajShoVcI9CjNf4PpQaHtAZfSRml2D3q13t0CdYrsqkQIIGBAAGgw5MjQxODg2MjYxMzQiDH39zS0PWLVSnHf3ISruAUPrJvUOEnFClHMWwkn+ELUvyEY4HKXNXNU9nOVx8mJC9lGguY2Oatvw9PnDp4RnkzB4ORePCArfcSjtyzWTy8TtdpvHfUxRzI41xaM1M/wNjTTHiBd4WserRyQTOKIs86pI5VT36OPA2nI93JVIEVFP7tmUp7npVJM6EFjN33a/TXKnPwgv5L5Xstmv8o9zY96n8b7uFK1ikR3DZAwlzbdrN3KPSer4Aq8gus68Aq0Zr5GfkLNGeKHZ+bjVXyHt2WSpPHp1dqG5vqeTc42sQHlslbIkwLkaQpRCKBb1PLQ+h9BjpCrJRtyAzSFL54gw9+nUqQY6nQHat+BvdAdYsix6heBMVi/k6wSV+7sOcfT1NENblABF52BEJXKSvUXbhHtUcmfdx5n/u5/KcMARAM9moOxLBuY9LjZ4w9xXevlaMgf6FMlOfkc3hniYxj+A0q/DiPL24jWjmWTVV8qBkHuPVJU2eZT53BrEvxGicMqT1Zpz3EKC8Nog63H0ZIeTyE5eEeIBZdSUvQe1XvG5fa0DSdbK', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIA5OLPPJTLPS7UY3MI/20231022/ap-south-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=56c73f5cf3502b5e61d4d17ea72f67f2b0448e78cfb5841dab20ef7c41be1028', 'Content-Length': '0'}>
2023-10-22 17:17:39,661 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/awscli/botocore/cacert.pem
2023-10-22 17:17:39,661 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: umi-ww-temp-3.s3.ap-south-1.amazonaws.com
2023-10-22 17:17:40,462 - MainThread - urllib3.connectionpool - DEBUG - https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com:443 "DELETE /umi-ww-temp-3 HTTP/1.1" 204 0
2023-10-22 17:17:40,463 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'A3hFS0A2e53qVjJ3IUF/3im07a25DkRAA2N4zowZiwFI+w8XwvdqlCQG6TmOPpHS+4C+ZTS3B5s=', 'x-amz-request-id': 'RKE7MN564DHXGCNE', 'Date': 'Sun, 22 Oct 2023 15:17:41 GMT', 'Server': 'AmazonS3'}
2023-10-22 17:17:40,463 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2023-10-22 17:17:40,464 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.DeleteBucket: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x1106aea50>>
2023-10-22 17:17:40,464 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-10-22 17:17:40,464 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.DeleteBucket: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x10ff75990>>
2023-10-22 17:17:40,464 - MainThread - botocore.utils - DEBUG - S3 request was previously redirected, not redirecting.
2023-10-22 17:17:40,475 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.DeleteBucket: calling handler <function enhance_error_msg at 0x10fe64860>
2023-10-22 17:17:40,475 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.DeleteBucket: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x1106add90>>

remove_bucket: umi-ww-temp-3
John DB
answered 6 months ago
0

Another issue:
The delete-attempt seems invisible to CloudTrail logs, even with a falsely claimed success.
Is this a ** potential security-bug** in CloudTrail API logging?

Delete attempts:

$ date; aws s3 ls --profile umi2a
    Mon Oct 23 07:16:49 CEST 2023
    2023-10-18 18:16:44 umi-ww-temp-3
    2023-10-17 08:25:15 umi2-stdout

$ date; aws s3 rb s3://umi-ww-temp-3  --profile umi2a  --endpoint-url https://umi-ww-temp-3.s3.ap-south-1.amazonaws.com
    Mon Oct 23 07:17:02 CEST 2023
    remove_bucket: umi-ww-temp-3

$ date; aws s3 ls --profile umi2a
    Mon Oct 23 07:17:07 CEST 2023
    2023-10-18 18:16:44 umi-ww-temp-3
    2023-10-17 08:25:15 umi2-stdout

I mount the S3 log bucket on a Linux server, using Mountpoint for Amazon S3.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint.html

The two CLI ls commands are visible, but not the rb command.
The same IAM policy allowing CloudTrail logging to S3 should suffice for both.

$ date; find s3-trails -wholename '*2023/10/23/*' -type f | \
      while read qq; do \
         x=$(zcat $qq | jqz | grep aws-cli | grep command/s3); \
         rc=$?; \
         if [[ $rc -eq 0 ]]; then echo; echo $qq; echo "$x"; fi; \
      done 

Mon Oct 23 07:30:00 CEST 2023
s3-trails/AWSLogs/924188600000/CloudTrail/eu-west-1/2023/10/23/924188600000_CloudTrail_eu-west-1_20231023T0520Z_ForqqyHffvzxha6G.json.gz
 "userAgent": "aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off command/s3.ls",
 "userAgent": "[aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off command/s3.ls]",
 "userAgent": "[aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off command/s3.ls]",


$ zcat s3-trails/AWSLogs/924188600000/CloudTrail/eu-west-1/2023/10/23/924188600000_CloudTrail_eu-west-1_20231023T0520Z_ForqqyHffvzxha6G.json.gz | \
  jq -S -M | \
  egrep -i 'userAgent|eventTime'

 "eventTime": "2023-10-23T05:16:50Z",
 "userAgent": "aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off command/s3.ls",

 "eventTime": "2023-10-23T05:16:50Z",
 "userAgent": "[aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off command/s3.ls]",

 "eventTime": "2023-10-23T05:17:07Z",
 "userAgent": "[aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off command/s3.ls]",
John DB
answered 6 months ago
0

Here's the full CloudTrail log, with small privacy redactions:

$ zcat s3-trails/AWSLogs/924188600000/CloudTrail/eu-west-1/2023/10/23/924188600000_CloudTrail_eu-west-1_20231023T0520Z_ForqqyHffvzxha6G.json.gz | \
  jq -S -M
{
  "Records": [
    {
      "awsRegion": "eu-west-1",
      "eventCategory": "Management",
      "eventID": "727b252c-c734-48f6-ab48-c59a9e01c74e",
      "eventName": "AssumeRole",
      "eventSource": "sts.amazonaws.com",
      "eventTime": "2023-10-23T05:16:50Z",
      "eventType": "AwsApiCall",
      "eventVersion": "1.08",
      "managementEvent": true,
      "readOnly": true,
      "recipientAccountId": "924188600000",
      "requestID": "0f52d5ad-55fd-4b21-9f5a-567146ca6652",
      "requestParameters": {
        "roleArn": "arn:aws:iam::924188600000:role/Johns-UMI2-Admin-Role",
        "roleSessionName": "John"
      },
      "resources": [
        {
          "ARN": "arn:aws:iam::924188600000:role/Johns-UMI2-Admin-Role",
          "accountId": "924188600000",
          "type": "AWS::IAM::Role"
        }
      ],
      "responseElements": {
        "assumedRoleUser": {
          "arn": "arn:aws:sts::924188600000:assumed-role/Johns-UMI2-Admin-Role/John",
          "assumedRoleId": "AROA5OLPPJTLCC5K7XXXX:John"
        },
        "credentials": {
          "accessKeyId": "ASIA5OLPPJTLESZ5XXXX",
          "expiration": "Oct 23, 2023, 6:16:50 AM",
          "sessionToken": "IQoJb3...Jbtn"
        }
      },
      "sourceIPAddress": "85...",
      "tlsDetails": {
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "sts.eu-west-1.amazonaws.com",
        "tlsVersion": "TLSv1.2"
      },
      "userAgent": "aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off command/s3.ls",
      "userIdentity": {
        "accessKeyId": "AKIA5OLPPJTLFGVLXXXX",
        "accountId": "924188600000",
        "arn": "arn:aws:iam::924188600000:user/umi2-admin",
        "principalId": "AIDA5OLPPJTLMOZZXXXXX",
        "type": "IAMUser",
        "userName": "umi2-admin"
      }
    },
    {
      "additionalEventData": {
        "AuthenticationMethod": "AuthHeader",
        "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "SignatureVersion": "SigV4",
        "bytesTransferredIn": 0,
        "bytesTransferredOut": 464,
        "x-amz-id-2": "lKOT9wu...ZWQ="
      },
      "awsRegion": "eu-west-1",
      "eventCategory": "Management",
      "eventID": "ac8bb0ac-2364-4fc1-a4c3-b6ffc55a9aa2",
      "eventName": "ListBuckets",
      "eventSource": "s3.amazonaws.com",
      "eventTime": "2023-10-23T05:16:50Z",
      "eventType": "AwsApiCall",
      "eventVersion": "1.09",
      "managementEvent": true,
      "readOnly": true,
      "recipientAccountId": "924188600000",
      "requestID": "32DTK1AFBKASXXXX",
      "requestParameters": {
        "Host": "s3.eu-west-1.amazonaws.com"
      },
      "responseElements": null,
      "sourceIPAddress": "85...",
      "tlsDetails": {
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "s3.eu-west-1.amazonaws.com",
        "tlsVersion": "TLSv1.2"
      },
      "userAgent": "[aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off command/s3.ls]",
      "userIdentity": {
        "accessKeyId": "ASIA5OLPPJTLESZ5XXXX",
        "accountId": "924188600000",
        "arn": "arn:aws:sts::924188600000:assumed-role/Johns-UMI2-Admin-Role/John",
        "principalId": "AROA5OLPPJTLCC5K7XXXX:John",
        "sessionContext": {
          "attributes": {
            "creationDate": "2023-10-23T05:16:50Z",
            "mfaAuthenticated": "false"
          },
          "sessionIssuer": {
            "accountId": "924188600000",
            "arn": "arn:aws:iam::924188600000:role/Johns-UMI2-Admin-Role",
            "principalId": "AROA5OLPPJTLCC5K7XXXX",
            "type": "Role",
            "userName": "Johns-UMI2-Admin-Role"
          }
        },
        "type": "AssumedRole"
      }
    },
    {
      "additionalEventData": {
        "AuthenticationMethod": "AuthHeader",
        "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "SignatureVersion": "SigV4",
        "bytesTransferredIn": 0,
        "bytesTransferredOut": 464,
        "x-amz-id-2": "kbJc3...2Ek="
      },
      "awsRegion": "eu-west-1",
      "eventCategory": "Management",
      "eventID": "88b42a51-e679-4a8b-b39e-5ac92eddec74",
      "eventName": "ListBuckets",
      "eventSource": "s3.amazonaws.com",
      "eventTime": "2023-10-23T05:17:07Z",
      "eventType": "AwsApiCall",
      "eventVersion": "1.09",
      "managementEvent": true,
      "readOnly": true,
      "recipientAccountId": "924188600000",
      "requestID": "1KCB4BR4VNA2EH6V",
      "requestParameters": {
        "Host": "s3.eu-west-1.amazonaws.com"
      },
      "responseElements": null,
      "sourceIPAddress": "85...",
      "tlsDetails": {
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "s3.eu-west-1.amazonaws.com",
        "tlsVersion": "TLSv1.2"
      },
      "userAgent": "[aws-cli/2.13.17 Python/3.11.5 Darwin/22.6.0 exe/x86_64 prompt/off command/s3.ls]",
      "userIdentity": {
        "accessKeyId": "ASIA5OLPPJTLESZ5XXXX",
        "accountId": "924188600000",
        "arn": "arn:aws:sts::924188600000:assumed-role/Johns-UMI2-Admin-Role/John",
        "principalId": "AROA5OLPPJTLCC5K7XXXX:John",
        "sessionContext": {
          "attributes": {
            "creationDate": "2023-10-23T05:16:50Z",
            "mfaAuthenticated": "false"
          },
          "sessionIssuer": {
            "accountId": "924188600000",
            "arn": "arn:aws:iam::924188600000:role/Johns-UMI2-Admin-Role",
            "principalId": "AROA5OLPPJTLCC5K7XXXX",
            "type": "Role",
            "userName": "Johns-UMI2-Admin-Role"
          }
        },
        "type": "AssumedRole"
      }
    }
  ]
}
John DB
answered 6 months ago
0

FYI: Here's the CloudTrail config, which should allow logging of AWS CLI ls and rb commands.

$ aws cloudtrail get-trail --name "arn:aws:cloudtrail:eu-central-2:710455280000:trail/aws-cloudtrail-umi1"
{
    "Trail": {
        "Name": "aws-cloudtrail-umi1",
        "S3BucketName": "aws-cloudtrail-logs-umi4",
        "IncludeGlobalServiceEvents": true,
        "IsMultiRegionTrail": true,
        "HomeRegion": "eu-central-2",
        "TrailARN": "arn:aws:cloudtrail:eu-central-2:710455280000:trail/aws-cloudtrail-umi1",
        "LogFileValidationEnabled": false,
        "HasCustomEventSelectors": true,
        "HasInsightSelectors": false,
        "IsOrganizationTrail": false
    }
}

$ aws cloudtrail get-event-selectors --trail-name "arn:aws:cloudtrail:eu-central-2:710455280000:trail/aws-cloudtrail-umi1"
{
    "TrailARN": "arn:aws:cloudtrail:eu-central-2:710455280000:trail/aws-cloudtrail-umi1",
    "AdvancedEventSelectors": [
        {
            "Name": "Management events selector",
            "FieldSelectors": [
                {
                    "Field": "eventCategory",
                    "Equals": [
                        "Management"
                    ]
                },
                {
                    "Field": "eventSource",
                    "NotEquals": [
                        "kms.amazonaws.com",
                        "rdsdata.amazonaws.com"
                    ]
                }
            ]
        }
    ]
}
John DB
answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content