Skip to content

Pod using kube2iam on k3s, starts fine but fails during rollout restart

0

I have a k3s cluster on which kube2iam is deployed as a daemonset. It seems to start okay getting access to the correct iam role through kube2iam and is able to access the appropriate aws resource. However, when I do a rollout restart of the deployment it fails with the following error message:

AccessDeniedException: User: arn:aws:sts::XX:assumed-role/NODE_IAM_ROLE/NODE_NAME is not authorized to perform: ACTION

Indeed, the intention is only provide the access to the pod's iam role and NOT to the node's iam role but I am puzzled as to why the pod's role is not being assumed.

If I delete the deployment and install it afresh, it works again. Not using rollout is not really an option so seeking helpful hints on what I might be doing wrong? Wondering if it might be our upgrade from amazonlinux2 to amazonlinux2023 that broke our kube2iam setup that seemed to be working fine all this while.

Relevant details: Tried both kube2iam:0.11.1 and 0.11.2 (with IMDSv2 optional and required) but the behavior is the same.

Node's iam policy:

data "aws_iam_policy_document" "trust-policy-node" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

data "aws_iam_policy_document" "role-policy-node" {
  statement {
    actions = [
      "sts:AssumeRole"
    ]
    resources = ["*"]
  }
  statement {
    actions = [
      "ec2:DescribeRegions",
    ]
    resources = ["*"]
  }
}

resource "aws_iam_role" "role-node" {
  name               = local.project_prefix
  assume_role_policy = data.aws_iam_policy_document.trust-policy-node.json

  inline_policy {
    name   = local.project_prefix
    policy = data.aws_iam_policy_document.role-policy-node.json
  }
}

Pod's iam policy:

data "aws_iam_policy_document" "role-policy-app" {
  statement {
    actions = [
      "s3:Get*",
      "s3:List*"
    ]
    resources = [
      "arn:aws:s3:::mybucket",
      "arn:aws:s3:::mybucket/*"
    ]
  }
}

data "aws_iam_policy_document" "trust-policy-pod" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.role-node.arn]
    }
  }
}

resource "aws_iam_role" "role-app" {
  name               = "${local.project_prefix}-app"
  assume_role_policy = data.aws_iam_policy_document.trust-policy-pod.json

  inline_policy {
    name   = "${local.project_prefix}-app"
    policy = data.aws_iam_policy_document.role-policy-app.json
  }
}

kube2iam daemonset:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube2iam
spec:
  template:
    spec:
      serviceAccountName: kube2iam
      hostNetwork: true
      containers:
      - image: jtblin/kube2iam:0.11.2
        imagePullPolicy: Always
        name: kube2iam
        args:
        - "--auto-discover-base-arn"
        - "--app-port=8181"
        - "--iam-role-key=iam.amazonaws.com/role"
        - "--iam-role-session-ttl=900s"
        - "--node=$(NODE_NAME)"
        - "--debug"
        - "--log-level=info"
        env:
        - name: AWS_REGION
           value: <region>
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        ports:
        - name: http
          containerPort: 8181
          hostPort: 8181

The iptables command in cloud-init

# Reroute metadata service requests to kube2iam
      iptables \
        --append PREROUTING \
        --protocol tcp \
        --destination 169.254.169.254 \
        --dport 80 \
        --in-interface cni0 \
        --jump DNAT \
        --table nat \
        --to-destination `curl 169.254.169.254/latest/meta-data/local-ipv4`:8181
1 Answer
0

Hello,

Regarding the error message you receive ("AccessDeniedException: User: arn:aws:sts::XX:assumed-role/NODE_IAM_ROLE/NODE_NAME is not authorized to perform: ACTION"), it's pretty important whether "ACTION" is an "sts:AssumeRole" or anything else. As in the former case it was kube2iam failed to assume the role (hence you need to check kube2iam's logs), while the latter means the role was assumed but lacking some privileges.

One of the likely reasons are setting up the re-routing rule for IMDS. The command curl 169.254.169.254/latest/meta-data/local-ipv4 will not work in AL2023, as it only accepts IMDSv2 queries [1] (requiring an access token to be used with the query, see [2]), while it worked in AL2.

To make it IMDSv2-aware, you can either use the "imds" utility (available on most AL2023 AMIs) or the method described on reference [2] in order to gain and use access token with your query.

I.e. getting the local IPV4 address, you can use one of the methods below:

LOCAL_IPV4=`imds latest/meta-data/local-ipv4`
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
LOCAL_IPV4=`curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-ipv4`

References:

[1] IMDSv2 - https://docs.aws.amazon.com/linux/al2023/ug/imdsv2.html

[2] How Instance Metadata Service Version 2 works - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-metadata-v2-how-it-works.html

AWS
SUPPORT ENGINEER
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.