til
← All writing

EKS access entries are not IAM

6 min read tilawsekskubernetesiam

Spent a day chasing a 401 Unauthorized from a cross-account EKS cluster and came out the other side with a mental model I wish someone had handed me up front: access entries and IAM roles answer two completely different questions, and once you see which question each one answers, a bunch of confusing cross-account behavior becomes obvious.

Setup: a management account (call it 111111111111) runs a control-plane cluster with our platform operators and ArgoCD on it. Workload accounts (call one 222222222222) hold the clusters those operators manage. An operator pod needed to write one Secret into a workload cluster. Should be easy.

the 401 that started it

The operator built its Kubernetes client from the ArgoCD cluster secret — ArgoCD deploys to these clusters all day, so its secret must contain working credentials, right? For EKS clusters it doesn’t, at least not directly:

{
  "awsAuthConfig": {
    "clusterName": "wc-1",
    "roleARN": "arn:aws:iam::111111111111:role/argocd-mgmt"
  },
  "tlsClientConfig": { "caData": "..." }
}

No token. Just a role ARN and a CA. ArgoCD authenticates by assuming that role and minting a presigned STS token per request. Our code copied the bearer/cert fields out of the config — all empty — and happily built a client with no identity at all. The apiserver returned 401 before IAM was ever consulted.

I stared at the cluster’s access entries for a long time. They were fine. That was the first lesson: a 401 means “I don’t know who you are”, not “you’re not allowed”. No amount of access-entry fiddling fixes an anonymous client.

access entries take principals from any account

Second surprise. The failing cluster already had this entry, created by Terraform:

principal: arn:aws:iam::111111111111:role/argocd-mgmt   # mgmt account!
policy:    AmazonEKSClusterAdminPolicy

A role from the management account, granted admin on a cluster in the workload account. No trust policy in the workload account, no mirrored role, nothing. And it works, because an access entry is not IAM — it’s the cluster’s own authorization list (the successor to the aws-auth ConfigMap). The EKS token is just a presigned sts:GetCallerIdentity; the apiserver calls STS, learns “this caller is role X from account Y”, and looks that ARN up in its entries. The ARN is matched as a string. Any account’s principal can appear there.

I didn’t quite believe it until I did it by hand:

$ aws eks create-access-entry --cluster-name wc-1 \
    --principal-arn arn:aws:iam::111111111111:role/operator-mgmt
{
    "accessEntry": {
        "principalArn": "arn:aws:iam::111111111111:role/operator-mgmt",
        "username": "arn:aws:sts::111111111111:assumed-role/operator-mgmt/{{SessionName}}",
        ...
    }
}

Cross-account, accepted, works. (Fun trap: the AWS console can’t do this — its principal picker only lists roles in the cluster’s own account, which had me convinced for a while that cross-account entries weren’t a thing. The API takes any ARN.)

So wait — one role in one account, add it as an access entry everywhere, done? What are per-account roles even for?

the two questions

Here’s the split that makes everything click:

  • “May you assume this role?” — answered by IAM, specifically the role’s trust policy, in the role’s own account. Our operator role trusts the control-plane cluster’s OIDC issuer for system:serviceaccount:platform:*. That’s ordinary IRSA and it never changes in this story.
  • “May you talk to this cluster’s Kubernetes API?” — answered by the cluster, via its access entries. Not by IAM. The principal can live anywhere; whoever holds eks:CreateAccessEntry in the cluster’s account decides who gets in. That’s the security boundary — the cluster’s owner invites you, same as writing you into aws-auth used to be.

Neither question is “may you call AWS APIs in the cluster’s account” — and that third question is what dedicated per-account roles are for. Which mattered immediately, because of one sneaky API call:

DescribeCluster is the catch

To build an EKS client from scratch you need the endpoint URL and cluster CA, and the natural way to get them is eks:DescribeCluster. That is an AWS API call in the cluster’s account. A management-account role cannot make it — there are no resource policies on EKS; IAM in the workload account only honors workload-account principals.

Which explains why ArgoCD never hits this: its cluster secret has the endpoint and CA pre-baked at registration time (written by the Terraform that created the cluster, which ran with workload-account credentials). ArgoCD just mints its STS token with its home-account role and dials the stored endpoint. It never discovers anything, so it never needs foreign IAM. One home-account role plus one access entry per cluster, forever.

Our operator had no such secret to lean on — and here’s the part that killed the “just read ArgoCD’s secret” workaround: the secret for a given cluster lives on whichever cluster hosts that cluster’s ArgoCD. In a hub-and-spoke platform that’s not where your operator runs. Reading it from the root would have meant: fetch credentials for the mid-tier cluster, read the leaf cluster’s secret off it, then finally build the client. Three hops, each with its own failure modes, coupled to ArgoCD’s internal storage format forever.

The alternative is one flat hop: a role in the workload account whose trust policy points back at the control-plane cluster’s OIDC issuer (register the issuer as an IAM OIDC provider in the workload account, trust the operator’s service account subject). The operator’s pod token assumes it directly via web identity. Those credentials can DescribeCluster and authenticate to the kube-API — self-sufficient, no secret chasing. In our case the role already existed, because the thing that provisions clusters in that account needs exactly the same kind of role.

the rule I’m keeping

Two patterns, both correct, pick by one question — who supplies the endpoint and CA?

  1. Handed to you at registration (the ArgoCD model): keep your home-account role, add it as an access entry on each cluster. No IAM in the target account, ever. Works across accounts indefinitely.
  2. You must discover it yourself (the controller/operator model): you need an in-account role anyway for DescribeCluster, so use it for the access entry too. One credential does discovery and auth.

And the corollaries that saved me hours, written down so I stop re-deriving them:

  • 401 from EKS = your client is anonymous. Check what credentials you’re actually sending before touching access entries.
  • ArgoCD EKS cluster secrets contain no static credentials — awsAuthConfig names ArgoCD’s role, which your other components probably can’t assume. Don’t borrow the secret unless you can borrow the identity.
  • Access entries: cluster-scoped, any account’s principal, created with cluster-account permissions, console picker lies about cross-account.
  • Per-account IAM roles: for AWS APIs in that account (provisioning, ECR, SSM, DescribeCluster) — and for the softer reasons too: the account owner can revoke them unilaterally, their CloudTrail sees the AssumeRole, their SCPs apply.