Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatibility with EKS 1.21 and token service account rotation #44

Open
costap opened this issue May 13, 2022 · 0 comments
Open

Compatibility with EKS 1.21 and token service account rotation #44

costap opened this issue May 13, 2022 · 0 comments

Comments

@costap
Copy link

costap commented May 13, 2022

What happened:
After our EKS was upgraded to 1.21, we saw annotations like the following appear in api server audit logs in AWS, for service accounts that Splunk Connect pods are using:

subject: system:serviceaccount:<namespace here>:<sa name here>, seconds after warning threshold: 3989

This is due to changes in token expiry in K8s 1.21 as described here:
https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html#identify-pods-using-stale-tokens

It would appear that there is 90d grace period, after which tokens will be rejected.
It looks like the solarwinds snap agents needs to use a later client SDK version, or is there a workaround?

What you expected to happen:
More recent k8s client sdk was used so that the tokens would be refreshed. At some kube version when AWS will change to the default 1h tokens, the pods will get errors from api server after an hour (unless they are restarted earlier, as that would refresh the token I think as well).

How to reproduce it (as minimally and precisely as possible):
Install or upgrade EKS to 1.21 and check EKS cluster api server audit logs with this query:

fields @timestamp
| filter @logStream like /kube-apiserver-audit/
| filter @message like /seconds after warning threshold/
| parse @message "subject: *, seconds after warning threshold:*\"" as subject, elapsedtime

based on: https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html#identify-pods-using-stale-tokens

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version): 1.21
Image tag: solarwinds/solarwinds-snap-agent-docker:4.4.0-4.3.0.1156 (latest)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant