Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support bound service account tokens #6136

Open
maxcao13 opened this issue Sep 4, 2024 · 9 comments · May be fixed by #6272
Open

Support bound service account tokens #6136

maxcao13 opened this issue Sep 4, 2024 · 9 comments · May be fixed by #6272
Labels
feature-request All issues for new features that have not been committed to needs-discussion

Comments

@maxcao13
Copy link
Contributor

maxcao13 commented Sep 4, 2024

Proposal

Currently, there exists the normal Secret Auth Provider which you can use to reference an embedded token for your scaler. Since Kubernetes 1.22, there is the Bound service account token feature that allows short-lived tokens to be mounted as projectedVolumes to your pods vs. long lived explicit tokens that you manually embed in a Secret.

I propose that Keda should support bound service account tokens as a new AuthenticationProvider (maybe called BoundServiceAccountToken or something).

I'm not exactly sure how this would be implemented but I was thinking that users would reference a ServiceAccount to create the token from, Keda calls the k8s TokenRequest API, and embeds that short-lived token in its own Secret. We would like to eliminate secrets completely here, but I'm not sure how we would store the state of an existing token otherwise. Very open to discussions and suggestions.

Use-Case

This would remove the extra step of the user explicitly creating a secret for a long-lived service account token and move away from legacy long-lived API tokens towards the k8s recommended TokenRequest API instead.

Here's a proof of concept of what I was thinking:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: bound-trigger
  namespace: myproject
spec:
  boundServiceAccountToken:
    - parameter: token
      serviceAccountName: mysa
      secretName: boundsecret
      expiry: 1h

parameter: identical to Secret and ConfigMap auth provider
serviceAccountName: name of the serviceAccount you want to generate the token from
secretName: name of a secret you want to store the short lived token in
expiry: when the token expires

Is this a feature you are interested in implementing yourself?

Yes

Anything else?

@maxcao13 maxcao13 added feature-request All issues for new features that have not been committed to needs-discussion labels Sep 4, 2024
@JorTurFer
Copy link
Member

JorTurFer commented Sep 5, 2024

Hello
I think that it's a good idea as projected volumes is the same approach that workload identity federations use but I don't see the use case as kubernetes tokens have to be verified against kubernetes API. Do your backends support auth-delegation with k8s api?

@maxcao13
Copy link
Contributor Author

maxcao13 commented Sep 5, 2024

Thanks for your response! To clarify, the use case that spawned this idea was autoscaling an Ingress Controller in OpenShift and in that context, we authenticate with Thanos via service accounts: link for more context. In this case our "backend" does support k8s (openshift) api verified tokens, but we are forced to create a long-lived service account secret.

The proposal is to replace this manual process with something more aligned with short-lived bound service account tokens, and I'm very open to hearing about alternative ideas on how to support this as a first step towards best practices. Is this something that should be worked on, and what do you think?

@JorTurFer
Copy link
Member

@zroubalik @wozniakjan

@zroubalik
Copy link
Member

Sorry, I missed this issue. I think it sounds reasonable, I am curious to hear if there any potential security issues or something that should concern us in general.

@maxcao13
Copy link
Contributor Author

maxcao13 commented Oct 3, 2024

Unfortunately the only objects you can bind the tokens to (as of now), are to pods and secrets. Since we are sort of trying to eliminate the use of secrets, and it doesn't make sense to bind to any pods, what if we "bind" generated tokens to a TriggerAuth that implements the boundServiceAccountToken auth instead, and implement a simple version of the token rotation, finalizers, etc. ourselves? And we embed the encoded token as an annotation in the TriggerAuth metadata (or somewhere else).

However, a user who can get/list [Cluster]TriggerAuths will also be able to see the embedded service account tokens, so there might be some extra permissions handling to do there. Instead of exposing the tokens in the TriggerAuth, keeping the tokens to be embedded in Secrets, but doing the renewal process on them, could also work. What do we think?

@maxcao13 maxcao13 linked a pull request Oct 24, 2024 that will close this issue
7 tasks
@joelsmith
Copy link
Contributor

Unfortunately the only objects you can bind the tokens to (as of now), are to pods and secrets. Since we are sort of trying to eliminate the use of secrets, and it doesn't make sense to bind to any pods, what if we "bind" generated tokens to a TriggerAuth that implements the boundServiceAccountToken auth instead, and implement a simple version of the token rotation, finalizers, etc. ourselves? And we embed the encoded token as an annotation in the TriggerAuth metadata (or somewhere else).

Doing our own object binding won't prevent an attacker who has obtained the token from using it until expiration, even if the corresponding TriggerAuth is removed, so I'm not sure that there is enough benefit in doing our own explicit binding. IMO, we should bind it to the keda-operator pod which requested the token and force a short expiration. I think we shouldn't expose the expiration time in our API, since the token is requested directly by KEDA and KEDA is its only user. If we select an expiration like 1 hour, then if the referencing TriggerAuth is removed, the token would not be renewed by us after its expiration.

@maxcao13
Copy link
Contributor Author

Doing our own object binding won't prevent an attacker who has obtained the token from using it until expiration, even if the corresponding TriggerAuth is removed, so I'm not sure that there is enough benefit in doing our own explicit binding. IMO, we should bind it to the keda-operator pod which requested the token and force a short expiration.

The new implementation in the PR gets rid of the binding aspect and ignores any form of token renewal before the expiry date. So once keda figures out that the token is actually expired, it will retry immediately with a new generated token with a new expiry.

I think we shouldn't expose the expiration time in our API, since the token is requested directly by KEDA and KEDA is its only user. If we select an expiration like 1 hour, then if the referencing TriggerAuth is removed, the token would not be renewed by us after its expiration.

Whether we have a short expiry or a new expiry, it shouldn't? affect the user's workflow. So the tradeoff is either security (shorter expirys mean less time for attackers to actually use the token), or API calls (since we have short expiry, there will be a high frequency of token creation). Maybe the user should still be able to specify which tradeoff they want? WDWT?

@joelsmith
Copy link
Contributor

Whether we have a short expiry or a new expiry, it shouldn't? affect the user's workflow. So the tradeoff is either security (shorter expirys mean less time for attackers to actually use the token), or API calls (since we have short expiry, there will be a high frequency of token creation). Maybe the user should still be able to specify which tradeoff they want? WDWT?

Hmmm... yes, that is a trade-off, but maybe it's one that the cluster admin should be making, not cluster users. In other words, maybe it would make most sense to make it globally configurable as a command-line switch for KEDA. I don't feel strongly about it though.

@maxcao13
Copy link
Contributor Author

maxcao13 commented Nov 29, 2024

Although, now that I think about it, there is no way to invalidate the token sooner like what happens when you bind it to a pod or a service account where you can just delete the pod/secret. Implementing this seems a bit more complicated than needed, so maybe setting a really short expiry (or range of short expiries so they we don't spike API requests) is "enough"?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request All issues for new features that have not been committed to needs-discussion
Projects
Status: To Triage
Development

Successfully merging a pull request may close this issue.

4 participants