gha-runner-scale-set-0.10.0

[Link-]

This release includes major improvements to the runner provisioning duration. In short, you should see less latency between queueing a workflow run and having a runner available to execute the job.

Make sure to check #3832 and #3848 for details on how to fine-tune that behavior.

Major changes

1. Add exponential backoff when generating runner reg tokens #3724
2. Make EphemeralRunnerController MaxConcurrentReconciles configurable #3832
3. Make EphemeralRunnerReconciler create runner pods earlier #3831
4. Make k8s client rate limiter parameters configurable #3848

Minor changes

1. Bump github.com/bradleyfalzon/ghinstallation/v2 from 2.8.0 to 2.12.0 #3837
2. Bump golang.org/x/crypto from 0.22.0 to 0.31.0 #3844
3. Update docs with details for the dashboard visualizations #3696

Full Changelog: gha-runner-scale-set-0.9.3...gha-runner-scale-set-0.10.0

---

This discussion was created from the release gha-runner-scale-set-0.10.0.

[Link-]

TLDR;

Thanks to contributions from @mumoshu @tfujiwar @int128 @ventsislav-georgiev and others, we shipped a couple of changes that will improve the duration between a workflow job queue and a runner being assigned to handle the job. This will also allow ARC to handle a higher amount of concurrent job runs.

Warning

The following requires advanced k8s knowledge and experience. Do not deploy these changes in a shared cluster. Always test & validate changes in a non-production environment. Follow the guidelines for upgrading ARC if you wish to enable these changes.

How to configure the MaxConcurrentReconciles and k8s client rate limiters

3 new flags have been introduced to the gha-runner-scale-set-controller helm chart:

1. runnerMaxConcurrentReconciles:
   

   actions-runner-controller/charts/gha-runner-scale-set-controller/values.yaml

   Line 112 in 59cb1d2

   runnerMaxConcurrentReconciles: 2

2. k8sClientRateLimiterQPS:
   

   actions-runner-controller/charts/gha-runner-scale-set-controller/values.yaml

   Line 140 in 59cb1d2

   # k8sClientRateLimiterQPS: 20

3. k8sClientRateLimiterBurst:
   

   actions-runner-controller/charts/gha-runner-scale-set-controller/values.yaml

   Line 141 in 59cb1d2

   # k8sClientRateLimiterBurst: 30

runnerMaxConcurrentReconciles

This will update the MaxConcurrentReconciles value for the ephemeral runner controller which will increase the number of go-routines processing the reconcile queue. The previous default value was 1 and now it's set to 2.

Keep in mind:

1. If you observe rate limiting events from the Kubernetes API or GitHub's APIs, you might want to revert this value to 1.
2. Higher values will also lead to higher CPU (and other resource consumption), so increase this number gradually and test the outcomes.
3. Test these changes during peak usage.

k8sClientRateLimiterQPS and k8sClientRateLimiterBurst

• k8sClientRateLimiterQPS (Queries Per Second): This parameter sets the maximum number of requests per second that the client can make to the Kubernetes API server. The default value is 20 QPS.
• k8sClientRateLimiterBurst: This parameter defines the maximum number of requests that can be made in a short burst. The default value is 30.

Depending on the configuration of your cluster, modifying these values could help further improve the overall ARC performance as it allows the controllers to make more requests to the k8s API server.

Modifying these values will not affect the limits imposed by the k8s API server! These are client side parameters.

Read more: https://kubernetes.io/docs/concepts/cluster-administration/flow-control/

[tfujiwar]

@Link- Hi. Thank you for including my #3848 in the release.

• k8sClientRateLimiterQPS (Queries Per Second): This parameter sets the maximum number of requests per second that the client can make to the Kubernetes API server. The default value is 5 QPS
• k8sClientRateLimiterBurst: This parameter defines the maximum number of requests that can be made in a short burst. The default value is 10

I think these are the default values of client-go but controller-runtime has another default values (20 QPS, 30 burst). I chose the latter as the default values of this controller.
#3833 (comment)

[Link-]

I'll update my message, good callout. I think in the next release, we should lower these to the defaults (5 and 10). Some folks might already be running hot on their cluster and this might be a problematic change for them.

Revert PR: #3853

[tfujiwar]

No, you don't need to revert them. Those parameters were already 20 QPS and 30 bursts before 0.10.0 because the main function calls ctrl.GetConfigOrDie() to get the config, which returns 20 QPS and 30 bursts.