KEP-4977: Dedicated Priority Level For Event Requests

[linxiulei]

• One-line PR description: Dedicated Priority Level For Event Requests

• Issue link: Dedicated priority level for event requests #4977

• Other comments:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: linxiulei
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/sig-api-machinery/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[linxiulei]

/cc @MikeSpreitzer @tkashem @wojtek-t

[wojtek-t]

If we take a cluster (let's even say a scalability test) - what is a typical percentage of seats that events consume?
I would like to avoid a situation that we suddenly start visibly throttling events without good reason...

FWIW @serathius created an internal GKE dashboard for debugging cases like that, so he might be able to help with answering this question

[serathius]

Don't have a number on hand for a scalability test, but I got 4% APF used for events from aggregation of multiple tests we run.

[wojtek-t]

The aggregation of different test sounds even better - thanks!

[wojtek-t]

So following up on it. Currently, the sum of all shares in default configuration across all PLs is 245, with this change it will be bumped to 250. With 5 shares by default and 100% borrowing limit we get a max of 10.
This effectively mean 10 out of 250, which is exactly 4%.

If we say that in tests we see 4% used for events across aggregated tests, I believe we need to set the upper bound higher (probably by increasing the borrowing limit?)

@MikeSpreitzer - for your thoughts too

[MikeSpreitzer]

I agree with @wojtek-t

[wojtek-t]

And the comment that I brought up in some different place we were discussing it.

I would like to understand how we're going to safely roll this out, given we will effectively change APF configuration for (almost) all clusters in the world.

[MikeSpreitzer]

There are some attributes omitted here. You might compare how I filled them in in kubernetes/kubernetes#128974

[linxiulei]

Do you mean the weight and queue?

I can add weight if you believe the weighted borrow will be landed before this KEP, which I am more than happy to see. As for queue, I am inclined to make it reject right away given that the volume of events may easily overwhelm any queue size we will specify here.