add 100-node higher QPS limit scheduler job

[adtac]

/hold until kubernetes/kubernetes#96813 is merged

So sig/scheduling has made numerous improvements to the scheduler over the past few releases, but these improvements have not been realised in practice due to the scheduler's QPS limits. With API priority and fairness being graduated to beta in 1.20 (kubernetes/kubernetes#96527), we now have a path towards safely removing the scheduler's rate limiting in the 1.21+.

Of course, this would need to be done slowly. @ahg-g and I discussed this offline and we believe that increasing the scheduler's QPS limit (not removing) in 1.21 is the first step. Once things are observed to be stable over a couple of releases, the client-side rate limiting will be entirely limit and kube-scheduler will depend on APF to make sure it doesn't overload the API server.

While I have done some preliminary benchmarking of APF with higher scheduler QPS limits (not public yet, will share with everyone in a short while), there would be more confidence in this approach if there were public, periodic benchmarks that run on test-infra. The results could be displayed in perf-dash. These benchmarks should run on the higher QPS limit. The job should be carefully observed for signs of instability. The job would be temporary and is only expected to run for a 3-4 releases; once the scheduler's client-side QPS limits are removed, just the regular job (ci-kubernetes-kubemark-100-gce) would be sufficient.

A separate job is needed to do this because all changes happen during cluster creation. As a result, existing jobs cannot be re-used for this.

/sig scheduling
cc @ahg-g

[adtac]

@wojtek-t updated this PR to use --env=CONTROLLER_MANAGER_TEST_ARGS and --env=SCHEDULER_TEST_ARGS (ref: kubernetes/kubernetes#96813 (comment))

[adtac]

/assign @wojtek-t

[wojtek-t]

I'm fine with starting with kubemark, but eventually we should consider migrating to real clusters. Kubemark is still visibly different than real clusters (easier from the point-of-view of control-plane).

[wojtek-t]

@adtac - next time please ping me if I'm not responding for week+.

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adtac, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/jobs/kubernetes/sig-scalability/OWNERS [wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[adtac]

/hold cancel

[k8s-ci-robot]

@adtac: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key sig-scalability-periodic-jobs.yaml using file config/jobs/kubernetes/sig-scalability/sig-scalability-periodic-jobs.yaml

In response to this:

/hold until kubernetes/kubernetes#96813 is merged

So sig/scheduling has made numerous improvements to the scheduler over the past few releases, but these improvements have not been realised in practice due to the scheduler's QPS limits. With API priority and fairness being graduated to beta in 1.20 (kubernetes/kubernetes#96527), we now have a path towards safely removing the scheduler's rate limiting in the 1.21+.

Of course, this would need to be done slowly. @ahg-g and I discussed this offline and we believe that increasing the scheduler's QPS limit (not removing) in 1.21 is the first step. Once things are observed to be stable over a couple of releases, the client-side rate limiting will be entirely limit and kube-scheduler will depend on APF to make sure it doesn't overload the API server.

While I have done some preliminary benchmarking of APF with higher scheduler QPS limits (not public yet, will share with everyone in a short while), there would be more confidence in this approach if there were public, periodic benchmarks that run on test-infra. The results could be displayed in perf-dash. These benchmarks should run on the higher QPS limit. The job should be carefully observed for signs of instability. The job would be temporary and is only expected to run for a 3-4 releases; once the scheduler's client-side QPS limits are removed, just the regular job (ci-kubernetes-kubemark-100-gce) would be sufficient.

A separate job is needed to do this because all changes happen during cluster creation. As a result, existing jobs cannot be re-used for this.

/sig scheduling
cc @ahg-g

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

is there a reason to be checking out kubernetes and doing a source build here?
most CI jobs consume our existing builds (--extract=ci/latest or similar)

this is cheaper and faster

[adtac]

not really, but IIRC to test the PR I had to build from my fork and I must have forgotten to change back to --extract=ci/latest. I'll open a PR to switch to reuse builds.

[BenTheElder]

Ack thanks!
I'm working on switching over how we build (bazel=>quick) but suspected this one shouldn't be building anyhow

[BenTheElder]

#21062 covers this

[adtac]

thanks @BenTheElder!