Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add 100-node higher QPS limit scheduler job #20023

Merged
merged 2 commits into from
Dec 21, 2020
Merged

add 100-node higher QPS limit scheduler job #20023

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2d55b91
add 100-node higher QPS limit scheduler job
Nov 23, 2020
0204e3e
100Nodes-scheduler: reduce timeout to 150 minutes
Dec 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 68 additions & 2 deletions config/jobs/kubernetes/sig-scalability/sig-scalability-periodic-jobs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ periodics:
- --repo=k8s.io/kubernetes=master
- --repo=k8s.io/perf-tests=master
- --root=/go/src
- --timeout=260
- --timeout=170
- --scenario=kubernetes_e2e
- --
- --cluster=kubemark-100
Expand Down Expand Up @@ -219,7 +219,73 @@ periodics:
- --test-cmd-args=--testoverrides=./testing/experiments/enable_restart_count_check.yaml
- --test-cmd-args=--testoverrides=./testing/experiments/use_simple_latency_query.yaml
- --test-cmd-name=ClusterLoaderV2
- --timeout=240m
- --timeout=150m
- --use-logexporter
# docker-in-docker needs privileged mode
securityContext:
privileged: true
resources:
requests:
cpu: 2
memory: "2Gi"
limits:
cpu: 2
memory: "2Gi"

- name: ci-kubernetes-kubemark-100-gce-scheduler-highqps
tags:
- "perfDashPrefix: kubemark-100Nodes-scheduler-highqps"
- "perfDashJobType: performance"
interval: 24h
labels:
preset-service-account: "true"
preset-k8s-ssh: "true"
preset-dind-enabled: "true"
preset-e2e-kubemark-common: "true"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with starting with kubemark, but eventually we should consider migrating to real clusters. Kubemark is still visibly different than real clusters (easier from the point-of-view of control-plane).

preset-e2e-scalability-periodics: "true"
preset-e2e-scalability-periodics-master: "true"
annotations:
testgrid-dashboards: sig-scalability-kubemark
testgrid-tab-name: kubemark-100-scheduler-highqps
testgrid-num-failures-to-alert: '1'
spec:
containers:
- image: gcr.io/k8s-testimages/kubekins-e2e:v20201111-a263fd7-master
args:
- --repo=k8s.io/kubernetes=master
- --repo=k8s.io/perf-tests=master
- --root=/go/src
- --timeout=170
- --scenario=kubernetes_e2e
- --
- --cluster=kubemark-100-scheduler-highqps
- --build=bazel
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason to be checking out kubernetes and doing a source build here?
most CI jobs consume our existing builds (--extract=ci/latest or similar)

this is cheaper and faster

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really, but IIRC to test the PR I had to build from my fork and I must have forgotten to change back to --extract=ci/latest. I'll open a PR to switch to reuse builds.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack thanks!
I'm working on switching over how we build (bazel=>quick) but suspected this one shouldn't be building anyhow

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#21062 covers this

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @BenTheElder!

- --extract=bazel
- --gcp-master-size=e2-standard-2
- --gcp-node-image=gci
- --gcp-node-size=e2-standard-4
- --gcp-nodes=4
- --gcp-project-type=scalability-project
- --gcp-zone=us-east1-b
- --kubemark
- --kubemark-nodes=100
- --provider=gce
- --env=CONTROLLER_MANAGER_TEST_ARGS=--profiling --kube-api-qps=300 --kube-api-burst=300
- --env=SCHEDULER_TEST_ARGS=--profiling --kube-api-qps=300 --kube-api-burst=300
- --env=CL2_ENABLE_CLUSTER_OOMS_TRACKER=true
- --test=false
- --test_args=--ginkgo.focus=xxxx
- --test-cmd=$GOPATH/src/k8s.io/perf-tests/run-e2e.sh
- --test-cmd-args=cluster-loader2
- --test-cmd-args=--nodes=100
- --test-cmd-args=--prometheus-scrape-node-exporter=true
- --test-cmd-args=--provider=kubemark
- --test-cmd-args=--report-dir=/workspace/_artifacts
- --test-cmd-args=--testsuite=testing/density/scheduler-suite.yaml
- --test-cmd-args=--testoverrides=./testing/experiments/enable_restart_count_check.yaml
- --test-cmd-args=--testoverrides=./testing/experiments/use_simple_latency_query.yaml
- --test-cmd-name=ClusterLoaderV2
- --timeout=150m
- --use-logexporter
# docker-in-docker needs privileged mode
securityContext:
Expand Down