Parametrize DRA test config #3654

emerbe · Oct 23, 2025

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

This PR modifies DRA test logic a bit so it's parametrized to simplify running test with different drivers and with different scale.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

k8s-ci-robot · Oct 23, 2025

Hi @emerbe. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · Oct 23, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: emerbe
Once this PR has been reviewed and has the lgtm label, please assign marseel for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

clusterloader2/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

emerbe · Oct 23, 2025

/assign @alaypatel07

clusterloader2/testing/dra/config.yaml

alaypatel07 · Oct 30, 2025

clusterloader2/testing/dra/config.yaml

 {{$LOAD_TEST_THROUGHPUT := DefaultParam .CL2_LOAD_TEST_THROUGHPUT 10}}
 {{$STEADY_STATE_QPS := DefaultParam .CL2_STEADY_STATE_QPS 5}}
+{{$RESOURCE_SLICES_PER_NODE := DefaultParam .CL2_RESOURCE_SLICES_PER_NODE 1}}
+{{$UPSIZE_THRESHOLD := DefaultParam .CL2_UPSIZE_THRESHOLD "10m"}}


This value does not have any impact except from printing the threshold value. Why is the change needed?

Those thresholds define the acceptable limits for the pod startup times.

The initial values in this PT were too stretched for 100 nodes but in bigger scale I observed that those need to be higher. Even for 100 nodes in GKE 5 seconds is not enough:

E1102 21:24:30.207908 25461 clusterloader.go:258] Errors: [measurement call PodStartupLatency - FastFillPodStartupLatency error: pod startup: too high latency 99th percentile: got 8.792346318s expected: 5s]

I've been using this test with different than drivers and observed that times needed for different measurements may vary.

alaypatel07 · Oct 30, 2025

clusterloader2/testing/dra/config.yaml

 # Node resource configuration
 {{$gpusPerNode := DefaultParam .CL2_GPUS_PER_NODE 8}}
+{{$resourceSlicesPerNode := DefaultParam .CL2_RESOURCE_SLICES_PER_NODE 1}}
+{{$workerNodeCount := MultiplyInt $resourceSlicesPerNode .Nodes}}


nit: should this be named $totalResourceSliceCount?

alaypatel07 · Oct 30, 2025

clusterloader2/testing/dra/config.yaml

 {{$namespaces := DivideInt .Nodes $NODES_PER_NAMESPACE}}

+# dra
+{{$draNamespace := DefaultParam .CL2_DRA_NAMESPACE "dra-example-driver"}}


whats the point of exposing these as env variables?

Good point. I removed it and made namepsace name in dra.go generic

emerbe

@alaypatel07 answered your comments. PTAL

emerbe · Oct 31, 2025

clusterloader2/testing/dra/config.yaml

 {{$namespaces := DivideInt .Nodes $NODES_PER_NAMESPACE}}

+# dra
+{{$draNamespace := DefaultParam .CL2_DRA_NAMESPACE "dra-example-driver"}}


Good point. I removed it and made namepsace name in dra.go generic

emerbe · Oct 31, 2025

clusterloader2/testing/dra/config.yaml

 # Node resource configuration
 {{$gpusPerNode := DefaultParam .CL2_GPUS_PER_NODE 8}}
+{{$resourceSlicesPerNode := DefaultParam .CL2_RESOURCE_SLICES_PER_NODE 1}}
+{{$workerNodeCount := MultiplyInt $resourceSlicesPerNode .Nodes}}


clusterloader2/testing/dra/config.yaml

emerbe · Oct 31, 2025

clusterloader2/testing/dra/config.yaml

 {{$LOAD_TEST_THROUGHPUT := DefaultParam .CL2_LOAD_TEST_THROUGHPUT 10}}
 {{$STEADY_STATE_QPS := DefaultParam .CL2_STEADY_STATE_QPS 5}}
+{{$RESOURCE_SLICES_PER_NODE := DefaultParam .CL2_RESOURCE_SLICES_PER_NODE 1}}
+{{$UPSIZE_THRESHOLD := DefaultParam .CL2_UPSIZE_THRESHOLD "10m"}}


Those thresholds define the acceptable limits for the pod startup times.

The initial values in this PT were too stretched for 100 nodes but in bigger scale I observed that those need to be higher. Even for 100 nodes in GKE 5 seconds is not enough:

E1102 21:24:30.207908 25461 clusterloader.go:258] Errors: [measurement call PodStartupLatency - FastFillPodStartupLatency error: pod startup: too high latency 99th percentile: got 8.792346318s expected: 5s]

I've been using this test with different than drivers and observed that times needed for different measurements may vary.

alaypatel07 · Nov 3, 2025

clusterloader2/pkg/dependency/dra/dra.go

 }

 func isResourceSlicesPublished(config *dependency.Config, namespace string) (bool, error) {
-	// Get a list of all nodes


not sure why changes in this file are needed.

I've noticed that I haven't deleted commented code in the previous PR :/

clusterloader2/pkg/dependency/dra/dra.go

alaypatel07 · Nov 5, 2025

clusterloader2/testing/dra/config.yaml

+{{$CHURN_POD_STARTUP_PERC50_THRESHOLD := DefaultParam .CL2_CHURN_POD_STARTUP_PERC50_THRESHOLD "5s"}}
+{{$CHURN_POD_STARTUP_PERC90_THRESHOLD := DefaultParam .CL2_CHURN_POD_STARTUP_PERC90_THRESHOLD "5s"}}
+{{$CHURN_POD_STARTUP_PERC99_THRESHOLD := DefaultParam .CL2_CHURN_POD_STARTUP_PERC99_THRESHOLD "5s"}}
+{{$FINISHED_JOBS_THRESHOLD := DefaultParam .CL2_FINISHED_JOBS_THRESHOLD "10m"}}


I think CL2_LONG_LIVED_JOBS_WAIT_THRESHOLD and CL2_CHURN_JOBS_WAIT_THRESHOLD will good env variable names for this.

Also other places where we refer to these jobs in variable names can be changed for example:

{{$smallJobCompletions := DefaultParam .CL2_SMALL_JOB_COMPLETIONS 10}}

clusterloader2/testing/dra/config.yaml

mengqiy · Nov 6, 2025

clusterloader2/testing/dra/config.yaml

 {{$LOAD_TEST_THROUGHPUT := DefaultParam .CL2_LOAD_TEST_THROUGHPUT 10}}
 {{$STEADY_STATE_QPS := DefaultParam .CL2_STEADY_STATE_QPS 5}}
+{{$RESOURCE_SLICES_PER_NODE := DefaultParam .CL2_RESOURCE_SLICES_PER_NODE 1}}
+{{$UPSIZE_THRESHOLD := DefaultParam .CL2_UPSIZE_THRESHOLD "5s"}}


It's a bit unclear about the meaning of this when we have UPSIZE_PERC50_THRESHOLD and UPSIZE_PERC90_THRESHOLD

That's absolute threshold.

I've changed name to: LONG_LIVED_JOBS_STARTUP_PERC100_THRESHOLD

mengqiy · Nov 6, 2025

clusterloader2/testing/dra/config.yaml

+{{$UPSIZE_PERC90_THRESHOLD := DefaultParam .CL2_UPSIZE_PERC90_THRESHOLD "5s"}}
+{{$CHURN_POD_STARTUP_PERC50_THRESHOLD := DefaultParam .CL2_CHURN_POD_STARTUP_PERC50_THRESHOLD "5s"}}
+{{$CHURN_POD_STARTUP_PERC90_THRESHOLD := DefaultParam .CL2_CHURN_POD_STARTUP_PERC90_THRESHOLD "5s"}}
+{{$CHURN_POD_STARTUP_PERC99_THRESHOLD := DefaultParam .CL2_CHURN_POD_STARTUP_PERC99_THRESHOLD "5s"}}


It doesn't make a lot of sense when we use the same default value for P50, P90 and P99.

Those are defaults for 100 nodes that can be changed for larger scale.

k8s-ci-robot added kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 23, 2025

k8s-ci-robot requested review from mborsz and wojtek-t October 23, 2025 12:36

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 23, 2025

k8s-ci-robot assigned alaypatel07 Oct 23, 2025

emerbe force-pushed the update-dra-test-config branch 3 times, most recently from 65b6445 to 93485a2 Compare October 24, 2025 17:40

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 30, 2025

emerbe force-pushed the update-dra-test-config branch from 93485a2 to e231a1a Compare October 30, 2025 15:53

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 30, 2025

emerbe force-pushed the update-dra-test-config branch 3 times, most recently from 7128f8c to e7621d2 Compare October 30, 2025 16:02

alaypatel07 reviewed Oct 30, 2025

View reviewed changes

clusterloader2/testing/dra/config.yaml Outdated Show resolved Hide resolved

alaypatel07 reviewed Oct 30, 2025

View reviewed changes

emerbe force-pushed the update-dra-test-config branch from e7621d2 to d405c3e Compare November 2, 2025 21:46

emerbe commented Nov 2, 2025

View reviewed changes

alaypatel07 reviewed Nov 3, 2025

View reviewed changes

alaypatel07 reviewed Nov 4, 2025

View reviewed changes

clusterloader2/pkg/dependency/dra/dra.go Outdated Show resolved Hide resolved

emerbe force-pushed the update-dra-test-config branch from d405c3e to cba8b3d Compare November 5, 2025 12:53

alaypatel07 reviewed Nov 5, 2025

View reviewed changes

clusterloader2/testing/dra/config.yaml Outdated Show resolved Hide resolved

emerbe force-pushed the update-dra-test-config branch from cba8b3d to b8380d8 Compare November 6, 2025 09:11

mengqiy reviewed Nov 6, 2025

View reviewed changes

emerbe force-pushed the update-dra-test-config branch 3 times, most recently from 194c6c7 to c2d5837 Compare November 6, 2025 20:48

Parametrize DRA test config

10666bf

emerbe force-pushed the update-dra-test-config branch from c2d5837 to 10666bf Compare November 6, 2025 22:07

alaypatel07 mentioned this pull request Dec 1, 2025

Reduce periodicity in DRA Extended Resources scalability job kubernetes/test-infra#35612

Closed

Search code, repositories, users, issues, pull requests...

Parametrize DRA test config #3654

Are you sure you want to change the base?

Parametrize DRA test config #3654

Uh oh!

Conversation

emerbe commented Oct 23, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Uh oh!

k8s-ci-robot commented Oct 23, 2025

Uh oh!

k8s-ci-robot commented Oct 23, 2025

Uh oh!

emerbe commented Oct 23, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emerbe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emerbe Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

emerbe Nov 6, 2025 •

edited

Loading