Add CPUManager policy option to distribute CPUs across NUMA nodes instead of packing them #105631

klueska · 2021-10-12T10:32:32Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR implements the logic describe in KEP-2902.

It adds a new CPUManager policy option called distribute-cpus-across-numa to the static CPUManager policy. When enabled, this will trigger the CPUManager to evenly distribute CPUs across NUMA nodes in cases where more than one NUMA node is required to satisfy the allocation.

Does this PR introduce a user-facing change?

Add a new `distribute-cpus-across-numa` option to the static `CPUManager` policy. When enabled, this will trigger the `CPUManager` to evenly distribute CPUs across NUMA nodes in cases where more than one NUMA node is required to satisfy the allocation.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2902-cpumanager-distribute-cpus-policy-option

klueska · 2021-10-12T10:32:47Z

/sig node
/hold

k8s-ci-robot · 2021-10-12T10:33:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: klueska

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/cm/cpumanager/OWNERS~~ [klueska]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

klueska · 2021-10-12T10:34:29Z

/triage accepted

klueska · 2021-10-12T10:37:19Z

/cc @fromanirh @swatisehgal

The core algorithm hasn't been implemented yet, but the base infrastructure is now in place.

Please take note of:

This PR currently includes changes from #102015 which have not been merged yet. Please ignore the overlapping commits from this PR when reviewing.

klueska · 2021-10-13T11:13:15Z

@fromanirh @swatisehgal
Added logic to implement the algorithm itself. Tests to come next.

This parameter ensures that CPUs are always allocated in groups of size 'cpuGroupSize'. This is important, for example, to ensure that all CPUs (i.e. hyperthreads) from the same core are handed out together. Signed-off-by: Kevin Klues <kklues@nvidia.com>

Signed-off-by: Kevin Klues <kklues@nvidia.com>

ffromani

again a batch of minor comments, mostly for future cleanups (when we move to beta or so). The algorithm itself is nicely written and very nicely commented. I want to review the tests carefully today/tomorrow, and if everything goes as I expect, I'll add my LGTM.

ffromani · 2021-10-18T07:40:03Z

pkg/kubelet/cm/cpumanager/cpu_assignment.go

@@ -264,6 +308,25 @@ func (a *cpuAccumulator) isFailed() bool {
 	return a.numCPUsNeeded > a.details.CPUs().Size()
 }

+func (a *cpuAccumulator) iterateCombinations(n []int, k int, f func([]int)) {


nit: this can (and probably should) be a free function as well, because it doesn't use cpuAccumulator fields at all. Not worth changing now, let's just not it down for the future refactoring.

ffromani · 2021-10-18T07:55:17Z

pkg/kubelet/cm/cpumanager/cpu_assignment.go

+type LoopControl bool
+
+const (
+	Continue LoopControl = true


minor: using bool here raise a flag. Could we use a naked bool maybe? or should we just move to a enum using integer?

I'm fine with changing this to an enum. It's only a bool, because my first iteration of this didn't use constants at all -- I had the functions directly returning a bool, with true for continue and false for break.

This can totally wait for the refactoring, but it seems to suggest a plain bool could be simpler and equally readable

klueska · 2021-10-18T08:50:04Z

@fromanirh

nit: this can (and probably should) be a free function as well, because it doesn't use cpuAccumulator fields at all. Not worth changing now, let's just not it down for the future refactoring.

Yes, as I pointed out here, I struggled with where to put this because I wanted it namespaced somehow. It ill be pulled out to a top-level function once we move to a cpassignment package.

Signed-off-by: Kevin Klues <kklues@nvidia.com>

ffromani · 2021-10-18T08:53:59Z

@fromanirh

nit: this can (and probably should) be a free function as well, because it doesn't use cpuAccumulator fields at all. Not worth changing now, let's just not it down for the future refactoring.

Yes, as I pointed out here, I struggled with where to put this because I wanted it namespaced somehow. It ill be pulled out to a top-level function once we move to a cpassignment package.

Sure, I was recording them all. I understand the issue at hand and I'm actually pushing (slightly :) ) to wait and address all these nits in the future cleanup work.

klueska · 2021-10-18T12:18:48Z

Based on the feedback above, I am removing the hold on this PR. Just need an lgtm from here then.
/unhold

ffromani · 2021-10-18T13:59:35Z

/test pull-kubernetes-integration

swatisehgal · 2021-10-19T14:47:18Z

/lgtm
/hold
@fromanirh please remove hold when you are happy with the PR

ffromani · 2021-10-19T16:51:37Z

/lgtm
/hold cancel
This PR is in great shape, much better than it's expected for alpha level. Thanks @klueska !

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. sig/node Categorizes an issue or PR as relevant to SIG Node. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 12, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 12, 2021

k8s-ci-robot requested review from krmayankk and mrunalp October 12, 2021 10:33

k8s-ci-robot added the area/kubelet label Oct 12, 2021

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 12, 2021

k8s-ci-robot requested review from ffromani and swatisehgal October 12, 2021 10:37

klueska force-pushed the upstream-distribute-cpus-across-numa branch 5 times, most recently from 357ba1d to 6f155d0 Compare October 13, 2021 12:27

k8s-ci-robot removed the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 13, 2021

klueska added 2 commits October 16, 2021 19:31

Support full-pcpus-only with the new NUMA distribution policy option

70e0f47

Signed-off-by: Kevin Klues <kklues@nvidia.com>

klueska force-pushed the upstream-distribute-cpus-across-numa branch 7 times, most recently from 002a080 to d9725e6 Compare October 17, 2021 14:23

ffromani reviewed Oct 18, 2021

View reviewed changes

Add optimizations to reduce iterations in distributed NUMA algorithm

86f9c26

Signed-off-by: Kevin Klues <kklues@nvidia.com>

klueska force-pushed the upstream-distribute-cpus-across-numa branch from d9725e6 to 86f9c26 Compare October 18, 2021 08:54

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 18, 2021

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 19, 2021

k8s-ci-robot assigned swatisehgal Oct 19, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 19, 2021

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 19, 2021

k8s-ci-robot assigned ffromani Oct 19, 2021

k8s-ci-robot merged commit b2c4269 into kubernetes:master Oct 19, 2021

SIG Node PR Triage automation moved this from Needs Reviewer to Done Oct 19, 2021

k8s-ci-robot added this to the v1.23 milestone Oct 19, 2021

klueska mentioned this pull request Oct 19, 2021

Add CPUManager policy option to distribute CPUs across NUMA nodes instead of packing them kubernetes/enhancements#2902

Open

4 tasks

Rajalakshmi-Girish mentioned this pull request Nov 20, 2021

[Flaky Test][sig-node] kubernetes-unit-test TestTakeByTopologyNUMADistributed is being Flaky #106571

Closed

klueska mentioned this pull request Nov 22, 2021

Fix Bugs in CPUManager distribute NUMA policy option #106599

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CPUManager policy option to distribute CPUs across NUMA nodes instead of packing them #105631

Add CPUManager policy option to distribute CPUs across NUMA nodes instead of packing them #105631

klueska commented Oct 12, 2021 •

edited

klueska commented Oct 12, 2021

k8s-ci-robot commented Oct 12, 2021

klueska commented Oct 12, 2021

klueska commented Oct 12, 2021

klueska commented Oct 13, 2021

ffromani left a comment

ffromani Oct 18, 2021

ffromani Oct 18, 2021

klueska Oct 18, 2021

ffromani Oct 18, 2021

klueska Oct 18, 2021

klueska commented Oct 18, 2021 •

edited

ffromani commented Oct 18, 2021

klueska commented Oct 18, 2021

ffromani commented Oct 18, 2021

swatisehgal commented Oct 19, 2021

ffromani commented Oct 19, 2021 •

edited

Add CPUManager policy option to distribute CPUs across NUMA nodes instead of packing them #105631

Add CPUManager policy option to distribute CPUs across NUMA nodes instead of packing them #105631

Conversation

klueska commented Oct 12, 2021 • edited

What type of PR is this?

What this PR does / why we need it:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

klueska commented Oct 12, 2021

k8s-ci-robot commented Oct 12, 2021

klueska commented Oct 12, 2021

klueska commented Oct 12, 2021

klueska commented Oct 13, 2021

ffromani left a comment

Choose a reason for hiding this comment

ffromani Oct 18, 2021

Choose a reason for hiding this comment

ffromani Oct 18, 2021

Choose a reason for hiding this comment

klueska Oct 18, 2021

Choose a reason for hiding this comment

ffromani Oct 18, 2021

Choose a reason for hiding this comment

klueska Oct 18, 2021

Choose a reason for hiding this comment

klueska commented Oct 18, 2021 • edited

ffromani commented Oct 18, 2021

klueska commented Oct 18, 2021

ffromani commented Oct 18, 2021

swatisehgal commented Oct 19, 2021

ffromani commented Oct 19, 2021 • edited

klueska commented Oct 12, 2021 •

edited

klueska commented Oct 18, 2021 •

edited

ffromani commented Oct 19, 2021 •

edited