Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use "Capacity" instead of "Allocatable" for an accurate node memory total size #102917

Merged
merged 1 commit into from Nov 5, 2021

Conversation

bysnupy
Copy link
Contributor

@bysnupy bysnupy commented Jun 16, 2021

What type of PR is this?

/kind bug
/sig cli

What this PR does / why we need it:

"Allocatable" is just a logical total memory size based on [Allocatable] = [Node Capacity] - [system-reserved] - [Hard-Eviction-Thresholds].
In contrast, the node memory usage collected by Metrics API(metrics.k8s.io) is based on real usage constantly on the node host. So if system-reserved or hard-eviction threshold is configured bigger, the MEMORY% can be overflow than 100%.

We focus the "Allocatable" value is useful to schedule pods to a node, at that time scheduler consider the pod memory requests size based on logical value. All values and calculation is under logical assumption.

But the real node total memory usage that is "MEMORY%" of "kubectl top node" column value is different.
It should be based on the actual hard threshold size (MemTotal in /proc/meminfo), because the collected metrics usage is also based on real usage on the node host.
We should not mix the logical and real values to calculate some value accurately, so we should replace "Allocatable" with "Capacity" which is the same with the MemTotal in "MEMORY%" calculation for accurate total node memory usage.

(before) [MEMORY%] =  [node memory usage of the Metrics API] / [node Allocatable] * 100
(after ) [MEMORY%] =  [node memory usage of the Metrics API] / [node Capacity]    * 100

Which issue(s) this PR fixes:

Fixes #86499

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Added show-capacity option to `kubectl top node` to show `Capacity` resource usage

@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jun 16, 2021
@k8s-ci-robot
Copy link
Contributor

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jun 16, 2021
@k8s-ci-robot
Copy link
Contributor

Welcome @bysnupy!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jun 16, 2021
@k8s-ci-robot
Copy link
Contributor

Hi @bysnupy. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jun 16, 2021
@k8s-ci-robot k8s-ci-robot added area/kubectl sig/cli Categorizes an issue or PR as relevant to SIG CLI. labels Jun 16, 2021
@bysnupy
Copy link
Contributor Author

bysnupy commented Jun 16, 2021

I signed it

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jun 16, 2021
@bysnupy
Copy link
Contributor Author

bysnupy commented Jun 16, 2021

/kind bug

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. and removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Jun 16, 2021
@bysnupy bysnupy changed the title Use "Capacity" instead of "Allocatable" for an accurate node memory t… Use "Capacity" instead of "Allocatable" for an accurate node memory total size Jun 16, 2021
@bysnupy
Copy link
Contributor Author

bysnupy commented Jun 24, 2021

/assign @soltysh

@lauchokyip
Copy link
Member

Hi @bysnupy , a similiar PR to this #100222

@bysnupy
Copy link
Contributor Author

bysnupy commented Jun 25, 2021

@lauchokyip Great ! Thank you for your kind information. I didn't realized the previous one.
As far as I'm concerned, kubectl top node metrics are also used by autoscaler on k8s. Basically autoscaler and scheduler is the related each other, if it's design, it makes sense for me.
One more thing, does it mean that the 100+% result does not affect the autoscaler negatively ?

Even if it's no problem on other feature, 100+% is confused sometimes on usual users. If possible we should notify this value can be more than 100% in the docs, or if the metrics is more than 100%, it should set as 100% is, not 100+%. e.g.> If actual calculation result is 125%, it should show users as 100%. How about your thought ?

@lauchokyip
Copy link
Member

lauchokyip commented Jun 26, 2021

@bysnupy , I am not familiar with this area, this is what @soltysh responded and you might help it helpful
#99945 (comment). Sorry!

@bysnupy
Copy link
Contributor Author

bysnupy commented Jun 28, 2021

@lauchokyip No problem. I've also created other PR here #103253 to fix this issue which is based on another standpoint. Could please TAL ? @soltysh I'd appreciate any help you can offer. Thank you !

@KnVerey
Copy link
Contributor

KnVerey commented Jun 28, 2021

@soltysh in https://github.com/kubernetes/kubernetes/pull/100222/files/7b43880b4c95e2928c59a3c2b72382aec8894e25#diff-c3a7260dd14d315707b6043e9ed76a19bfcdd233f465d7cd5b95dd2aaadcec6f you explain that:

The reason we pick Allocatable over Capacity is because it represents the value available for scheduling, ie. for running pods. That's by design, not a bug.

If I've understood correctly, this PR claims the problem is that the numerator and the denominator in the calculation don't have the same logical basis: we're dividing a number that includes reserved capacity (actual usage) by a number that excludes it (allocatable), yielding a result that doesn't make sense. Is that true based on your knowledge of the underlying metrics? If it is, this PR's solution makes more sense to me than #103253, which essentially rounds down to hide the values when they're visibly incorrect but does not fix them otherwise. I'm going to change the reviewers on that PR to match this one, since they are alternatives.

@pacoxu pacoxu moved this from Needs review to In progress in SIG CLI Jul 12, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 7, 2021
@bysnupy
Copy link
Contributor Author

bysnupy commented Oct 13, 2021

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 13, 2021
Copy link
Contributor

@soltysh soltysh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/ok-to-test
/triage accepted
/priority backlog
/lgtm
/approve
/sig cli
@bysnupy you'll need to update the release note in the initial comment to get this merged

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/backlog Higher priority than priority/awaiting-more-evidence. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 5, 2021
@soltysh
Copy link
Contributor

soltysh commented Nov 5, 2021

/hold cancel

@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Nov 5, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bysnupy, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 5, 2021
@soltysh
Copy link
Contributor

soltysh commented Nov 5, 2021

/sig cli

@soltysh soltysh removed the do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Nov 5, 2021
@bysnupy
Copy link
Contributor Author

bysnupy commented Nov 5, 2021

/retest

@k8s-ci-robot
Copy link
Contributor

@bysnupy: You must be an org member to edit the release note.

In response to this:

/release-note-edit

Added show-capacity option to `kubectl top node` to show actual resource usage

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Nov 5, 2021
@k8s-ci-robot k8s-ci-robot merged commit cee4aa0 into kubernetes:master Nov 5, 2021
SIG CLI automation moved this from In progress to Done Nov 5, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Nov 5, 2021
bysnupy added a commit to bysnupy/kubernetes that referenced this pull request Nov 27, 2021
Add an example for the show-capacity flag. If you specified this flag, show metrics based on capacity which is the total amount of resources that a Node has.
* References: kubernetes#102917
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubectl cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/backlog Higher priority than priority/awaiting-more-evidence. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/cli Categorizes an issue or PR as relevant to SIG CLI. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
Development

Successfully merging this pull request may close these issues.

kubectl shows node memory >100%?
7 participants