Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scheduler volumebinding plugin - handle Lost PVC as UnschedulableAndUnresolvable #105245

Merged
merged 1 commit into from Oct 5, 2021
Merged

Scheduler volumebinding plugin - handle Lost PVC as UnschedulableAndUnresolvable #105245

merged 1 commit into from Oct 5, 2021

Conversation

yibozhuang
Copy link
Contributor

@yibozhuang yibozhuang commented Sep 24, 2021 •
edited

What type of PR is this?

/kind feature

What this PR does / why we need it:

This change adds an additional check in the volumebinding scheduler
plugin to handle PVC with phase ClaimLost which will allow the
scheduler to return UnschedulableAndUnresolvable during the PreFilter
stage and skip the rest of the node evaluation since the PVC is
bound to a PV that does not exist.

Without this change, the FailedScheduling error message would look like:

0/10 nodes are available: 2 node(s) had taint {node/test: true},
that the pod didn't tolerate, 6 node(s) had taint {node/unhealthy: true},
that the pod didn't tolerate, 2 pvc(s) bound to non-existent pv(s)

Which is still evaluating every single node to determine that the pod
cannot be scheduled because the PVC is bound to a non-existent PV

With this change, the FailedScheduling error message would look like:

0/10 nodes are available: 1 persistentvolumeclaim "foo" bound to non-existent persistentvolume "bar"

Signed-off By: Yibo Zhuang yibzhuang@gmail.com

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Enhance scheduler volumebinding plugin to handle Lost PVC as UnschedulableAndUnresolvable during PreFilter stage

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 24, 2021
@k8s-ci-robot
Copy link
Contributor

@yibozhuang: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 24, 2021
@k8s-ci-robot
Copy link
Contributor

Hi @yibozhuang. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 24, 2021
@yibozhuang yibozhuang force-pushed the lost-pvc-prefilter-optimization branch from 492d064 to b30d832 Compare September 24, 2021 22:28
@ddebroy
Copy link
Member

ddebroy commented Sep 24, 2021

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 24, 2021
@yibozhuang yibozhuang force-pushed the lost-pvc-prefilter-optimization branch from b30d832 to 59fde30 Compare September 27, 2021 16:30
@yibozhuang
Copy link
Contributor Author

/assign @thockin

@msau42
Copy link
Member

msau42 commented Sep 29, 2021

/assign

msau42
msau42 reviewed Sep 30, 2021
View reviewed changes
pkg/scheduler/framework/plugins/volumebinding/binder.go Outdated
// GetPodVolumes returns a pod's PVCs separated into bound, unbound with delayed binding (including provisioning)
// and unbound with immediate binding (including prebound)
GetPodVolumes(pod *v1.Pod) (boundClaims, unboundClaimsDelayBinding, unboundClaimsImmediate []*v1.PersistentVolumeClaim, err error)
// GetPodVolumes returns a pod's PVCs separated into bound, lost,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we put the logic in https://github.com/kubernetes/kubernetes/blob/59fde306060fa7762d87feb80cbaee948361c507/pkg/scheduler/framework/plugins/volumebinding/volume_binding.go#L126 instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion! Moved the additional check to podHasPVCs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@msau42 I've made the change as suggested, can you have another look whenever you get a chance? Thanks!

@yibozhuang
Scheduler volumebinding plugin - handle Lost PVC as
b8fe514 
UnschedulableAndUnresolvable

This change adds an additional check in the volumebinding scheduler
plugin to handle PVC with phase ClaimLost which will allow the
scheduler to return UnschedulableAndUnresolvable during the PreFilter
stage and skip the rest of the node evaluation since the PVC is
bound to a PV that does not exist.

Without this change, the FailedScheduling error message would look like:

0/10 nodes are available: 2 node(s) had taint {node/test: true},
that the pod didn't tolerate, 6 node(s) had taint {node/unhealthy: true},
that the pod didn't tolerate, 2 pvc(s) bound to non-existent pv(s)

Which is still evaluating every single node to determine that the pod
cannot be scheduled because the PVC is bound to a non-existent PV

With this change, the FailedScheduling error message would look like:

0/10 nodes are available: 1 persistentvolumeclaim "foo" bound
to non-existent persistentvolume "bar"

Signed-off By: Yibo Zhuang <yibzhuang@gmail.com>
@yibozhuang yibozhuang force-pushed the lost-pvc-prefilter-optimization branch from 59fde30 to b8fe514 Compare October 1, 2021 04:58
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 1, 2021
@yibozhuang
Copy link
Contributor Author

/retest

@yibozhuang
Copy link
Contributor Author

/retest

@msau42
Copy link
Member

msau42 commented Oct 5, 2021

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 5, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: msau42, yibozhuang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 5, 2021
@k8s-ci-robot k8s-ci-robot merged commit 6c292ce into kubernetes:master Oct 5, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Oct 5, 2021
@yibozhuang yibozhuang mentioned this pull request Aug 5, 2022
Closed
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@msau42 msau42 msau42 left review comments

@gnufied gnufied Awaiting requested review from gnufied

@jingxu97 jingxu97 Awaiting requested review from jingxu97

Assignees

@thockin thockin

@msau42 msau42

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
v1.23
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@yibozhuang @k8s-ci-robot @ddebroy @msau42 @thockin