Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubelet: add probe termination to graceful shutdowns #105215

Merged
merged 1 commit into from Oct 12, 2021
Merged

kubelet: add probe termination to graceful shutdowns #105215

merged 1 commit into from Oct 12, 2021

Conversation

rphillips
Copy link
Member

What type of PR is this?

/kind bug
/sig node

What this PR does / why we need it:

Node Graceful Shutdown does not terminate probes upon kubelet shutdown. This can potentially cause a pod to be terminated from a Liveness probe failing.

This fix will shutdown all the probes on the pod upon shutdown.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

cc @bobbypage @mrunalp

Does this PR introduce a user-facing change?

Kubelet's Node Grace Shutdown will terminate probes when shutting down.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. sig/node Categorizes an issue or PR as relevant to SIG Node. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/kubelet labels Sep 23, 2021
@ehashman ehashman added this to Triage in SIG Node PR Triage Sep 23, 2021
@249043822
Copy link
Member

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 24, 2021
249043822
249043822 approved these changes Sep 24, 2021
View reviewed changes
Copy link
Member

@249043822 249043822 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 24, 2021
@pacoxu pacoxu moved this from Triage to Needs Reviewer in SIG Node PR Triage Sep 24, 2021
@pacoxu
Copy link
Member

pacoxu commented Sep 24, 2021

/cc @wzshiming

@249043822
Copy link
Member

Sorry, if fixed #105014, this pr may not need to do this again

@ehashman ehashman moved this from Needs Reviewer to Triage in SIG Node PR Triage Sep 27, 2021
ehashman
ehashman reviewed Sep 27, 2021
View reviewed changes
Copy link
Member

@ehashman ehashman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/priority important-soon

Bug affects usability of node graceful shutdown feature, so could be a candidate for backport.

/lgtm

@k8s-ci-robot k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Sep 27, 2021
@ehashman ehashman moved this from Triage to Needs Approver in SIG Node PR Triage Sep 27, 2021
@rphillips
Copy link
Member Author

/assign @bobbypage

@bobbypage
Copy link
Member

Thanks for adding this.

/lgtm

Also /cc @wzshiming since it will require #102915 to be updated.

@wzshiming
Copy link
Member

Thanks for adding this.

/lgtm

Also /cc @wzshiming since it will require #102915 to be updated.

Yes, I need to resolve the conflict again, I wish could merge #102915 sooner

/lgtm
/assign @mrunalp

@mrunalp
Copy link
Contributor

mrunalp commented Oct 11, 2021

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 249043822, mrunalp, rphillips, wzshiming

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 11, 2021
@k8s-triage-robot
Copy link

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

  • The PR does have any do-not-merge/* labels
  • The PR does not have the needs-ok-to-test label
  • The PR is mergeable (does not have a needs-rebase label)
  • The PR is approved (has cncf-cla: yes, lgtm, approved labels)
  • The PR is failing tests required for merge

You can:

/retest

@k8s-ci-robot k8s-ci-robot merged commit a923852 into kubernetes:master Oct 12, 2021
SIG Node PR Triage automation moved this from Needs Approver to Done Oct 12, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Oct 12, 2021
@openshift-ci-robot openshift-ci-robot mentioned this pull request Oct 13, 2021
Closed
@249043822 249043822 mentioned this pull request Oct 18, 2021
Closed
@SergeyKanzhelev
Copy link
Member

Is it expected that readiness probes will stop running? If graceful termination is set for a long time, we may need to ensure that the pod will turn into not ready state before the removing the pod.

We discussed similar in this PR: #92385

@bobbypage
Copy link
Member

@rphillips , @SergeyKanzhelev brings up a good point that the pod should be marked not ready, i.e. to ensure that traffic will not be sent to the pod during graceful shutdown. Do we need to handle readiness vs liveness probe in special case?

@SergeyKanzhelev SergeyKanzhelev mentioned this pull request Oct 20, 2021
Open
@SergeyKanzhelev
Copy link
Member

I think Ideally we just keep the probe running. Especially for the cases of a long termination.

Created #105780

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ehashman ehashman ehashman left review comments

@mrunalp mrunalp mrunalp approved these changes

@wzshiming wzshiming wzshiming approved these changes

@249043822 249043822 249043822 approved these changes

@vishh vishh Awaiting requested review from vishh

@yujuhong yujuhong Awaiting requested review from yujuhong

Assignees

@mrunalp mrunalp

@bobbypage bobbypage

@ehashman ehashman

@wzshiming wzshiming

@249043822 249043822

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node PR Triage
Done
Milestone
v1.23
Development

Successfully merging this pull request may close these issues.

None yet
10 participants
@rphillips @249043822 @pacoxu @bobbypage @wzshiming @mrunalp @k8s-ci-robot @k8s-triage-robot @SergeyKanzhelev @ehashman