Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't prematurely close reflectors in case of slow initialization in watch based manager #104604

Merged
merged 1 commit into from Aug 26, 2021
Merged

Don't prematurely close reflectors in case of slow initialization in watch based manager #104604

merged 1 commit into from Aug 26, 2021

Conversation

wojtek-t
Copy link
Member 

Don't prematurely close reflectors in case of slow initialization in watch based manager to fix issues with inability to properly mount secrets/configmaps.

/kind bug
/priority important-soon
/sig node

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. sig/node Categorizes an issue or PR as relevant to SIG Node. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. area/kubelet and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 26, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 26, 2021
@wojtek-t
Copy link
Member Author

/assign @chenyw1990

@wojtek-t wojtek-t mentioned this pull request Aug 26, 2021
Closed
mborsz
mborsz reviewed Aug 26, 2021
View reviewed changes
pkg/kubelet/util/manager/watch_based_manager.go Outdated
@@ -287,11 +291,14 @@ func (c *objectCache) Get(namespace, name string) (runtime.Object, error) {
if !exists {
return nil, fmt.Errorf("object %q/%q not registered", namespace, name)
}
// Record last access time independently if it succeeded or not.
// This protects from premature reflector closure in case of slow initializations.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I wouldn't say that if the watches are initializing fast, there is no problem to fix: In any case (fast or slow initialization) there is a race between a startRecycleIdleWatch goroutine and all initializing reflectors, it's just more visible in case of slow initialization.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed the last part of the sentence

@mborsz
Copy link
Member

mborsz commented Aug 26, 2021

/lgtm
/hold

[hold in case you want to wait for review from sig-node, please unhold if you don't]

@k8s-ci-robot k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Aug 26, 2021
@chenyw1990
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Aug 26, 2021
@wojtek-t
Copy link
Member Author

@mborsz @chenyw1990 - would you mind reapplying lgtm (I fixed the gofmt issue and fixed the comment as pointed out by Maciek)

@chenyw1990
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 26, 2021
@wojtek-t
Copy link
Member Author

/retest

/hold cancel

Let's have it merged to allow cherrypicking.

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 26, 2021
marseel
marseel reviewed Aug 26, 2021
View reviewed changes
pkg/kubelet/util/manager/watch_based_manager_test.go
@@ -405,3 +405,94 @@ func TestMaxIdleTimeStopsTheReflector(t *testing.T) {
// Reflector should be running when the get function is called periodically.
assert.True(t, reflectorRunning())
}

func TestReflectorNotStopedOnSlowInitialization(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: Stopped

@wojtek-t
Copy link
Member Author

/retest

1 similar comment
@wojtek-t
Copy link
Member Author

/retest

@k8s-ci-robot k8s-ci-robot merged commit c262d09 into kubernetes:master Aug 26, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Aug 26, 2021
@jingxu97
Copy link
Contributor

Thanks @wojtek-t! Let me cherrypick this to 1.22 and 1.21

This was referenced Aug 26, 2021
Merged
Merged
@cpanato
Copy link
Member

cpanato commented Aug 27, 2021

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 27, 2021
k8s-ci-robot added a commit that referenced this pull request Aug 27, 2021
@k8s-ci-robot
Merge pull request #104615 from jingxu97/automated-cherry-pick-of-#10…
6c76143 
…4604-upstream-release-1.22

Automated cherry pick of #104604: Don't prematurely close reflectors in case of slow
k8s-ci-robot added a commit that referenced this pull request Aug 27, 2021
@k8s-ci-robot
Merge pull request #104616 from jingxu97/automated-cherry-pick-of-#10…
7fcffec 
…4604-upstream-release-1.21

Automated cherry pick of #104604: Don't prematurely close reflectors in case of slow
@wojtek-t wojtek-t mentioned this pull request Jan 25, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marseel marseel marseel left review comments

@mborsz mborsz mborsz left review comments

@mtaufen mtaufen Awaiting requested review from mtaufen

@Random-Liu Random-Liu Awaiting requested review from Random-Liu

Assignees

@chenyw1990 chenyw1990

@mborsz mborsz

@jingxu97 jingxu97

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.23
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@wojtek-t @k8s-ci-robot @mborsz @chenyw1990 @jingxu97 @cpanato @marseel