Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updating kube-proxy to ignore unready endpoints for Topology Hints #106507

Merged
merged 1 commit into from Nov 20, 2021
Merged

Updating kube-proxy to ignore unready endpoints for Topology Hints #106507

merged 1 commit into from Nov 20, 2021

Conversation

robscott
Copy link
Member

@robscott robscott commented Nov 17, 2021 •
edited

What type of PR is this?

/kind bug

What this PR does / why we need it:

This fixes a bug identified by @danwinship in #106497 (comment). The bug meant that if hints were assigned to only unready endpoints in a zone, kube-proxy would still select those endpoints in filtering instead of falling back to the full list of endpoints. This fixes that.

A follow up bug fix will be coming soon that will update the controller to exclude unready endpoints from hint calculations. Update: That follow up fix has now been filed as #106510.

Does this PR introduce a user-facing change?

Kube-Proxy now correctly filters out unready endpoints for Services with Topology Aware Hints enabled.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/cc @aojea @danwinship
/sig network
/priority important-soon

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. sig/network Categorizes an issue or PR as relevant to SIG Network. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 17, 2021
@robscott
Copy link
Member Author

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 17, 2021
@aojea
Copy link
Member

aojea commented Nov 17, 2021

the events failures will be fixed by #106508

@aojea
Copy link
Member

aojea commented Nov 17, 2021

/retest
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 17, 2021
This was referenced Nov 17, 2021
Merged
Merged
@danwinship
Copy link
Contributor

The bug meant that if hints were assigned to unready endpoints, they would still receive traffic.

No, unready endpoints get filtered out correctly later. The bug is just that if a zone has only unread endpoints, we will make the wrong decision about whether to use topology; it should say "there are no usable endpoints for this zone, so ignore topology and use endpoints from other zones as well", but instead it decides that topology is usable, but then filters out the unready endpoints, and so then the nodes in that zone think that the service has no endpoints and install a reject rule for the service.

@danwinship
Copy link
Contributor

Eventually, #106510 will turn these changes into no-ops, right? But I guess we still need this for now to deal with kube-proxy / kube-controller-manager version skew.

@robscott
Copy link
Member Author

Eventually, #106510 will turn these changes into no-ops, right? But I guess we still need this for now to deal with kube-proxy / kube-controller-manager version skew.

Yep, this is also a much smaller/more contained change, so if it should be easier to cherry-pick this one if we can only choose one. It's also possible that other controllers are generating EndpointSlices with hints, so good to have the logic here.

@robscott
Copy link
Member Author

No, unready endpoints get filtered out correctly later. The bug is just that if a zone has only unread endpoints, we will make the wrong decision about whether to use topology; it should say "there are no usable endpoints for this zone, so ignore topology and use endpoints from other zones as well", but instead it decides that topology is usable, but then filters out the unready endpoints, and so then the nodes in that zone think that the service has no endpoints and install a reject rule for the service.

Good catch, I've updated the PR description to be more accurate.

aojea
aojea reviewed Nov 18, 2021
View reviewed changes
pkg/proxy/topology.go
Comment on lines +71 to +73
if !endpoint.IsReady() {
continue
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to do the same in the loop of L96 in this file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. TrafficPolicy=Local doesn't have a similar failover mechanism, if there are no local endpoints we shouldn't fall back to all endpoints like we're doing here. Also with the related work around terminating endpoints, we will send traffic to non-ready terminating local endpoints if no others are available.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ic, 👍

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 18, 2021
@robscott
Copy link
Member Author

/retest

@danwinship
Copy link
Contributor

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danwinship, robscott

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 19, 2021
@reylejano
Copy link
Member

/milestone v1.23

@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Nov 19, 2021
@k8s-ci-robot k8s-ci-robot merged commit 37ae94f into kubernetes:master Nov 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@danwinship danwinship danwinship left review comments

@aojea aojea aojea left review comments

Assignees

@danwinship danwinship

@aojea aojea

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/network Categorizes an issue or PR as relevant to SIG Network. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.23
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@robscott @aojea @danwinship @k8s-ci-robot @reylejano