Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix race condition in logging when request times out #105734

Merged
merged 1 commit into from Oct 28, 2021

Conversation

marseel
Copy link
Member

@marseel marseel commented Oct 18, 2021

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR fixes race condition:

This can cause unrecoverable panic that crashes apiserver(this stacktrace was based on 1.21 version):

goroutine 943732 [running]:
runtime.throw
        /usr/local/go/src/runtime/panic.go:1117
runtime.mapaccess1_faststr
        /usr/local/go/src/runtime/map_faststr.go:21
net/textproto.MIMEHeader.Get
        /usr/local/go/src/net/textproto/header.go:34
net/http.Header.Get(...)
        /usr/local/go/src/net/http/header.go:46
net/http.(*Request).UserAgent(...)
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).LogArgs(...)
        k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:185 

I've also looked briefly and found two other possible race conditions when accessing/modifying addedInfo or addedKeyValuePairs (although I haven't observed such error in real cluster as compared to this panic above).

Which issue(s) this PR fixes:

Fixes #105759

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fix concurrent map access causing panics when logging timed-out API calls.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Oct 18, 2021
@marseel
Copy link
Member Author

marseel commented Oct 18, 2021

/assign @wojtek-t

@k8s-ci-robot k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Oct 18, 2021
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Oct 18, 2021
@marseel
Copy link
Member Author

marseel commented Oct 19, 2021

/retest

@marseel
Copy link
Member Author

marseel commented Oct 19, 2021

/retest

Copy link
Member

@wojtek-t wojtek-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/assign @liggitt

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 19, 2021
@marseel
Copy link
Member Author

marseel commented Oct 19, 2021

/test pull-kubernetes-kubemark-e2e-gce-big

@fedebongio
Copy link
Contributor

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 19, 2021
@wojtek-t
Copy link
Member

/assign @sttts

@sttts
Copy link
Contributor

sttts commented Oct 26, 2021

/assign @tkashem

@tkashem
Copy link
Contributor

tkashem commented Oct 27, 2021

/lgtm

Thanks!

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 27, 2021
@wojtek-t
Copy link
Member

/lgtm
/approve

@marseel - can you please open cherrypicks once this PR merges?

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: marseel, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 28, 2021
@k8s-triage-robot
Copy link

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

  • The PR does have any do-not-merge/* labels
  • The PR does not have the needs-ok-to-test label
  • The PR is mergeable (does not have a needs-rebase label)
  • The PR is approved (has cncf-cla: yes, lgtm, approved labels)
  • The PR is failing tests required for merge

You can:

/retest

@k8s-ci-robot k8s-ci-robot merged commit afff019 into kubernetes:master Oct 28, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Oct 28, 2021
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Nov 4, 2021
k8s-ci-robot added a commit that referenced this pull request Nov 4, 2021
…734-upstream-release-1.20

Automated cherry pick of #105734: Fix race condition in logging when request times out
k8s-ci-robot added a commit that referenced this pull request Nov 4, 2021
…734-upstream-release-1.22

Automated cherry pick of #105734: Fix race condition in logging when request times out
k8s-ci-robot added a commit that referenced this pull request Nov 4, 2021
…734-upstream-release-1.21

Automated cherry pick of #105734: Fix race condition in logging when request times out
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Race condition in apiserver when request times out
8 participants