Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

package log runner as part of Kubernetes releases #106123

Merged
merged 1 commit into from Nov 15, 2021

Conversation

pohly
Copy link
Contributor

@pohly pohly commented Nov 3, 2021

What type of PR is this?

/kind feature

What this PR does / why we need it:

kube-log-runner was formerly known as go-runner when it was originally introduced
in #90804.
It was moved to kubernetes/release/images/build/go-runner later but is now
needed again in Kubernetes itself as replacement for the deprecated --log-file
klog feature: when bringing up a Windows node, kube-proxy.exe and kubelet.exe must be wrapped
with the helper binary to redirect output.

It got renamed to avoid a naming conflict with
test/conformance/image/go-runner.

Other downstream Kubernetes users may have a similar need, therefore it makes
sense to provide a pre-built binary.

Which issue(s) this PR fixes:

Fixes #106101

Special notes for your reviewer:

Does this PR introduce a user-facing change?

action required: [kube-log-runner](https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/component-base/logs/kube-log-runner) is included in release tar balls. It can be used to replace the deprecated `--log-file` parameter.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/issues/2845

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/provider/gcp Issues or PRs related to gcp provider sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 3, 2021
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Nov 3, 2021
@pohly
Copy link
Contributor Author

pohly commented Nov 3, 2021

/test pull-kubernetes-e2e-windows-gce

@@ -0,0 +1,123 @@
/*
Copyright 2020 The Kubernetes Authors.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the original go-runner.go from @dims. I kept the file name, but perhaps I should better rename also it to kube-runner.go?

@BenTheElder
Copy link
Member

We should really have this de-duped vs https://github.com/kubernetes/release/tree/master/images/build/go-runner

I don't think it makes sense to have two versions of this, maybe we can just publish the one in kubernetes/release somewhere and optionally also include that in our release tarballs?

See also: kubernetes/release#2301

cc @serathius

@pohly
Copy link
Contributor Author

pohly commented Nov 3, 2021

We should really have this de-duped vs https://github.com/kubernetes/release/tree/master/images/build/go-runner

Agreed. But before worrying about how to do that, I wanted to first check that:

  • adding a new binary to the release tar balls is acceptable
  • that it solves the problem with Window GCE clusters

I don't think it makes sense to have two versions of this, maybe we can just publish the one in kubernetes/release somewhere and optionally also include that in our release tarballs?

"Publish the one in kubernetes/release" - do you mean source code or binaries?

My gut feeling is that sharing source code via go.mod dependencies might be simpler, but I haven't tried it.

@pohly
Copy link
Contributor Author

pohly commented Nov 3, 2021

Also, everybody's favorite discussion: what should be the name of the tool, given that go-runner is already taken in k/k? I don't think we can have two with the same name because they get copied into the same output directory.

@BenTheElder
Copy link
Member

"Publish the one in kubernetes/release" - do you mean source code or binaries?

A binary, right now I think only the distroless based image is published but we could publish a binary as well (and potentially re-publish it with kubernetes releases)

One thing we have to be careful of: These things are an API in the images they wind up in where users may explicitly set the entrypoint, so at the very least if we pick a new name for either of these we need to add a symlink in the images.

I'm not sure what's best here. I wish we'd kept more of this in-repo and avoided the duplicate naming to begin with 🙃

@pohly
Copy link
Contributor Author

pohly commented Nov 4, 2021

A binary, right now I think only the distroless based image is published but we could publish a binary as well (and potentially re-publish it with kubernetes releases)

I'm currently leaning the other way: let's publish the source code as part of k/k and then pick that up in k/release via go install when building the image. Having the source in k/k allows us to use the normal build and publishing rules and it matches user expectations about where to find Kubernetes code.

Downloading a precompiled binary sounds like it will be more work.

Here's a more complete proposal:

  • let's stick with flag-style single-dash parameters (instead of pflag double-dash): simpler binary, compatible with current go-runner
  • no changes for the current images (go-runner as image name and name of the binary inside them): avoids another migration
  • kube-log-runner as full name elsewhere: kube-runner is still too vague
  • hosted under staging/src/k8s.io/component-base/logs/kube-log-runner: that's better than in-tree (= cmd/kube-log-runner) because go install k8s.io/component-base/logs/kube-log-runner will be more efficient; a separate repo would be overkill. It also puts the tool under control of the logging maintainers.

@pohly
Copy link
Contributor Author

pohly commented Nov 4, 2021

Not having a kubelet.log under artifacts is normal for the pull-kubernetes-e2e-windows-gce, it also behaves that way without these changes (https://gcsweb.k8s.io/gcs/kubernetes-jenkins/pr-logs/pull/106090/pull-kubernetes-e2e-windows-gce/1455962754056720384/artifacts/e2e-3032f9a532-95646-windows-node-group-2nqm/).

@pohly
Copy link
Contributor Author

pohly commented Nov 4, 2021

kube-log-runner was formerly known as go-runner when it was originally introduced
in kubernetes@393e095
It was moved to kubernetes/release/images/build/go-runner later but is now
needed again in Kubernetes itself as replacement for the deprecated --log-file
klog feature: when bringing up a Windows node, kube-proxy.exe and kubelet.exe must be wrapped
with the helper binary to redirect output.

It got renamed to avoid a naming conflict with
test/conformance/image/go-runner and because the name was too vague.

Other downstream Kubernetes users may have a similar need, therefore it makes
sense to provide a prebuilt binary also in the release archives.
@pohly
Copy link
Contributor Author

pohly commented Nov 4, 2021

/test pull-kubernetes-e2e-windows-gce

@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. labels Nov 4, 2021
@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 4, 2021
@BenTheElder
Copy link
Member

I think we should get agreement from more folks first, so we don't keep going in circles on this one.

@pohly
Copy link
Contributor Author

pohly commented Nov 4, 2021

Yes, let's keep this open. @dims will have time to comment on Monday.

FWIW, the PR currently implements my proposal above (kube-log-runner as name, hosted in component-base/logs).

@serathius
Copy link
Contributor

/milestone v1.23
This is part of work on deprecating KEP, we need to merge it to provide users an alternative to klog flags for writing files.

@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Nov 11, 2021
@serathius
Copy link
Contributor

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 11, 2021
@pohly
Copy link
Contributor Author

pohly commented Nov 11, 2021

@vishh: may I have your approval for the cluster/gce and hack part of this PR?

@pohly
Copy link
Contributor Author

pohly commented Nov 11, 2021

/hold

Let's make sure we have sufficient feedback (@dims, @BenTheElder?) before merging.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 11, 2021
@pohly
Copy link
Contributor Author

pohly commented Nov 11, 2021

/assign @vishh

@dims
Copy link
Member

dims commented Nov 11, 2021

/retest
/approve
/lgtm

yes, +1 to ask @BenTheElder to peek as well.

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, pohly, serathius

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 11, 2021
@dims dims unassigned vishh Nov 11, 2021
@pohly
Copy link
Contributor Author

pohly commented Nov 11, 2021

I just learned that vishh is not active anymore, but Ben can cover hack.

/assign @mtaufen @wojtek-t

For approval of the cluster/gce changes.

@k8s-ci-robot
Copy link
Contributor

@pohly: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kubernetes-e2e-windows-gce d2a8a81 link false /test pull-kubernetes-e2e-windows-gce

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@pohly
Copy link
Contributor Author

pohly commented Nov 11, 2021

The previous run for pull-kubernetes-e2e-windows-gce and the current commit successfully brought up a cluster and only had some individual test failures, which seems to be normal.

The latest retest failed early with "You do not have access/write permission on kubernetes-release-pull".

@pohly
Copy link
Contributor Author

pohly commented Nov 15, 2021

/kind feature

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. and removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Nov 15, 2021
@pohly
Copy link
Contributor Author

pohly commented Nov 15, 2021

Looks like @BenTheElder is out and we should get this merged before the code freeze, so:

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 15, 2021
@k8s-ci-robot k8s-ci-robot merged commit da73a24 into kubernetes:master Nov 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/provider/gcp Issues or PRs related to gcp provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on. wg/structured-logging Categorizes an issue or PR as relevant to WG Structured Logging.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove deprecate klog flag usage from windows cluster setup
9 participants