refactor: disable insecure serving in kube-scheduler #96345

ingvagabund · 2020-11-08T12:45:03Z

What type of PR is this?

/kind deprecation
/kind cleanup

What this PR does / why we need it:

Disable insecure serving in kube-scheduler

Which issue(s) this PR fixes:

xref #91506

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

[ACTION REQUIRED] kube-scheduler: the following flags have no effect and would be removed in v1.24:
* `--port`
* `--address`
The insecure port flags `--port` may only be set to 0 now.
Also `metricsBindAddress` and `healthzBindAddress` fields from `kubescheduler.config.k8s.io/v1beta1` are no-op and expected to be empty. Removed in `kubescheduler.config.k8s.io/v1beta2` completely.

In addition, please be careful that:
* kube-scheduler MUST start with `--authorization-kubeconfig` and `--authentication-kubeconfig` correctly set to get authentication/authorization working.
* liveness/readiness probes to kube-scheduler MUST use HTTPS now, and the default port has been changed to 10259.
* Applications that fetch metrics from kube-scheduler should use a dedicated service account which is allowed to access nonResourceURLs `/metrics`.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2020-11-08T12:45:10Z

@ingvagabund: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ingvagabund · 2020-11-08T12:48:56Z

@knight42 I have followed your changes in #96216 combined with what I found in https://github.com/kubernetes/kubernetes/pull/95522/files. I have copied testComponentWithoutInsecureServing from your PR for testing purposes only. Once your PR is merged, I will rebase mine to drop it.

ingvagabund · 2020-11-08T12:49:19Z

@alculquicondor @ahg-g FYI

knight42 · 2020-11-08T13:42:25Z

@ingvagabund Thanks for your effort! Now the major obstacle is how to make Prometheus fetch metrics from the secure port of controller-manager/scheduler in the job pull-kubernetes-e2e-gce-100-performance: #96216 (comment). Once this is solved, I think we are good to go.

ingvagabund · 2020-11-08T14:17:48Z

Now the major obstacle is how to make Prometheus fetch metrics from the secure port of controller-manager/scheduler in the job pull-kubernetes-e2e-gce-100-performance: #96216 (comment).

There are few lines of code, that needs to be updated:

Checking the logs you are referring to W1108 13:22:08.530] I1108 13:22:08.529881 98292 util.go:93] 114/115 targets are ready, example not ready target: {map[endpoint:kube-scheduler instance:10.40.0.3:10251 job:master namespace:monitoring service:master] down} I suppose.

Grepping for 10251 port:

alculquicondor · 2020-11-09T16:41:13Z

Are you targeting 1.20 or 1.21?

ingvagabund · 2020-11-09T16:49:09Z

1.20 would be great, though with 3 days left before the code freeze, finishing the PR might be too rushed

alculquicondor · 2020-11-09T16:54:22Z

@x13n does any metric pipeline rely on the insecure serving for kube-scheduler?

alculquicondor

minor comments.

alculquicondor · 2021-09-13T14:50:23Z

cmd/kube-scheduler/app/options/deprecated.go

+	)
+	fs.IPVar(&bindAddr, "address", bindAddr,
+		"The IP address on which to serve the insecure --port (set to 0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces).")
+	fs.MarkDeprecated("address", "This flag has no effect now and will be removed in v1.24.")


Mention the alternative: --bind-address

alculquicondor · 2021-09-13T14:50:44Z

cmd/kube-scheduler/app/options/deprecated.go

+	fs.MarkDeprecated("address", "This flag has no effect now and will be removed in v1.24.")
+
+	fs.IntVar(&bindPort, "port", bindPort, "The port on which to serve unsecured, unauthenticated access. Set to 0 to disable.")
+	fs.MarkDeprecated("port", "This flag has no effect now and will be removed in v1.24.")


Mention the alternative: --secure-port

alculquicondor · 2021-09-13T14:56:00Z

cmd/kube-scheduler/app/server.go

@@ -118,6 +131,11 @@ func runCommand(cmd *cobra.Command, opts *options.Options, registryOptions ...Op
 	verflag.PrintAndExitIfRequested()
 	cliflag.PrintFlags(cmd.Flags())

+	err := checkNonZeroInsecurePort(cmd.Flags())


do this in Deprecated.Validate.

If you mean DeprecatedOptions, the type does not provide the port flag through it. Also, adding a new Port field would require changing other places.

Is it that many places to add it?

Actually, not too many after all. Updated

alculquicondor · 2021-09-13T15:19:03Z

staging/src/k8s.io/kube-scheduler/config/v1beta1/types.go

-	// HealthzBindAddress is the IP address and port for the health check server to serve on,
-	// defaulting to 0.0.0.0:10251
+
+	// Note: Both HealthzBindAddress and MetricsBindAddress fields are deprecated


Put the deprecated note in each field's go comment. And say that only empty or port 0 is allowed.

alculquicondor · 2021-09-13T15:19:31Z

staging/src/k8s.io/kube-scheduler/config/v1beta2/types.go

-	// HealthzBindAddress is the IP address and port for the health check server to serve on,
-	// defaulting to 0.0.0.0:10251
+
+	// Note: Both HealthzBindAddress and MetricsBindAddress fields are deprecated


ravisantoshgudimetla

Thank you for working on this @ingvagabund

/lgtm

liggitt · 2021-09-14T13:48:08Z

/approve

k8s-ci-robot · 2021-09-14T13:48:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, alculquicondor, dims, ingvagabund, JAORMX, knight42, liggitt, ravisantoshgudimetla, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kube-scheduler/OWNERS~~ [ahg-g,alculquicondor,liggitt]
~~pkg/registry/OWNERS~~ [liggitt]
~~pkg/scheduler/apis/config/OWNERS~~ [ahg-g,alculquicondor,liggitt]
~~staging/src/k8s.io/kube-scheduler/config/OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ingvagabund · 2021-09-14T13:52:38Z

/hold cancel

alculquicondor

/lgtm

From logs: Flag --port has been deprecated, This flag has no effect now and will be removed in v1.24. So that's what we do. We had this flag only set to disable insecure serving, and insecure serving has been removed in upstream, thereby rendering the use of this flag a no-op. Controller-manager PR: kubernetes/kubernetes#96216 Scheduler PR: kubernetes/kubernetes#96345 Change-Id: If9009aa6f7c72a5ec8b7baf2326964167059c0a1 Reviewed-on: https://review.monogon.dev/c/monogon/+/665 Reviewed-by: Lorenz Brun <lorenz@monogon.tech>

k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. area/e2e-test-framework Issues or PRs related to refactoring the kubernetes e2e test framework labels Nov 8, 2020

k8s-ci-robot requested review from dims and draveness November 8, 2020 12:45

ingvagabund mentioned this pull request Nov 8, 2020

WIP: remove insecure-port and deprecated insecure serving #95522

Closed

ingvagabund force-pushed the disable-insecure-port-in-scheduler branch from fb21bd9 to fe3e2a9 Compare November 8, 2020 13:43

ingvagabund force-pushed the disable-insecure-port-in-scheduler branch from fe3e2a9 to e5692e3 Compare November 9, 2020 08:57

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 15, 2020

ingvagabund force-pushed the disable-insecure-port-in-scheduler branch from ee3f679 to 15b9ac7 Compare September 13, 2021 08:54

alculquicondor reviewed Sep 13, 2021

View reviewed changes

ingvagabund force-pushed the disable-insecure-port-in-scheduler branch 2 times, most recently from 4aca9d4 to ec206eb Compare September 13, 2021 19:47

refactor: disable insecure serving in kube-scheduler

07af669

ingvagabund force-pushed the disable-insecure-port-in-scheduler branch from ec206eb to 07af669 Compare September 14, 2021 05:52

ravisantoshgudimetla approved these changes Sep 14, 2021

View reviewed changes

k8s-ci-robot assigned ravisantoshgudimetla Sep 14, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 14, 2021

liggitt moved this from Changes requested to API review completed, 1.23 in API Reviews Sep 14, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 14, 2021

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 14, 2021

alculquicondor reviewed Sep 14, 2021

View reviewed changes

k8s-ci-robot assigned alculquicondor Sep 14, 2021

k8s-ci-robot merged commit c10be98 into kubernetes:master Sep 14, 2021

ingvagabund deleted the disable-insecure-port-in-scheduler branch September 14, 2021 17:00

pacoxu mentioned this pull request Sep 15, 2021

kubeadm: remove the flag --port from scheduler manifest #105034

Merged

ingvagabund mentioned this pull request Sep 15, 2021

bug 1889488: UPSTREAM: 96345: refactor: disable insecure serving in kube-scheduler openshift/kubernetes#958

Closed

rifelpet mentioned this pull request Sep 16, 2021

Use secure port for kube-scheduler liveness probe kubernetes/kops#12331

Closed

aiyengar2 mentioned this pull request Dec 4, 2021

[RKE1] kube-controller-manager / kube-scheduler HTTPs ports cannot perform authn / authz by default rancher/rancher#35751

Closed

pacoxu mentioned this pull request Dec 8, 2021

kube-scheduler fail to start after upgrading to v1.23.0 #106866

Closed

liggitt mentioned this pull request Dec 8, 2021

kube-scheduler default port (breaking) change not mentioned in 1.23 changelog #106885

Closed

rfranzke mentioned this pull request Jan 12, 2022

☂️-Issue for "Support for Kubernetes v1.23” gardener/gardener#5102

Closed

20 tasks

sjenning mentioned this pull request Jan 17, 2022

Fix kube scheduler health/liveness probes openshift/hypershift#868

Merged

jinja2 mentioned this pull request Mar 28, 2023

Update the port used to monitor the Kubernetes scheduler (10251->10259) signalfx/splunk-otel-collector-chart#711

Merged

ingvagabund mentioned this pull request Mar 30, 2023

bind http metrics to localhost openshift/cluster-kube-scheduler-operator#474

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: disable insecure serving in kube-scheduler #96345

refactor: disable insecure serving in kube-scheduler #96345

ingvagabund commented Nov 8, 2020 •

edited by liggitt

k8s-ci-robot commented Nov 8, 2020

ingvagabund commented Nov 8, 2020

ingvagabund commented Nov 8, 2020

knight42 commented Nov 8, 2020

ingvagabund commented Nov 8, 2020

alculquicondor commented Nov 9, 2020

ingvagabund commented Nov 9, 2020

alculquicondor commented Nov 9, 2020

alculquicondor left a comment

alculquicondor Sep 13, 2021

alculquicondor Sep 13, 2021

alculquicondor Sep 13, 2021

ingvagabund Sep 13, 2021

alculquicondor Sep 13, 2021

ingvagabund Sep 14, 2021

alculquicondor Sep 13, 2021

alculquicondor Sep 13, 2021

ravisantoshgudimetla left a comment

liggitt commented Sep 14, 2021

k8s-ci-robot commented Sep 14, 2021

ingvagabund commented Sep 14, 2021

alculquicondor left a comment

refactor: disable insecure serving in kube-scheduler #96345

refactor: disable insecure serving in kube-scheduler #96345

Conversation

ingvagabund commented Nov 8, 2020 • edited by liggitt

k8s-ci-robot commented Nov 8, 2020

ingvagabund commented Nov 8, 2020

ingvagabund commented Nov 8, 2020

knight42 commented Nov 8, 2020

ingvagabund commented Nov 8, 2020

alculquicondor commented Nov 9, 2020

ingvagabund commented Nov 9, 2020

alculquicondor commented Nov 9, 2020

alculquicondor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ravisantoshgudimetla left a comment

Choose a reason for hiding this comment

liggitt commented Sep 14, 2021

k8s-ci-robot commented Sep 14, 2021

ingvagabund commented Sep 14, 2021

alculquicondor left a comment

Choose a reason for hiding this comment

ingvagabund commented Nov 8, 2020 •

edited by liggitt