Add feature gate to disable in-tree credential providers #102507

ostrain · 2021-06-01T20:43:31Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Implementing DisableKubeletCloudCredentialProviders FG according to proposal kubernetes/enhancements#2443

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Introduce a feature gate DisableKubeletCloudCredentialProviders which allows disabling the in-tree kubelet credential providers.

The DisableKubeletCloudCredentialProviders FeatureGate is currently in Alpha, which means is currently disabled by default. Once the FeatureGate moves to beta, in-tree credential providers will be disabled by default, and users will need to migrate to using external credential providers.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/pull/2443
- [KEP]: https://github.com/kubernetes/enhancements/tree/1a66671cf0e01ceb19f56bfb563100909cb67867/keps/sig-cloud-provider/2133-out-of-tree-credential-provider

k8s-ci-robot · 2021-06-01T20:43:34Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2021-06-01T20:43:39Z

Welcome @ostrain!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2021-06-01T20:43:39Z

Hi @ostrain. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ostrain · 2021-06-02T22:14:53Z

I believe the CLA should be all set now, commenting to trigger the bot again.

DangerOnTheRanger · 2021-06-03T22:21:14Z

/ok-to-test

ostrain · 2021-06-04T19:16:56Z

/sig cloud-provider

cheftako · 2021-06-22T04:39:16Z

/cc @cheftako @andrewsykim @DangerOnTheRanger @jpbetz

ostrain · 2021-07-19T16:27:12Z

/retest

pkg/credentialprovider/plugins.go

pkg/credentialprovider/gcp/metadata.go

cheftako · 2021-07-19T20:08:20Z

pkg/credentialprovider/gcp/metadata.go

@@ -142,7 +145,17 @@ func onGCEVM() bool {

 // Enabled implements DockerConfigProvider for all of the Google implementations.
 func (g *MetadataProvider) Enabled() bool {
-	return onGCEVM()
+	if credentialprovider.AreLegacyCloudCredentialProvidersDisabled() {


Not sure I agree with this ordering. I agree about the relative costs but if we need to enter either switch the system is miconfigured. At the point the cost seems like a non issue. For the good case you need to pass both switched anyway. In this case we may have a misleading error message suggesting you are on GCP. I think it would be better to put back the original order and add a warning you are attempting to use a GCP credential provider on a non GCP VM to the !onGCEVM case.

Avoiding the misleading error message was my original thinking, happy to switch it back.

I'm not sure if the other warning you propose would add much value though: IIUC these credential providers are on by default (thus the need for this PR) so it's not like the user misconfigured something if we're getting this far on a non-GCP VM. The warning would just get logged all the time and wouldn't be actionable.

I thought we would only have this on if it was configured. If thats not true, then I think its even more important to switch back. (As you say to avoid the misleading error message)

Switched this back around.

I'm not following the logic here to switch the order back to run onGCEVM first. If the feature gate is enabled (i.e. disabled), then there's no reason to ever call onGCEVM, because the plugin is explicitly disabled. In this case, the warning message makes sense too, because the plugin is disabled anyways. When it is disabled, why bother running onGCEVM which is a relatively expensive operation that reads a file from disk?

With the in-tree cloud providers, you have to pass --cloud-provider=foo to enable them so it makes sense to print a warning if you pass that flag while the feature gate that disables them is turned on, since that's obviously a misconfiguration of either the flag or the feature gate.

With the in-tree credential providers OTOH, there's no flag to enable them (as far as I'm aware) -- we just rely on the onGCEVM check alone. So if the feature gate is turned on but we're not on a GCP VM, there's no reason to print a warning about the GCP provider being disabled... Nothing is misconfigured in that case, the warning is just confusing and unactionable.

Gotcha, thanks for clarifying.

ostrain · 2021-07-19T21:47:05Z

/retest

ostrain · 2021-07-19T23:09:05Z

/retest

cheftako · 2021-07-20T01:43:20Z

/lgtm

andrewsykim

/assign @liggitt

pkg/credentialprovider/gcp/metadata.go

pkg/credentialprovider/aws/aws_credentials.go

pkg/credentialprovider/gcp/metadata.go

ostrain · 2021-08-04T21:33:04Z

/retest

ostrain · 2021-08-04T22:18:40Z

/retest

ostrain · 2021-08-05T00:08:07Z

/retest

pkg/credentialprovider/aws/aws_credentials.go

liggitt · 2021-08-05T17:05:29Z

/lgtm
/approve

k8s-ci-robot · 2021-08-05T17:05:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewsykim, Danil-Grigorev, liggitt, ostrain

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/credentialprovider/OWNERS~~ [liggitt]
~~pkg/features/OWNERS~~ [andrewsykim,liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pacoxu · 2021-08-06T03:57:26Z

pkg/features/kube_features.go

@@ -595,6 +595,12 @@ const (
 	// Disable any functionality in kube-apiserver, kube-controller-manager and kubelet related to the `--cloud-provider` component flag.
 	DisableCloudProviders featuregate.Feature = "DisableCloudProviders"

+	// owner: @andrewsykim
+	// alpha: v1.22


NIT: just see this: it would be 1.23

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jun 1, 2021

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jun 1, 2021

k8s-ci-robot requested review from justinsb and lavalamp June 1, 2021 20:44

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jun 2, 2021

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 3, 2021

k8s-ci-robot added sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jun 4, 2021

ostrain changed the title ~~[WIP] Add feature gate to disable in-tree credential providers~~ Add feature gate to disable in-tree credential providers Jun 4, 2021

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 4, 2021

k8s-ci-robot requested review from andrewsykim, cheftako, DangerOnTheRanger and jpbetz June 22, 2021 04:39

andrewsykim reviewed Jul 19, 2021

View reviewed changes

pkg/credentialprovider/plugins.go Outdated Show resolved Hide resolved

pkg/credentialprovider/gcp/metadata.go Outdated Show resolved Hide resolved

ostrain force-pushed the feature/ostrain/legacy-credential-provider-fg branch from 49f16ae to 8571113 Compare July 19, 2021 20:03

cheftako reviewed Jul 19, 2021

View reviewed changes

ostrain force-pushed the feature/ostrain/legacy-credential-provider-fg branch from 8571113 to 5932233 Compare July 19, 2021 22:17

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 20, 2021

andrewsykim reviewed Jul 20, 2021

View reviewed changes

pkg/credentialprovider/gcp/metadata.go Show resolved Hide resolved

ostrain requested a review from liggitt August 2, 2021 18:47

liggitt reviewed Aug 4, 2021

View reviewed changes

pkg/credentialprovider/aws/aws_credentials.go Outdated Show resolved Hide resolved

pkg/credentialprovider/gcp/metadata.go Show resolved Hide resolved

ostrain force-pushed the feature/ostrain/legacy-credential-provider-fg branch from 5932233 to 16ce57f Compare August 4, 2021 20:22

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 4, 2021

ostrain force-pushed the feature/ostrain/legacy-credential-provider-fg branch from 16ce57f to c1515ad Compare August 4, 2021 20:51

liggitt reviewed Aug 5, 2021

View reviewed changes

pkg/credentialprovider/aws/aws_credentials.go Show resolved Hide resolved

Add feature gate to disable in-tree credential providers

a947c32

ostrain force-pushed the feature/ostrain/legacy-credential-provider-fg branch from c1515ad to a947c32 Compare August 5, 2021 16:50

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 5, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 5, 2021

k8s-ci-robot merged commit 542829d into kubernetes:master Aug 5, 2021

k8s-ci-robot added this to the v1.23 milestone Aug 5, 2021

pacoxu reviewed Aug 6, 2021

View reviewed changes

andrewsykim mentioned this pull request Aug 10, 2021

Fix documented version for DisableKubeletCloudCredentialProviders feature gate #104278

Merged

elmiko mentioned this pull request Sep 29, 2023

Removing In-Tree Cloud Providers kubernetes/enhancements#2395

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add feature gate to disable in-tree credential providers #102507

Add feature gate to disable in-tree credential providers #102507

ostrain commented Jun 1, 2021

k8s-ci-robot commented Jun 1, 2021

k8s-ci-robot commented Jun 1, 2021

k8s-ci-robot commented Jun 1, 2021

ostrain commented Jun 2, 2021

DangerOnTheRanger commented Jun 3, 2021

ostrain commented Jun 4, 2021

cheftako commented Jun 22, 2021

ostrain commented Jul 19, 2021

cheftako Jul 19, 2021

ostrain Jul 19, 2021

cheftako Jul 19, 2021

ostrain Jul 19, 2021

andrewsykim Jul 20, 2021

ostrain Jul 20, 2021

andrewsykim Jul 20, 2021

ostrain commented Jul 19, 2021

ostrain commented Jul 19, 2021

cheftako commented Jul 20, 2021

andrewsykim left a comment

ostrain commented Aug 4, 2021

ostrain commented Aug 4, 2021

ostrain commented Aug 5, 2021

liggitt commented Aug 5, 2021

k8s-ci-robot commented Aug 5, 2021

pacoxu Aug 6, 2021

andrewsykim Aug 10, 2021

Add feature gate to disable in-tree credential providers #102507

Add feature gate to disable in-tree credential providers #102507

Conversation

ostrain commented Jun 1, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Jun 1, 2021

k8s-ci-robot commented Jun 1, 2021

k8s-ci-robot commented Jun 1, 2021

ostrain commented Jun 2, 2021

DangerOnTheRanger commented Jun 3, 2021

ostrain commented Jun 4, 2021

cheftako commented Jun 22, 2021

ostrain commented Jul 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ostrain commented Jul 19, 2021

ostrain commented Jul 19, 2021

cheftako commented Jul 20, 2021

andrewsykim left a comment

Choose a reason for hiding this comment

ostrain commented Aug 4, 2021

ostrain commented Aug 4, 2021

ostrain commented Aug 5, 2021

liggitt commented Aug 5, 2021

k8s-ci-robot commented Aug 5, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment