KubeEnv var to enable Hyper-V in GCE Windows nodes created with kube-up #105999

mauriciopoppe · 2021-10-29T06:59:03Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

We'd like to create VHDs as part of the E2E tests of Local Volume Provisioner (kubernetes-sigs/sig-storage-local-static-provisioner#277), VHDs require Hyper-V to be enabled, this PR adds a new KubeEnv flag to enable Hyper-V in GCE Windows VMs, the feature is opt-in and disabled by default.

Testing

I created a GCE cluster with the following env vars (in addition to the common ones):

export NODE_SIZE=e2-standard-2
export WINDOWS_ENABLE_HYPERV=true

Once the Windows VM was up I SSHd to it and verified that the command New-VHD is available, I could create a VHD with it:

PS C:\Users\mauriciopoppe> New-VHD -Path C:\var\lib\kubelet\plugins\testplugin-0.csi.io\disk-0.vhdx -SizeBytes 10485760
ComputerName            : E2E-TEST-MAURIC
Path                    : C:\var\lib\kubelet\plugins\testplugin-0.csi.io\disk-0.vhdx
VhdFormat               : VHDX
VhdType                 : Dynamic
FileSize                : 4194304
Size                    : 10485760
MinimumSize             :
LogicalSectorSize       : 512
PhysicalSectorSize      : 4096
BlockSize               : 33554432
ParentPath              :
DiskIdentifier          : <REDACTED>
FragmentationPercentage : 0
Alignment               : 1
Attached                : False
DiskNumber              :
IsPMEMCompatible        : False
AddressAbstractionType  : None
Number                  :

PS C:\Users\mauriciopoppe> Mount-VHD -Path C:\var\lib\kubelet\plugins\testplugin-0.csi.io\disk-0.vhdx

PS C:\Users\mauriciopoppe> Get-Disk

Number Friendly Name                                                           Serial Number                    HealthStatus         OperationalStatus      Total Size Partition
                                                                                                                                                                       Style
------ -------------                                                           -------------                    ------------         -----------------      ---------- ----------
0      Google PersistentDisk                                                                                    Healthy              Online                     100 GB GPT
1      Msft Virtual Disk                                                                                        Healthy              Online                      10 MB RAW

Installation logs:

2021/10/29 06:42:44 GCEMetadataScripts: windows-startup-script-ps1: Enabling Windows 'HyperV' feature
2021/10/29 06:42:45 GCEInstanceSetup: Activating Windows(R), ServerDatacenter edition (REDACTED) ...
2021/10/29 06:42:46 GCEInstanceSetup: Product activated successfully.
2021/10/29 06:42:48 GCEInstanceSetup: Activation successful.
2021/10/29 06:43:03 GCEMetadataScripts: windows-startup-script-ps1: WARNING: Restart is suppressed because NoRestart is specified.
2021/10/29 06:43:03 GCEMetadataScripts: windows-startup-script-ps1:
2021/10/29 06:43:03 GCEMetadataScripts: windows-startup-script-ps1:
2021/10/29 06:43:03 GCEMetadataScripts: windows-startup-script-ps1: Path          :
2021/10/29 06:43:03 GCEMetadataScripts: windows-startup-script-ps1: Online        : True
2021/10/29 06:43:03 GCEMetadataScripts: windows-startup-script-ps1: RestartNeeded : True
2021/10/29 06:43:03 GCEMetadataScripts: windows-startup-script-ps1:
2021/10/29 06:43:08 GCEMetadataScripts: windows-startup-script-ps1: WARNING: Restart is suppressed because NoRestart is specified.
2021/10/29 06:43:08 GCEMetadataScripts: windows-startup-script-ps1: Path          :
2021/10/29 06:43:08 GCEMetadataScripts: windows-startup-script-ps1: Online        : True
2021/10/29 06:43:08 GCEMetadataScripts: windows-startup-script-ps1: RestartNeeded : True
2021/10/29 06:43:08 GCEMetadataScripts: windows-startup-script-ps1:
2021/10/29 06:43:08 GCEMetadataScripts: windows-startup-script-ps1: Restarting computer after enabling Windows Hyper-V feature
2021/10/29 06:43:21 GCEMetadataScripts: windows-startup-script-ps1: Installing Windows 'Containers' feature
2021/10/29 06:43:46 GCEMetadataScripts: windows-startup-script-ps1:
2021/10/29 06:43:46 GCEMetadataScripts: windows-startup-script-ps1: Success       : True
2021/10/29 06:43:46 GCEMetadataScripts: windows-startup-script-ps1: RestartNeeded : Yes
2021/10/29 06:43:46 GCEMetadataScripts: windows-startup-script-ps1: FeatureResult : {Containers}
2021/10/29 06:43:46 GCEMetadataScripts: windows-startup-script-ps1: ExitCode      : SuccessRestartRequired
2021/10/29 06:43:46 GCEMetadataScripts: windows-startup-script-ps1:
2021/10/29 06:43:46 GCEMetadataScripts: windows-startup-script-ps1: WARNING: You must restart this server to finish the installation process.
2021/10/29 06:43:46 GCEMetadataScripts: windows-startup-script-ps1: Restarting computer after enabling Windows Containers feature
2021/10/29 06:43:46 GCEMetadataScripts: windows-startup-script-ps1 exit status 0

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?

Support to enable Hyper-V in GCE Windows Nodes created with kube-up

/cc @pjh @jingxu97

mauriciopoppe · 2021-10-29T07:03:56Z

/sig windows

mauriciopoppe · 2021-10-29T16:51:00Z

/test pull-kubernetes-e2e-gce-ubuntu-containerd

jingxu97 · 2021-10-29T17:24:25Z

cluster/gce/windows/configure.ps1

@@ -118,17 +118,32 @@ try {

  Dump-DebugInfoToConsole

+  $kube_env = Fetch-KubeEnv


why moved this part up?

the first check to see if we have to install Hyper-V is to see the value of kube_env[WINDOWS_ENABLE_HYPERV], these lines fetch kubeEnv from the instance metadata

jingxu97 · 2021-10-29T17:25:46Z

cluster/gce/windows/k8s-node-setup.psm1

+function Enable-HyperVFeature {
+  Log-Output "Enabling Windows 'HyperV' feature"
+  Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All -NoRestart
+  Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Management-PowerShell -All -NoRestart


why -NoRestart option here?

We could restart after enabling each feature but instead we could batch all of the installations that require a restart to do a restart after all of them completed, what I saw is:

Install Hyper-V (requires restart but it's deferred) Install Hyper-V PowerShell tools Install Container support (requires restart but it's deferred) Restart Computer

jingxu97 · 2021-10-29T17:54:36Z

/lgtm

pjh · 2021-10-29T22:13:14Z

/triage accepted
/priority important-soon

pjh · 2021-10-29T22:19:55Z

/lgtm
/approve

Thanks for the clean code and the detailed PR description.

k8s-ci-robot · 2021-10-29T22:20:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mauriciopoppe, pjh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cluster/gce/OWNERS~~ [pjh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot requested review from jingxu97 and pjh October 29, 2021 06:59

mauriciopoppe changed the title ~~KubeEnv var to enable Hyper-V in Windows~~ KubeEnv var to enable Hyper-V in GCE Windows nodes created with kube-up Oct 29, 2021

k8s-ci-robot added area/provider/gcp Issues or PRs related to gcp provider sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 29, 2021

k8s-ci-robot added the sig/windows Categorizes an issue or PR as relevant to SIG Windows. label Oct 29, 2021

KubeEnv var to enable Hyper-V in Windows

9e29cc0

mauriciopoppe force-pushed the windows-hyperv branch from 15ec1e1 to 9e29cc0 Compare October 29, 2021 07:07

jingxu97 reviewed Oct 29, 2021

View reviewed changes

k8s-ci-robot assigned jingxu97 Oct 29, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 29, 2021

k8s-ci-robot assigned pjh Oct 29, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 29, 2021

k8s-ci-robot merged commit 324eff5 into kubernetes:master Oct 29, 2021

k8s-ci-robot added this to the v1.23 milestone Oct 29, 2021

mauriciopoppe mentioned this pull request Jan 9, 2022

Scripts to run the integration tests with a real kubernetes VM kubernetes-csi/csi-proxy#190

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KubeEnv var to enable Hyper-V in GCE Windows nodes created with kube-up #105999

KubeEnv var to enable Hyper-V in GCE Windows nodes created with kube-up #105999

mauriciopoppe commented Oct 29, 2021 •

edited

mauriciopoppe commented Oct 29, 2021

mauriciopoppe commented Oct 29, 2021

jingxu97 Oct 29, 2021

mauriciopoppe Oct 29, 2021

jingxu97 Oct 29, 2021

mauriciopoppe Oct 29, 2021 •

edited

jingxu97 commented Oct 29, 2021

pjh commented Oct 29, 2021

pjh commented Oct 29, 2021

k8s-ci-robot commented Oct 29, 2021

		@@ -118,17 +118,32 @@ try {

		Dump-DebugInfoToConsole

		$kube_env = Fetch-KubeEnv

KubeEnv var to enable Hyper-V in GCE Windows nodes created with kube-up #105999

KubeEnv var to enable Hyper-V in GCE Windows nodes created with kube-up #105999

Conversation

mauriciopoppe commented Oct 29, 2021 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Does this PR introduce a user-facing change?

mauriciopoppe commented Oct 29, 2021

mauriciopoppe commented Oct 29, 2021

jingxu97 Oct 29, 2021

Choose a reason for hiding this comment

mauriciopoppe Oct 29, 2021

Choose a reason for hiding this comment

jingxu97 Oct 29, 2021

Choose a reason for hiding this comment

mauriciopoppe Oct 29, 2021 • edited

Choose a reason for hiding this comment

jingxu97 commented Oct 29, 2021

pjh commented Oct 29, 2021

pjh commented Oct 29, 2021

k8s-ci-robot commented Oct 29, 2021

mauriciopoppe commented Oct 29, 2021 •

edited

mauriciopoppe Oct 29, 2021 •

edited