CredentialProvider fails to read environment variable passed into the CredentialProviderConfig file #102750
Comments
k8s-ci-robot
added
needs-sig
|
/sig Node |
k8s-ci-robot
added
sig/node
|
/assign |
|
@jicowan from the code it appears that env variables are passed to the plugin from kubelet logs the plugin is only returning exit status 1, without any details about the error!! |
|
Thanks @adisky. Yes, I saw that. If I uncomment |
|
Hello @jicowan, need some clarification. Is the intent of the above code to convey that Below code works as expected for me package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("env")
cmd.Env = []string{"AWS_PROFILE=developer", "FOO=BAR"}
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", out)
}
➜ ~ go run main.go
AWS_PROFILE=developer
FOO=BAR
➜ ~ go version
go version go1.16.5 linux/amd64
➜ ~ uname -a
Linux turing 5.12.8-051208-generic #202105281232 SMP Fri May 28 12:35:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Also, could you tell if |
|
Yes @n4j, it is not passing the Envs to the spawned process. The code for ecr-credential-provider is in the AWS cloud controller repository, https://github.com/kubernetes/cloud-provider-aws/blob/master/cmd/ecr-credential-provider/main.go. |
|
@jicowan Quick update, I had setup two test beds to reproduce the above issue, but both of them gave expected results and the issue was not reproducible. Below are my test setups: ECR Repository Test Bed 1 Execute Code import (
"fmt"
"io"
"log"
"os/exec"
)
var (
body = `{
"kind": "CredentialProviderRequest",
"apiVersion": "credentialprovider.kubelet.k8s.io/v1alpha1",
"image": "7604xxxx39.dkr.ecr.ap-south-1.amazonaws.com/whalesay:v1"
}`
)
func main() {
cmd := exec.Command("./ecr-credential-provider", "get-credentials")
cmd.Env = []string{"AWS_PROFILE=dev"}
cmd.Args = []string{"get-credentials"}
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
go func() {
defer stdin.Close()
io.WriteString(stdin, body)
}()
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", out)
}Observed Result - Below credentials were generated {
"kind": "CredentialProviderResponse",
"apiVersion": "credentialprovider.kubelet.k8s.io/v1alpha1",
"cacheKeyType": "Registry",
"cacheDuration": "21.6µs",
"auth":
{
"7604xxxx39.dkr.ecr.ap-south-1.amazonaws.com":
{
"username": "AWS",
"password": "**redacted**"
}
}
}Expected Result - Credentials should be generated Test Bed 2 Use Provider Config kind: CredentialProviderConfig
apiVersion: kubelet.config.k8s.io/v1alpha1
providers:
- name: ecr-credential-provider
matchImages:
- "*.dkr.ecr.*.amazonaws.com"
defaultCacheDuration: "12h"
apiVersion: credentialprovider.kubelet.k8s.io/v1alpha1
args:
- get-credentials
env:
- name: AWS_PROFILE
value: devExpected result - Pods should be created based off custom Observed result - Pods were created based off custom Though, my test bed runs on kubernetes version Could you give me any other minor but important detail which you feel you could had missed out earlier which could help me in replicating the issue? Or could you double check if |
|
@n4j i am using credential_process, https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html in my profile. I am also using KinD for testing, but I don't think that should matter. I don't have a credentials file. I only have a config file. |
|
This is the binary I'm calling with credential_process, https://github.com/jicowan/cognito. |
|
@n4j can you check kubelet logs for |
|
@n4j It is fetching credentials because it is falling back to other options available, It might be fetching from other credential provider if failed from external one. To confirm this you have to check kubelet logs. |
|
@adisky In my env I just have one provider and there are no fallbacks. Though I was keeping an eye on kubelet logs while performing this activity and I saw no such logs. The creds were fetched in one shot via the profile which I had specified. And as I had mentioned earlier I don't see any issues in passing env vars to credential process. Also, I did try negative cases as well by passing invalid profile names and it did fail and pods weren't pulled from ECR |
|
@n4j kubelet registers intree credential providers while startup, you need not to explicitly add any!! And I recently tried GCP credential provider, It does fall back to intree GCP credential provider after getting failed response from the one i set up. |
|
@adisky Sure, makes sense. Though as I mentioned earlier I was keeping an eye on the kubelet logs and I didn't see the logs you had mentioned |
|
@adisky i see |
|
@jicowan Had a light bulb moment and I was able to reproduce this issue, it might not be applicable in your case but it could be worth trying out. In my local setup I ran the cluster as root and
I have my aws credentials configure under my user ( Since, I have access to Would it be possible for you to check |
|
@n4j I am getting the error you mentioned, The Go code I wrote to test this fails when using cmd.Env; it's only when I import the OS environment variables that it works. This may because I am using |
|
@jicowan In order to simplify the sprawl and to pinpoint the offending component,could you remove the credential_process and Cognito and just try fetching creds via plain AWS credentials profile? |
|
@n4j, sure, however, supporting credential process is critical for my use case. I'm using Cognito as a gating mechanism to ECR. Only licensed users will be allowed to pull images from ECR. These users will not have IAM credentials (access key/secret key). They will authenticate to AWS using Cognito. |
|
The following configuration works when I specify access key/secret key: |
|
When I switch to a profile with credential process I get the following error: |
|
@jicowan isn't it the issue of ecr-credential-provider? This is the job of ecr-credential-provider to fetch credentials and return result in expected format to kubelet. |
|
@adisky why does it work when I run it from the command line? Why does it work when I import envs from the OS? |
@adisky Here's the execution flow @jicowan Does the above paints a fairly accurate picture? If not then feel free to correct :) |
I included my KinD configuration in the beginning of this thread. Can you use that @n4j? The The flow @adisky is as @n4j describes above. The During my testing (without the kubelet) I am able to get the proper response for the kubelet when referencing the AWS profile using |
|
@jicowan Let me replicate your setup in my local env and see if I can reproduce this issue. Also, could you tell what your AWS You can use placeholder values for sensitive information but correct the binary path if it's not |
|
@n4j yes, that looks right. Ideally the cognito binary is located in a directory in the shell's PATH but it also works when you specify the full path in the config file like you've done. Are you planning to setup cognito in AWS? You'll need a cognito user pool and configure federation. |
|
@jicowan Yes, I am planning to setup Cognito Identity pool in my AWS Account. |
|
@jicowan So I replicated your setup on my local machine with slight change, instead of using AWS Cognito for generating credentials via Below was my initial config file After I ran local cluster with this setup I noticed couple of log lines like below: I did some basic validation of perms + I even moved the binary to couple of well know locations but nothing worked Finally I modified my # Env defines additional environment variables to expose to the process. These
# are unioned with the host's environment, as well as variables client-go uses
# to pass argument to the plugin.
# +optional
env:
- name: AWS_PROFILE
value: "cognito"
- name: AWS_CONFIG_FILE
value: "/root/.aws/config"
- name: AWS_SDK_LOAD_CONFIG
value: "1"
- name: PATH
value: "$PATH:/usr/local/bin:/usr/bin"The logs which I had put in my custom credential provider printed following env vars validating the fact that, env vars are being passed In your logs do you see a log line something like above? If not then could you pass full contents of |
|
@n4j thanks for doing this. I never tried adding the PATH to the list of environment variables. I honestly didn't think it was necessary because I was referencing the full path to the binary in credential_process. I also never saw an error about not being able to find the executable in $PATH. I'll re-test after adding PATH as an ENV and let you know the results. |
|
@n4j Adding the PATH did not work for me. I'm happy to send you the cognito binary with a username/password that you can use for testing. I have only tested this with KinD, but I don't think that is the issue here since it fails with exit code when when I try using |
^^ is my credential provider config |
|
@jicowan Thanks for the logs. I have cloned your code in my repo and I'll substitute my Cognito creds. If it doesn't work out then I'll reach out to you for creds 🙂 |
|
@jicowan After several rounds of troubleshooting I was able to get My credential loading setup is Initially, I got the same error as yours, then I added couple of custom log lines to print env vars in What I noticed in my KinD setup is that AWS SDK was logging an error because it couldn't find any "providers", though I could verify in the I altered my env vars section of the env:
- name: AWS_PROFILE
value: "cognito"
- name: AWS_SDK_LOAD_CONFIG
value: "1"
- name: HOME # this is what did the trick for me
value: "/root"
- name: PATH
value: "$PATH:/usr/local/bin:/usr/bin:/usr/local/kubelet"Apparently, what I noticed is Can you please add If this doesn't work, then I'll send you the modified version of the |
|
@n4j That did it! Thanks for helping me troubleshoot this. I wonder if this is unique to KinD or whether we would see this behavior in a "normal" cluster (i'll test it later). I will tell the developers who wrote the ecr-credential-provider to include an argument to increase the verbosity or to output logs to a file so this is easier to troubleshoot in the future. |
|
@jicowan What I have observed is, Could you please re-open this issue? |
|
@n4j sure thing. |
|
@n4j: The label In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@fromanirh @liggitt To me this looks like an issue with the kubelet and I have observed this behaviour with What are your thoughts on this? If this looks like a bug then can you accept the triage? I would fix this. |
|
@n4j Thanks for looking into the issue, you are right the spec does says that. Feel free to fix it |
k8s-ci-robot
added
triage/accepted
|
@adisky Thanks!!! |
|
/area kubelet |
|
@n4j : I am facing the similar issue on my bare metal cluster This is my config Can you help. It works when I try to execute the credentials binary from local but doesn't work in the pod. Can you pls guide me |
What happened:
I am attempting to pull an image from ECR using the new kubelet image credential provider feature which is an alpha features in Kubernetes v1.20. My CredentialProviderConfig file includes the environment variable AWS_PROFILE which resides in the default (~/.aws) directory. The profile references a
credential_processwhich allows me to execute a binary that returns a set of temporary AWS credentials. When I try running a pod with a container from ECR, I see the following errors in the kubelet logs:However, when I type the following into the command line on the kubelet:
I get the following response:
It doesn't appear the environment variables in my Config are being passed to the Kubelet when it executes the ecr-credential-provider binary.
Below is Config file:
What you expected to happen:
I expect for the Kubelet to pass the environment variables in the Config file when it executes the ecr-credential-provider binary.
How to reproduce it (as minimally and precisely as possible):
I am using KinD to test this feature. Here is my config:
Anything else we need to know?:
No
Environment:
kubectl version): 1.20cat /etc/os-release): Ubuntu 20.10uname -a): 5.10cc @ayberk @andrewsykim
The text was updated successfully, but these errors were encountered: