Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

helmv4: Authentication issue when pulling helm charts from private OCI registries. #3193

Open
jonnylangefeld opened this issue Sep 4, 2024 · 6 comments · May be fixed by #3314
Open

helmv4: Authentication issue when pulling helm charts from private OCI registries. #3193

jonnylangefeld opened this issue Sep 4, 2024 · 6 comments · May be fixed by #3314
Assignees
@EronWright
Labels
awaiting-upstream The issue cannot be resolved without action in another repository (may be owned by Pulumi). kind/bug Some behavior is incorrect or out of spec

Comments

@jonnylangefeld
Copy link

jonnylangefeld commented Sep 4, 2024 •
edited
Loading

What happened?

helmv4 allows to download helm charts from private OCI registries by specifying username and password:

args := &helmv4.ChartArgs{
	Chart: pulumi.String("oci://myregistry.com/my-chart"),
	RepositoryOpts: helmv4.RepositoryOptsArgs{
		Username: pulumi.String("username"),
		Password: pulumi.String("password"),
	},
}

However, this results in authentication errors despite correct credentials. Different kinds for the different cloud provider registries:

on AWS using ECR 
error: an unhandled error occurred: waiting for RPCs: rpc error: code = Unknown desc = unable to locate the chart: pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials
on GCP using GCR 
error: an unhandled error occurred: waiting for RPCs: rpc error: code = Unknown desc = unable to locate the chart: unexpected status from HEAD request to https://gcr.io/v2/gke-shared-dev/upstream/helm.cilium.io/tetragon/mainfests/1.1.2: 401 Unauthorized

We did some investigation and the following turns out to be the culprit:

When reproducing the issue with the CLI then this works:

helm registry login myregistry.com -u "<username>" -p "<token>"
helm pull oci://myregistry.com/my-chart

but this doesn’t, even though it should:

helm pull oci://myregistry.com/my-chart --username="<username>" --password="<token>"

The Pulumi helm v4 provider does the equivalent to the latter.

This is recognized as an open upstream helm issue: helm/helm#12769

The main issue is the inconsistency. helm pull with username and password without previous helm registry login works for https:// helm registries, but not for oci://.

If this is an upstream helm issue, why is this posted on the pulumi-kubernetes repo?

The main reason I'm posting this here is that the helmv4 Pulumi provider copy/pastes some private upstream functions. For instance the newRegistryClient() function that the upstream helm bugfix PR is attempting to fix here by adding the username and password to the signature, is copy/pasted into this repo because it's a private function:

pulumi-kubernetes/provider/pkg/helm/tool.go

Line 341 in f5f368d

func newRegistryClient(settings *cli.EnvSettings, certFile, keyFile, caFile string, insecureSkipTLSverify, plainHTTP bool) (*registry.Client, error) {

This means that even if the upstream helm PR is merged, the helmv4 Pulumi provider would still suffer from this issue. So it has to be fixed here as well. But we might still need to wait for the upstream fix, for the new upstream NewClient() function.

The upstream helm PR has been open for 7 months with most recent activity 1 month ago. We could help push it along.

Output of pulumi about

╰─ pulumi about
CLI
Version      3.124.0
Go Version   go1.22.5
Go Compiler  gc

Host
OS       darwin
Version  14.6.1
Arch     arm64

Backend
Name           pulumi.com
URL            https://app.pulumi.com/jonny-langefeld
User           jonny-langefeld
Organizations  jonny-langefeld, snowflake
Token type     personal

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@jonnylangefeld jonnylangefeld added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Sep 4, 2024
@EronWright
Copy link
Contributor

I believe it is a known limitation that Helm OCI support relies on ambient credentials, as set by helm registry login or by docker login. The upstream PR seeks to allow for adhoc credentials via flags, and Pulumi would assumedly incorporate that by wiring up the RepositoryOpts. At this time, the RepositoryOpts is applicable only to HTTP-based chart repositories.

@EronWright EronWright removed the needs-triage Needs attention from the triage team label Sep 4, 2024
@blampe
Copy link
Contributor

blampe commented Sep 5, 2024

Strictly speaking I don't think we would need to wait for the fix upstream, we could probably plumb the credentials through with what we already have.

Related #2911

@mjeffryes mjeffryes added this to the 0.110 milestone Sep 12, 2024
@stepan-romankov
Copy link

Is it any walk around for this issue @blampe @EronWright? I'm stuck with installation of AWS Gateway AIP Controller chart as it is located only in OCI repository.

@blampe
Copy link
Contributor

blampe commented Sep 16, 2024

@stepan-romankov the AWS registry is public and doesn't need credentials.

chart: "oci://public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller-chart"

@stepan-romankov
Copy link

@stepan-romankov the AWS registry is public and doesn't need credentials.

chart: "oci://public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller-chart"

error: an unhandled error occurred: program failed:
waiting for RPCs: rpc error: code = Unknown desc = unable to locate the chart: unexpected status from HEAD request to https://public.ecr.aws/v2/aws-application-networking-k8s/aws-gateway-controller-chart/manifests/v1.0.7: 403 Forbidden

	gatewayApiController, err := helmv4.NewChart(ctx, "gateway-api-controller", &helmv4.ChartArgs{
		Name:      pulumi.String("gateway-api-controller"),
		Chart:     pulumi.String("oci://public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller-chart"),
		Version:   pulumi.String("v1.0.7"),
		Namespace: namespace.Metadata.Name(),
		Values: pulumi.Map{
			"region":       pulumi.String(region.Name),
			"clusterVpcId": eksCluster.Core.VpcId(),
			"serviceAccount": pulumi.Map{
				"create": pulumi.Bool(false),
			},
			"log": pulumi.Map{
				"level": pulumi.String("debug"),
			},
		},
	}, pulumi.Provider(kubernetesProvider))

@mjeffryes mjeffryes modified the milestones: 0.110, 0.111 Oct 2, 2024
@blampe
Copy link
Contributor

blampe commented Oct 3, 2024 •
edited
Loading

@stepan-romankov thanks for including your code. It works successfully for me:

 +   pulumi:pulumi:Stack                                                            p-k-3193-go-dev                                                                      create
 +   └─ kubernetes:helm.sh/v4:Chart                                                 gateway-api-controller                                                               create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:vpcassociationpolicies.application-networking.k8s.aws         create
 +      ├─ kubernetes:core/v1:Service                                               gateway-api-controller:p-k-3193/webhook-service                                      create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:serviceimports.application-networking.k8s.aws                 create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:dnsendpoints.externaldns.k8s.io                               create
 +      ├─ kubernetes:apps/v1:Deployment                                            gateway-api-controller:p-k-3193/gateway-api-controller-aws-gateway-controller-chart  create
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding               gateway-api-controller:gateway-api-controller-aws-gateway-controller-chart           create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:accesslogpolicies.application-networking.k8s.aws              create
 +      ├─ kubernetes:core/v1:Secret                                                gateway-api-controller:p-k-3193/webhook-cert                                         create
 +      ├─ kubernetes:core/v1:ServiceAccount                                        gateway-api-controller:p-k-3193/gateway-api-controller                               create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:serviceexports.application-networking.k8s.aws                 create
 +      ├─ kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration  gateway-api-controller:aws-appnet-gwc-mutating-webhook                               create
 +      ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole                      gateway-api-controller:gateway-api-controller-aws-gateway-controller-chart           create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:gatewayclasses.gateway.networking.k8s.io                      create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:iamauthpolicies.application-networking.k8s.aws                create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:targetgrouppolicies.application-networking.k8s.aws            create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:tlsroutes.gateway.networking.k8s.io                           create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:gateways.gateway.networking.k8s.io                            create
 +      ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:grpcroutes.gateway.networking.k8s.io                          create
 +      └─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition              gateway-api-controller:httproutes.gateway.networking.k8s.io                          create

As previously mentioned, Pulumi is currently using your ambient helm credentials. This means if you previously logged into public.ecr.aws your credentials are probably expired. I suspect helm registry logout public.ecr.aws or helm registry login will resolve your issue.

@mjeffryes mjeffryes assigned EronWright and unassigned blampe Oct 28, 2024
@mjeffryes mjeffryes modified the milestones: 0.111, 0.112 Oct 30, 2024
@EronWright EronWright added the awaiting-upstream The issue cannot be resolved without action in another repository (may be owned by Pulumi). label Oct 31, 2024
@mjeffryes mjeffryes modified the milestones: 0.112, 0.113 Nov 13, 2024
@EronWright EronWright linked a pull request Nov 20, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EronWright EronWright

Labels
awaiting-upstream The issue cannot be resolved without action in another repository (may be owned by Pulumi). kind/bug Some behavior is incorrect or out of spec
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

helmv4: Use username/password with private OCI registries pulumi/pulumi-kubernetes
5 participants
@blampe @mjeffryes @EronWright @stepan-romankov @jonnylangefeld