Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Passing args to helm #36

Open
moreiramarti opened this issue Mar 3, 2023 · 11 comments
Open

Passing args to helm #36

moreiramarti opened this issue Mar 3, 2023 · 11 comments

Comments

@moreiramarti
Copy link

Hello,

I'm experiencing a strange behavior when I'm using helmfile.d tree structure.
Some args are passed to helm pull command but they shouldn't (cf output below)

It seems passing args from helmfile to helm is not a good practice (doc)

Do you think it could be improved or make it optional ?
For now, I removed the --args parameter from the script like I'm not using it.
@moreiramarti
Copy link
Author 

argocd app get ephemeral-pull-helmfile --hard-refresh
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
WARN[0003] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
Name:               argocd/ephemeral-pull-helmfile
Project:            infra
Server:             testbed
Namespace:          ephemeral-pull-helmfile
URL:                https://argocd.voodoo-infra-sandbox.io/applications/ephemeral-pull-helmfile
Repo:               https://github.com/VoodooTeam/argocd-sandbox-infra.git
Target:             master
Path:               ephemeral/
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Unknown
Health Status:      Healthy

CONDITION        MESSAGE  LAST TRANSITION
ComparisonError  rpc error: code = Unknown desc = plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `bash -c /usr/local/bin/argo-cd-helmfile.sh generate` failed exit status 1: v3.10.3+g835b733
helmfile version 0.151.0
starting generate
Templating release=db, chart=bitnami/postgresql
Templating release=db-migration, chart=../charts/db-migration
Pulling 456072703506.dkr.ecr.eu-west-1.amazonaws.com/helm/stateless-service:6.0.1
in helmfile.d/01_application.yaml: [release "ephemeral": command "/usr/local/bin/helm" exited with non-zero status:

PATH:
  /usr/local/bin/helm

ARGS:
  0: /usr/local/bin/helm (19 bytes)
  1: pull (4 bytes)
  2: oci://456072703506.dkr.ecr.eu-west-1.amazonaws.com/helm/stateless-service (73 bytes)
  3: --version (9 bytes)
  4: 6.0.1 (5 bytes)
  5: --destination (13 bytes)
  6: /tmp/helmfile2452714502/ephemeral-pull-helmfile/ephemeral/stateless-service/6.0.1 (81 bytes)
  7: --untar (7 bytes)
  8: --kube-version=1.23 (19 bytes)
  9: --api-versions=acme.cert-manager.io/v1 (38 bytes)
  10: --api-versions=acme.cert-manager.io/v1/Challenge (48 bytes)
  11: --api-versions=acme.cert-manager.io/v1/Order (44 bytes)
  12: --api-versions=admissionregistration.k8s.io/v1 (46 bytes)
  13: --api-versions=admissionregistration.k8s.io/v1/MutatingWebhookConfiguration (75 bytes)
  14: --api-versions=admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration (77 bytes)
  15: --api-versions=apiextensions.k8s.io/v1 (38 bytes)
  16: --api-versions=apiextensions.k8s.io/v1/CustomResourceDefinition (63 bytes)
  17: --api-versions=apiregistration.k8s.io/v1 (40 bytes)
  18: --api-versions=apiregistration.k8s.io/v1/APIService (51 bytes)
  19: --api-versions=apps/v1 (22 bytes)
  20: --api-versions=apps/v1/ControllerRevision (41 bytes)
  21: --api-versions=apps/v1/DaemonSet (32 bytes)
  22: --api-versions=apps/v1/Deployment (33 bytes)
  23: --api-versions=apps/v1/ReplicaSet (33 bytes)
  24: --api-versions=apps/v1/StatefulSet (34 bytes)
  25: --api-versions=argoproj.io/v1alpha1 (35 bytes)
  26: --api-versions=argoproj.io/v1alpha1/AppProject (46 bytes)
  27: --api-versions=argoproj.io/v1alpha1/Application (47 bytes)
  28: --api-versions=argoproj.io/v1alpha1/ApplicationSet (50 bytes)
  29: --api-versions=argoproj.io/v1alpha1/ArgoCDExtension (51 bytes)
  30: --api-versions=autoscaling/v1 (29 bytes)
  31: --api-versions=autoscaling/v1/HorizontalPodAutoscaler (53 bytes)
  32: --api-versions=autoscaling/v2 (29 bytes)
  33: --api-versions=autoscaling/v2/HorizontalPodAutoscaler (53 bytes)
  34: --api-versions=autoscaling/v2beta1 (34 bytes)
  35: --api-versions=autoscaling/v2beta1/HorizontalPodAutoscaler (58 bytes)
  36: --api-versions=autoscaling/v2beta2 (34 bytes)
  37: --api-versions=autoscaling/v2beta2/HorizontalPodAutoscaler (58 bytes)
  38: --api-versions=batch/v1 (23 bytes)
  39: --api-versions=batch/v1/CronJob (31 bytes)
  40: --api-versions=batch/v1/Job (27 bytes)
  41: --api-versions=batch/v1beta1 (28 bytes)
  42: --api-versions=batch/v1beta1/CronJob (36 bytes)
  43: --api-versions=bitnami.com/v1alpha1 (35 bytes)
  44: --api-versions=bitnami.com/v1alpha1/SealedSecret (48 bytes)
  45: --api-versions=cert-manager.io/v1 (33 bytes)
  46: --api-versions=cert-manager.io/v1/Certificate (45 bytes)
  47: --api-versions=cert-manager.io/v1/CertificateRequest (52 bytes)
  48: --api-versions=cert-manager.io/v1/ClusterIssuer (47 bytes)
  49: --api-versions=cert-manager.io/v1/Issuer (40 bytes)
  50: --api-versions=certificates.k8s.io/v1 (37 bytes)
  51: --api-versions=certificates.k8s.io/v1/CertificateSigningRequest (63 bytes)
  52: --api-versions=coordination.k8s.io/v1 (37 bytes)
  53: --api-versions=coordination.k8s.io/v1/Lease (43 bytes)
  54: --api-versions=crd.k8s.amazonaws.com/v1alpha1 (45 bytes)
  55: --api-versions=crd.k8s.amazonaws.com/v1alpha1/ENIConfig (55 bytes)
  56: --api-versions=discovery.k8s.io/v1 (34 bytes)
  57: --api-versions=discovery.k8s.io/v1/EndpointSlice (48 bytes)
  58: --api-versions=discovery.k8s.io/v1beta1 (39 bytes)
  59: --api-versions=discovery.k8s.io/v1beta1/EndpointSlice (53 bytes)
  60: --api-versions=elbv2.k8s.aws/v1alpha1 (37 bytes)
  61: --api-versions=elbv2.k8s.aws/v1alpha1/TargetGroupBinding (56 bytes)
  62: --api-versions=elbv2.k8s.aws/v1beta1 (36 bytes)
  63: --api-versions=elbv2.k8s.aws/v1beta1/IngressClassParams (55 bytes)
  64: --api-versions=elbv2.k8s.aws/v1beta1/TargetGroupBinding (55 bytes)
  65: --api-versions=events.k8s.io/v1 (31 bytes)
  66: --api-versions=events.k8s.io/v1/Event (37 bytes)
  67: --api-versions=events.k8s.io/v1beta1 (36 bytes)
  68: --api-versions=events.k8s.io/v1beta1/Event (42 bytes)
  69: --api-versions=flowcontrol.apiserver.k8s.io/v1beta1 (51 bytes)
  70: --api-versions=flowcontrol.apiserver.k8s.io/v1beta1/FlowSchema (62 bytes)
  71: --api-versions=flowcontrol.apiserver.k8s.io/v1beta1/PriorityLevelConfiguration (78 bytes)
  72: --api-versions=flowcontrol.apiserver.k8s.io/v1beta2 (51 bytes)
  73: --api-versions=flowcontrol.apiserver.k8s.io/v1beta2/FlowSchema (62 bytes)
  74: --api-versions=flowcontrol.apiserver.k8s.io/v1beta2/PriorityLevelConfiguration (78 bytes)
  75: --api-versions=irsa.voodoo.io/v1alpha1 (38 bytes)
  76: --api-versions=irsa.voodoo.io/v1alpha1/IamRoleServiceAccount (60 bytes)
  77: --api-versions=irsa.voodoo.io/v1alpha1/Policy (45 bytes)
  78: --api-versions=irsa.voodoo.io/v1alpha1/Role (43 bytes)
  79: --api-versions=k6.io/v1alpha1 (29 bytes)
  80: --api-versions=k6.io/v1alpha1/K6 (32 bytes)
  81: --api-versions=monitoring.coreos.com/v1 (39 bytes)
  82: --api-versions=monitoring.coreos.com/v1/Alertmanager (52 bytes)
  83: --api-versions=monitoring.coreos.com/v1/PodMonitor (50 bytes)
  84: --api-versions=monitoring.coreos.com/v1/Probe (45 bytes)
  85: --api-versions=monitoring.coreos.com/v1/Prometheus (50 bytes)
  86: --api-versions=monitoring.coreos.com/v1/PrometheusRule (54 bytes)
  87: --api-versions=monitoring.coreos.com/v1/ServiceMonitor (54 bytes)
  88: --api-versions=monitoring.coreos.com/v1/ThanosRuler (51 bytes)
  89: --api-versions=monitoring.coreos.com/v1alpha1 (45 bytes)
  90: --api-versions=monitoring.coreos.com/v1alpha1/AlertmanagerConfig (64 bytes)
  91: --api-versions=monitoring.grafana.com/v1alpha1 (46 bytes)
  92: --api-versions=monitoring.grafana.com/v1alpha1/GrafanaAgent (59 bytes)
  93: --api-versions=monitoring.grafana.com/v1alpha1/Integration (58 bytes)
  94: --api-versions=monitoring.grafana.com/v1alpha1/LogsInstance (59 bytes)
  95: --api-versions=monitoring.grafana.com/v1alpha1/MetricsInstance (62 bytes)
  96: --api-versions=monitoring.grafana.com/v1alpha1/PodLogs (54 bytes)
  97: --api-versions=networking.k8s.io/v1 (35 bytes)
  98: --api-versions=networking.k8s.io/v1/Ingress (43 bytes)
  99: --api-versions=networking.k8s.io/v1/IngressClass (48 bytes)
  100: --api-versions=networking.k8s.io/v1/NetworkPolicy (49 bytes)
  101: --api-versions=node.k8s.io/v1 (29 bytes)
  102: --api-versions=node.k8s.io/v1/RuntimeClass (42 bytes)
  103: --api-versions=node.k8s.io/v1beta1 (34 bytes)
  104: --api-versions=node.k8s.io/v1beta1/RuntimeClass (47 bytes)
  105: --api-versions=policy/v1 (24 bytes)
  106: --api-versions=policy/v1/PodDisruptionBudget (44 bytes)
  107: --api-versions=policy/v1beta1 (29 bytes)
  108: --api-versions=policy/v1beta1/PodDisruptionBudget (49 bytes)
  109: --api-versions=policy/v1beta1/PodSecurityPolicy (47 bytes)
  110: --api-versions=rbac.authorization.k8s.io/v1 (43 bytes)
  111: --api-versions=rbac.authorization.k8s.io/v1/ClusterRole (55 bytes)
  112: --api-versions=rbac.authorization.k8s.io/v1/ClusterRoleBinding (62 bytes)
  113: --api-versions=rbac.authorization.k8s.io/v1/Role (48 bytes)
  114: --api-versions=rbac.authorization.k8s.io/v1/RoleBinding (55 bytes)
  115: --api-versions=scheduling.k8s.io/v1 (35 bytes)
  116: --api-versions=scheduling.k8s.io/v1/PriorityClass (49 bytes)
  117: --api-versions=storage.k8s.io/v1 (32 bytes)
  118: --api-versions=storage.k8s.io/v1/CSIDriver (42 bytes)
  119: --api-versions=storage.k8s.io/v1/CSINode (40 bytes)
  120: --api-versions=storage.k8s.io/v1/StorageClass (45 bytes)
  121: --api-versions=storage.k8s.io/v1/VolumeAttachment (49 bytes)
  122: --api-versions=storage.k8s.io/v1beta1 (37 bytes)
  123: --api-versions=storage.k8s.io/v1beta1/CSIStorageCapacity (56 bytes)
  124: --api-versions=v1 (17 bytes)
  125: --api-versions=v1/ConfigMap (27 bytes)
  126: --api-versions=v1/Endpoints (27 bytes)
  127: --api-versions=v1/Event (23 bytes)
  128: --api-versions=v1/LimitRange (28 bytes)
  129: --api-versions=v1/Namespace (27 bytes)
  130: --api-versions=v1/Node (22 bytes)
  131: --api-versions=v1/PersistentVolume (34 bytes)
  132: --api-versions=v1/PersistentVolumeClaim (39 bytes)
  133: --api-versions=v1/Pod (21 bytes)
  134: --api-versions=v1/PodTemplate (29 bytes)
  135: --api-versions=v1/ReplicationController (39 bytes)
  136: --api-versions=v1/ResourceQuota (31 bytes)
  137: --api-versions=v1/Secret (24 bytes)
  138: --api-versions=v1/Service (25 bytes)
  139: --api-versions=v1/ServiceAccount (32 bytes)
  140: --api-versions=velero.io/v1 (27 bytes)
  141: --api-versions=velero.io/v1/Backup (34 bytes)
  142: --api-versions=velero.io/v1/BackupStorageLocation (49 bytes)
  143: --api-versions=velero.io/v1/DeleteBackupRequest (47 bytes)
  144: --api-versions=velero.io/v1/DownloadRequest (43 bytes)
  145: --api-versions=velero.io/v1/PodVolumeBackup (43 bytes)
  146: --api-versions=velero.io/v1/PodVolumeRestore (44 bytes)
  147: --api-versions=velero.io/v1/ResticRepository (44 bytes)
  148: --api-versions=velero.io/v1/Restore (35 bytes)
  149: --api-versions=velero.io/v1/Schedule (36 bytes)
  150: --api-versions=velero.io/v1/ServerStatusRequest (47 bytes)
  151: --api-versions=velero.io/v1/VolumeSnapshotLocation (50 bytes)
  152: --api-versions=vpcresources.k8s.aws/v1beta1 (43 bytes)
  153: --api-versions=vpcresources.k8s.aws/v1beta1/SecurityGroupPolicy (63 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  Error: unknown flag: --kube-version

COMBINED OUTPUT:
  Error: unknown flag: --kube-version]  2023-03-03 10:31:41 +0100 CET


GROUP              KIND                 NAMESPACE                NAME                   STATUS   HEALTH   HOOK  MESSAGE
                   Secret               ephemeral-pull-helmfile  ephemeral-postgres     Unknown                 
                   Service              ephemeral-pull-helmfile  ephemeral              Unknown  Healthy        
                   Service              ephemeral-pull-helmfile  ephemeral-postgres     Unknown  Healthy        
                   Service              ephemeral-pull-helmfile  ephemeral-postgres-hl  Unknown  Healthy        
                   ServiceAccount       ephemeral-pull-helmfile  ephemeral              Unknown                 
apps               Deployment           ephemeral-pull-helmfile  ephemeral              Unknown  Healthy        
apps               StatefulSet          ephemeral-pull-helmfile  ephemeral-postgres     Unknown  Healthy        
batch              Job                  ephemeral-pull-helmfile  ephemeral-db-migrate   Unknown  Healthy        
networking.k8s.io  Ingress              ephemeral-pull-helmfile  ephemeral              Unknown  Healthy        
policy             PodDisruptionBudget  ephemeral-pull-helmfile  ephemeral              Unknown

@travisghansen
Copy link
Owner

Can you send over the app definition so I can see what you have set?

@travisghansen
Copy link
Owner

Or the helmfile content if you have the args defined there.

@moreiramarti
Copy link
Author

Like I'm using multiple yaml in helmfile.d directory, I don't have a helmfile.yaml.
This is my helmfile yaml :

  • 00_database.yaml
bases:
- ../common/repositories.yaml
- ../common/environments.yaml

releases:
- name: db
  chart: bitnami/postgresql
  version: 12.1.0
  values:
    - templates/postgresql.yaml.gotmpl
- name: db-migration
  chart: ../charts/db-migration
  version: 0.1.0
  values:
  - templates/db_migration.yaml.gotmpl
  • 01_application.yaml
bases:
- ../common/repositories.yaml
- ../common/environments.yaml
 
releases:
- name: ephemeral
  chart: voodooecr/stateless-service
  version: 6.0.1
  values:
    - templates/ephemeral.yaml.gotmpl

And my app definition :

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ephemeral-as
spec:
  generators:
  - pullRequest:
      github:
        owner: VoodooTeam
        repo: ephemeral-env
        tokenRef:
          secretName: argocd-github-secret
          key: pat
      requeueAfterSeconds: 1800
      filters:
      - branchMatch: pull-.*
  template:
    metadata:
      name: ephemeral-{{branch}}
    spec:
      source:
        repoURL: https://github.com/xxx/argocd-sandbox.git
        targetRevision: master
        path: ephemeral/
        plugin:
          parameters:
          - name: ephemeral_values
            map:
              image.tag: "{{head_sha}}"
              ingress.host: "{{branch_slug}}.xxx.xx"
              environment.CSRF_TRUSTED_ORIGINS: "https://{{branch_slug}}.xxx.xx"
      project: infra
      destination:
        name: testbed
        namespace: ephemeral-{{branch}}
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true

@travisghansen
Copy link
Owner

I suspect this has some behavior difference due to the repository being oci. Conceptually the goal is to make sure all repos are added and necessary charts downloads during the init phase (which occurs before generate phase). The error above is occurring during the generate phase which leads me to believe something fundamentally different with how helmfile handles oci vs non-oci registries. Can you send the (cleansed) content of the repositories.yaml file?

@travisghansen
Copy link
Owner

I've opened this: helmfile/helmfile#727

@moreiramarti
Copy link
Author

Sorry for the delay
The fiie :

repositories:
- name: bitnami
  url: https://charts.bitnami.com/bitnami
- name: localstack
  url: https://localstack.github.io/helm-charts
- name: ecr
  url: xxx.dkr.ecr.eu-west-1.amazonaws.com/helm
  oci: true

@travisghansen travisghansen mentioned this issue Mar 29, 2023
Open
@welderpb
Copy link

welderpb commented Apr 4, 2023

@travisghansen I have the same issue with oci repositories. I did some investigation and can say:
helmfile before pass template command passing pull command for oci repositories to helm.
helm 'pull' command doesn't support --kube-version and --api-versions arguments..
also I checked 'fetch' command before 'template' command. it is doesn't help. Template still pulling charts.

you are always passing this arguments to 'template' command:

 # TODO: support post process pipeline here
    ${helmfile} \
      template \
      --skip-deps ${INTERNAL_HELMFILE_TEMPLATE_OPTIONS} \
      --args "${INTERNAL_HELM_TEMPLATE_OPTIONS} ${HELM_TEMPLATE_OPTIONS}" \
      ${HELMFILE_TEMPLATE_OPTIONS}
    ;;

So, my suggestion is make this arguments are optional, or add a new ENV to possibility disable this args.

@travisghansen
Copy link
Owner

@welderpb thanks! I'm unaware of another option to ensure proper kubeapi versions etc can be passed down to helm, am I missing something?

@welderpb
Copy link

welderpb commented Apr 5, 2023

@travisghansen it is not necessary to pass this arguments to helm templating, it is should be optional.

@travisghansen
Copy link
Owner

In many situations they absolutely are required unfortunately :( we’ll work closely with helmfile to make sure we sanely cover the necessary use cases and the issue will go away.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@travisghansen @welderpb @moreiramarti