Failures: Nightly e2e "min" version tests

[sheidkamp]

Which tests failed?

Which tests failed?
https://github.com/solo-io/gloo/actions/runs/11966777661/job/33388385030

• TestK8sGatewayIstioAutoMtls/IstioIntegrationAutoMtls/TestMtlsDisablePeerAuth*
• TestK8sGatewayIstio/HeadlessSvc/TestConfigureRoutingHeadlessSvc
• TestK8sGatewayIstioRevision/IstioIntegration/TestPermissivePeerAuth

All seem to be failing with an error similar to:

=== NAME  TestK8sGatewayIstioAutoMtls
    objects.go:27: 
        Timed out after 20.001s.
        object /, Kind= default/gloo-proxy-gw should be available in cluster
        The function passed to Eventually failed at /home/runner/work/gloo/gloo/test/kubernetes/testutils/assertions/objects.go:22 with:
        object /, Kind= default/gloo-proxy-gw should be available in cluster
        Unexpected error:
            <*errors.StatusError | 0xc001a8d9a0>: 
            services "gloo-proxy-gw" not found
            {
                ErrStatus: {
                    TypeMeta: {Kind: "", APIVersion: ""},
                    ListMeta: {
                        SelfLink: "",
                        ResourceVersion: "",
                        Continue: "",
                        RemainingItemCount: nil,
                    },
                    Status: "Failure",
                    Message: "services \"gloo-proxy-gw\" not found",
                    Reason: "NotFound",
                    Details: {
                        Name: "gloo-proxy-gw",
                        Group: "",
                        Kind: "services",
                        UID: "",
                        Causes: nil,
                        RetryAfterSeconds: 0,
                    },
                    Code: 404,
                },
            }
        occurred

Reproducable locally, will only pass with ISTIO_VERSION=1.22.0

When using istio versions < 1.22.0, the Gateway is never accepted, gets stuck at:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  annotations:
    meta.helm.sh/release-name: gloo
    meta.helm.sh/release-namespace: istio-k8s-gw-test
  creationTimestamp: "2024-11-22T20:57:29Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    gloo: kube-gateway
  name: gloo-gateway
  resourceVersion: "8871"
  uid: 0f704975-5ec0-4dfd-9026-7cf4964df460
spec:
  controllerName: solo.io/gloo-gateway
  parametersRef:
    group: gateway.gloo.solo.io
    kind: GatewayParameters
    name: gloo-gateway
    namespace: istio-k8s-gw-test
status:
  conditions:
  - lastTransitionTime: "1970-01-01T00:00:00Z"
    message: Waiting for controller
    reason: Pending
    status: Unknown
    type: Accepted

When running with Istio 1.21.0, the following errors are in the gloo pod logs that are not present for 1.22.0:

2024-11-23T01:23:07.927679Z	info	klog	pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.DestinationRule: the server could not find the requested resource (get destinationrules.networking.istio.io)
2024-11-23T01:23:07.927704Z	error	watch error in cluster : failed to list *v1.DestinationRule: the server could not find the requested resource (get destinationrules.networking.istio.io)

and

2024-11-23T01:23:22.289033Z	info	klog	pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.DestinationRule: the server could not find the requested resource (get destinationrules.networking.istio.io)
2024-11-23T01:23:22.289214Z	error	watch error in cluster : failed to list *v1.DestinationRule: the server could not find the requested resource (get destinationrules.networking.istio.io)
2024-11-23T01:23:22.474372Z	info	waiting for sync...	name=DestinationRules attempt=150 time=14.625350673s
2024-11-23T01:23:27.548418Z	info	waiting for sync...	name=DestinationRules attempt=200 time=19.699403634s
2024-11-23T01:23:32.636446Z	info	waiting for sync...	name=DestinationRules attempt=250 time=24.787431053s
2024-11-23T01:23:37.733167Z	info	waiting for sync...	name=DestinationRules attempt=300 time=29.884149264s
2024-11-23T01:23:38.046401Z	info	klog	pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.DestinationRule: the server could not find the requested resource (get destinationrules.networking.istio.io)
2024-11-23T01:23:38.046526Z	error	watch error in cluster : failed to list *v1.DestinationRule: the server could not find the requested resource (get destinationrules.networking.istio.io)
2024-11-23T01:23:42.802034Z	info	waiting for sync...	name=DestinationRules attempt=350 time=34.953374682s
2024-11-23T01:23:47.896576Z	info	waiting for sync...	name=DestinationRules attempt=400 time=40.047937643s
2024-11-23T01:23:52.983262Z	info	waiting for sync...	name=DestinationRules attempt=450 time=45.134623603s
2024-11-23T01:23:58.063068Z	info	waiting for sync...	name=DestinationRules attempt=500 time=50.214431064s

Notably, for Isito 1.21.0 the destinationrules CRD is installed with versions v1alpha3,v1beta1 and for 1.22.0 it is installed with versions v1,v1alpha3,v1beta1

[sheidkamp]

Here is a spot where we seem to pull in the v1 version