Hubble Metrics show only small subset of the requests.

[shahargl]

TLDR - although I see all the flows in the hubble UI, metrics are not being created for these flows, only for a small subset of them.

Hey,

I'm using Google's boutique demo app to demonstrate the power of Cilium + Hubble.

• https://github.com/GoogleCloudPlatform/microservices-demo

I've enabled the metrics for httpv2 and all other protocols:

helm get values cilium -n kube-system
USER-SUPPLIED VALUES:
hubble:
  enabled: true
  metrics:
    enableOpenMetrics: true
    enabled:
    - dns
    - drop
    - tcp
    - flow
    - port-distribution
    - icmp
    - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
  relay:
    enabled: true
  ui:
    enabled: true
operator:
  prometheus:
    enabled: true
prometheus:
  enabled: true

and the L7 policy:

kubectl get cnp l7-visibility -o yaml

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cilium.io/v2","kind":"CiliumNetworkPolicy","metadata":{"annotations":{},"name":"l7-visibility","namespace":"default"},"spec":{"egress":[{"toPorts":[{"ports":[{"port":"6379","protocol":"TCP"},{"port":"5050","protocol":"TCP"}]}]},{"toPorts":[{"ports":[{"port":"53","protocol":"ANY"}],"rules":{"dns":[{"matchPattern":"*"}]}}]},{"toEndpoints":[{"matchLabels":{"k8s:io.kubernetes.pod.namespace":"default"}}],"toPorts":[{"ports":[{"port":"80","protocol":"TCP"},{"port":"8080","protocol":"TCP"},{"port":"3550","protocol":"TCP"},{"port":"9555","protocol":"TCP"},{"port":"7070","protocol":"TCP"},{"port":"7000","protocol":"TCP"},{"port":"5000","protocol":"TCP"},{"port":"50051","protocol":"TCP"}],"rules":{"http":[{}]}}]}],"endpointSelector":{"matchLabels":{"k8s:io.kubernetes.pod.namespace":"default"}}}}
  creationTimestamp: "2024-10-07T13:41:22Z"
  generation: 2
  name: l7-visibility
  namespace: default
  resourceVersion: "234977"
  uid: aa07dfe2-654d-4f81-bddc-35fa3e16eb43
spec:
  egress:
  - toPorts:
    - ports:
      - port: "6379"
        protocol: TCP
      - port: "5050"
        protocol: TCP
  - toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: '*'
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: default
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      - port: "8080"
        protocol: TCP
      - port: "3550"
        protocol: TCP
      - port: "9555"
        protocol: TCP
      - port: "7070"
        protocol: TCP
      - port: "7000"
        protocol: TCP
      - port: "5000"
        protocol: TCP
      - port: "50051"
        protocol: TCP
      rules:
        http:
        - {}
  endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: default
status:
  conditions:
  - lastTransitionTime: "2024-10-07T13:44:19Z"
    message: Policy validation succeeded
    status: "True"
    type: Valid

and everything looks fine from hubble UI:

But when I go to Prometheus, I see very small subset of these flows as metrics:

You can see that for example this flow shows as a L7 flow, but doesn't have any metrics representing it:

Flow Details
Timestamp
2024-10-28T09:04:27.535Z
Verdict
forwarded
Traffic direction
egress
Cilium event type
to-endpoint
Source pod
frontend-c7d978c8d-2zlns
Source identity
15377
Source labels
app=frontend
io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
io.cilium.k8s.policy.cluster=default
io.cilium.k8s.policy.serviceaccount=frontend
namespace=default
skaffold.dev/run-id=4ae81965-065c-4bbf-935b-85af86fab248
Source IP
10.24.0.44
Destination pod
productcatalogservice-6b48f7df8f-4pd4d
Destination identity
35104
Destination labels
app=productcatalogservice
io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default
io.cilium.k8s.policy.cluster=default
io.cilium.k8s.policy.serviceaccount=productcatalogservice
namespace=default
skaffold.dev/run-id=4ae81965-065c-4bbf-935b-85af86fab248
Destination IP
10.24.8.185
Destination port • protocol
3550 • TCP