No metrics are scraped

[avwsolutions]

Busy with a spike using promitor. My goal is to setup discovery of the vms I have created and use the scraper to gather the metrics i have defined. I'm using minikube cluster and a connection to Azure subscription.

I have everything configured as it should be, like mentioned in the documentation (walkthrough, etc). Also on prometheus I have created a scrape job, which is ok.

The only thing is that I don't see any metrics on the metrics endpoint coming in/page is empty. I also looked at the logs of the pods, but these seem ok (config validation was ok). The only thing I see is a warning in the scrape logs. Below the config, log details.

I'm looking at a way to troubleshoot the issue with a debug mode. It seems maybe the connection to azure is still not ok?
Would be great if somebody can help me with this.

logs scrape

[09:49:50 INF] Booting up Promitor v2.0.0 - Thank you for using Promitor!
[09:49:50 INF] Using configuration folder '/config/'
[09:49:51 INF] Metrics declaration is using spec version v1
[09:49:51 INF] Scheduled scraping job azure_vm_percentage_cpu-vms-a0de604f-e13a-4b05-99f1-2a0f93d58c73 for resource collection vms which will be reported as metric azure_vm_percentage_cpu
[09:49:51 INF] Scheduled scraping job azure_vm_network_in-vms-ef763017-de57-4df5-b67d-35afee4a37e0 for resource collection vms which will be reported as metric azure_vm_network_in
[09:49:51 INF] Scheduled scraping job azure_vm_network_out-vms-f9c6a311-d69b-4958-af79-efc26bc73b9d for resource collection vms which will be reported as metric azure_vm_network_out
[09:49:51 INF] Scheduled scraping job azure_vm_outbound_flows-vms-34a4a3c5-4e08-4971-8e12-3919eedaaa51 for resource collection vms which will be reported as metric azure_vm_outbound_flows
[09:49:51 INF] Starting validation of Promitor setup
[09:49:51 INF] Start Validation step 1/7: Metrics Declaration Path
[09:49:51 INF] Scrape configuration found at '/config/metrics-declaration.yaml'
[09:49:51 INF] Validation step 1/7 succeeded
[09:49:51 INF] Start Validation step 2/7: Azure Authentication
[09:49:51 INF] Validation step 2/7 succeeded
[09:49:51 INF] Start Validation step 3/7: Metrics Declaration
[09:49:51 INF] Metrics declaration is using spec version v1
[09:49:51 INF] Following metrics configuration was configured:
azureMetadata:
  resourceGroupName: promitor-spike
  subscriptionId: xxx
  tenantId: yyy
  cloud:
    name: AzureGlobalCloud
    authenticationEndpoint: https://login.microsoftonline.com/
    resourceManagerEndpoint: https://management.azure.com/
    graphEndpoint: https://graph.windows.net/
    managementEndpoint: https://management.core.windows.net/
    storageEndpointSuffix: core.windows.net
    keyVaultSuffix: vault.azure.net
metricDefaults:
  aggregation:
    interval: 00:01:00
  scraping: &o0
    schedule: '* * * * *'
metrics:
- azureMetricConfiguration:
    metricName: Percentage CPU
    aggregation:
      type: Average
      interval: 00:01:00
  prometheusMetricDefinition:
    name: azure_vm_percentage_cpu
    description: Average percentage cpu usage on an Azure virtual machine
    labels: {}
  scraping: *o0
  resourceType: VirtualMachine
  resourceDiscoveryGroups:
  - name: vms
  resources: []
- azureMetricConfiguration:
    metricName: Network In Total
    aggregation:
      type: Average
      interval: 00:01:00
  prometheusMetricDefinition:
    name: azure_vm_network_in
    description: The number of bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic)
    labels: {}
  scraping: *o0
  resourceType: VirtualMachine
  resourceDiscoveryGroups:
  - name: vms
  resources: []
- azureMetricConfiguration:
    metricName: Network Out Total
    aggregation:
      type: Average
      interval: 00:01:00
  prometheusMetricDefinition:
    name: azure_vm_network_out
    description: The number of bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic)
    labels: {}
  scraping: *o0
  resourceType: VirtualMachine
  resourceDiscoveryGroups:
  - name: vms
  resources: []
- azureMetricConfiguration:
    metricName: Outbound Flows
    aggregation:
      type: Average
      interval: 00:01:00
  prometheusMetricDefinition:
    name: azure_vm_outbound_flows
    description: Outbound Flows are number of current flows in the outbound direction (traffic going out of the VM)
    labels: {}
  scraping: *o0
  resourceType: VirtualMachine
  resourceDiscoveryGroups:
  - name: vms
  resources: []

[09:49:51 INF] Validation step 3/7 succeeded
[09:49:51 INF] Start Validation step 4/7: Resource Discovery
[09:49:51 INF] Validation step 4/7 succeeded
[09:49:51 INF] Start Validation step 5/7: StatsD Metric Sink
[09:49:51 INF] Validation step 5/7 succeeded
[09:49:51 INF] Start Validation step 6/7: Prometheus Scraping Endpoint Metric Sink
[09:49:51 INF] Validation step 6/7 succeeded
[09:49:51 INF] Start Validation step 7/7: Atlassian Statuspage Metric Sink
[09:49:51 INF] Validation step 7/7 succeeded
[09:49:51 INF] Promitor configuration is valid, we are good to go.
[09:49:51 INF] [Job][azure_vm_percentage_cpu-vms-a0de604f-e13a-4b05-99f1-2a0f93d58c73] is running under time zone: UTC
[09:49:51 INF] [Job][azure_vm_network_in-vms-ef763017-de57-4df5-b67d-35afee4a37e0] is running under time zone: UTC
[09:49:51 INF] [Job][azure_vm_network_out-vms-f9c6a311-d69b-4958-af79-efc26bc73b9d] is running under time zone: UTC
[09:49:51 INF] [Job][azure_vm_outbound_flows-vms-34a4a3c5-4e08-4971-8e12-3919eedaaa51] is running under time zone: UTC
[09:49:51 INF] Scraping resource collection vms - 01/16/2021 09:49:51 +00:00
[09:49:51 INF] Adding Prometheus sink to expose on /metrics
[09:49:51 INF] Start processing HTTP request GET http://promitor-agent-resource-discovery:88/api/v1/resources/groups/vms/discover
[09:49:51 INF] Sending HTTP request GET http://promitor-agent-resource-discovery:88/api/v1/resources/groups/vms/discover
[09:49:51 INF] Scraping resource collection vms - 01/16/2021 09:49:51 +00:00
[09:49:51 INF] Start processing HTTP request GET http://promitor-agent-resource-discovery:88/api/v1/resources/groups/vms/discover
[09:49:51 INF] Sending HTTP request GET http://promitor-agent-resource-discovery:88/api/v1/resources/groups/vms/discover

[09:51:31 WRN] HTTP Dependency promitor-agent-resource-discovery for GET /api/v1/resources/groups/vms/discover completed with 500 in 00:01:40.0288796 at 01/16/2021 09:49:51 +00:00 (Successful: False - Context: {})
[09:51:31 FTL] Failed to scrape resource collection vms: The operation was canceled.
System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendRequestToApiAsync(HttpRequestMessage request) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 72
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendGetRequestAsync(String uriPath) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 58
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.GetAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 40
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryRepository.GetResourceDiscoveryGroupAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryRepository.cs:line 23
   at Promitor.Agents.Scraper.Scheduling.ResourceDiscoveryGroupScrapingJob.ExecuteAsync(CancellationToken cancellationToken) in /src/Promitor.Agents.Scraper/ResourceDiscoveryGroupScrapingJob.cs:line 80
[09:51:31 WRN] HTTP Dependency promitor-agent-resource-discovery for GET /api/v1/resources/groups/vms/discover completed with 500 in 00:01:40.0031245 at 01/16/2021 09:49:51 +00:00 (Successful: False - Context: {})
[09:51:31 WRN] HTTP Dependency promitor-agent-resource-discovery for GET /api/v1/resources/groups/vms/discover completed with 500 in 00:01:40.0043776 at 01/16/2021 09:49:51 +00:00 (Successful: False - Context: {})
[09:51:31 WRN] HTTP Dependency promitor-agent-resource-discovery for GET /api/v1/resources/groups/vms/discover completed with 500 in 00:01:40.0021420 at 01/16/2021 09:49:51 +00:00 (Successful: False - Context: {})
[09:51:31 FTL] Failed to scrape resource collection vms: The operation was canceled.
System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendRequestToApiAsync(HttpRequestMessage request) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 72
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendGetRequestAsync(String uriPath) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 58
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.GetAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 40
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryRepository.GetResourceDiscoveryGroupAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryRepository.cs:line 23
   at Promitor.Agents.Scraper.Scheduling.ResourceDiscoveryGroupScrapingJob.ExecuteAsync(CancellationToken cancellationToken) in /src/Promitor.Agents.Scraper/ResourceDiscoveryGroupScrapingJob.cs:line 80
[09:51:31 FTL] Failed to scrape resource collection vms: The operation was canceled.
System.Threading.Tasks.TaskCanceledException: The operation was canceled.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendRequestToApiAsync(HttpRequestMessage request) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 72
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendGetRequestAsync(String uriPath) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 58
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.GetAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 40
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryRepository.GetResourceDiscoveryGroupAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryRepository.cs:line 23
   at Promitor.Agents.Scraper.Scheduling.ResourceDiscoveryGroupScrapingJob.ExecuteAsync(CancellationToken cancellationToken) in /src/Promitor.Agents.Scraper/ResourceDiscoveryGroupScrapingJob.cs:line 80

logs discovery

10:03:59 INF] Booting up Promitor v0.1.0 - Thank you for using Promitor!
[10:04:00 INF] Starting validation of Promitor setup
[10:04:00 INF] Start Validation step 1/3: Azure Landscape
[10:04:00 INF] Validation step 1/3 succeeded
[10:04:00 INF] Start Validation step 2/3: Resource Discovery Groups
[10:04:00 INF] Validation step 2/3 succeeded
[10:04:00 INF] Start Validation step 3/3: Azure Authentication
[10:04:00 INF] Validation step 3/3 succeeded
[10:04:00 INF] Promitor configuration is valid, we are good to go.
[10:04:00 INF] Now listening on: http://[::]:88
[10:04:00 INF] Application started. Press Ctrl+C to shut down.
[10:04:00 INF] Hosting environment: Production
[10:04:00 INF] Content root path: /app
[10:04:08 INF] Request starting HTTP/1.1 GET http://172.17.0.21:88/api/v1/health?includeDependencies=false  
[10:04:08 INF] Executing endpoint 'Promitor.Agents.ResourceDiscovery.Controllers.HealthController.Get (Promitor.Agents.ResourceDiscovery)'
[10:04:08 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(Boolean) on controller Promitor.Agents.ResourceDiscovery.Controllers.HealthController (Promitor.Agents.ResourceDiscovery).
[10:04:08 INF] Executing ObjectResult, writing value of type 'Microsoft.Extensions.Diagnostics.HealthChecks.HealthReport'.
[10:04:08 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.HealthController.Get (Promitor.Agents.ResourceDiscovery) in 316.4578ms
[10:04:08 INF] Executed endpoint 'Promitor.Agents.ResourceDiscovery.Controllers.HealthController.Get (Promitor.Agents.ResourceDiscovery)'
[10:04:09 WRN] HTTP Request GET http://172.17.0.21:88//api/v1/health completed with 200 in 00:00:00.6779794 at 01/16/2021 10:04:09 +00:00 - (Context: {"Connection": ["close"], "Accept-Encoding": ["gzip"], "Host": ["172.17.0.21:88"], "User-Agent": ["kube-probe/1.16"]})
[10:04:09 INF] Request finished in 951.4473ms 200 application/json; charset=utf-8
[10:04:12 INF] Request starting HTTP/1.1 GET http://172.17.0.21:88/api/v1/health?includeDependencies=false  
[10:04:12 INF] Executing endpoint 'Promitor.Agents.ResourceDiscovery.Controllers.HealthController.Get (Promitor.Agents.ResourceDiscovery)'
[10:04:12 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(Boolean) on controller Promitor.Agents.ResourceDiscovery.Controllers.HealthController (Promitor.Agents.ResourceDiscovery).
[10:04:13 INF] Executing ObjectResult, writing value of type 'Microsoft.Extensions.Diagnostics.HealthChecks.HealthReport'.
[10:04:13 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.HealthController.Get (Promitor.Agents.ResourceDiscovery) in 2.7259ms
[10:04:13 INF] Executed endpoint 'Promitor.Agents.ResourceDiscovery.Controllers.HealthController.Get (Promitor.Agents.ResourceDiscovery)'
[10:04:13 WRN] HTTP Request GET http://172.17.0.21:88//api/v1/health completed with 200 in 00:00:00.0155513 at 01/16/2021 10:04:13 +00:00 - (Context: {"Connection": ["close"], "Accept-Encoding": ["gzip"], "Host": ["172.17.0.21:88"], "User-Agent": ["kube-probe/1.16"]})
[10:04:13 INF] Request finished in 17.4084ms 200 application/json; charset=utf-8

http://172.17.0.19:88/metrics | UP | container="promitor-agent-scraper" endpoint="http" instance="172.17.0.19:88" job="promitor-agent-scraper" namespace="default" pod="promitor-agent-scraper-56966dd548-7tnmq" service="promitor-agent-scraper"
-- | -- | --

I have added a service principal where I have created a VM to gather metrics from.

az ad sp create-for-rbac \
  --role="Monitoring Reader" \
  --scopes="/subscriptions/<guid-subscription-id>/resourceGroups/promitor-spike"

I have both the pods up and running

promitor-agent-resource-discovery-5c64d44c6d-hz4lm         1/1     Running   0          3m19s
promitor-agent-scraper-56966dd548-7tnmq                    1/1     Running   0          17m

Used the following config with Helm3

resource discovery values.yaml

azureAuthentication:
  appId: myid
  appKey: mykey
azureLandscape:
  cloud: Global
  tenantId: mytenant
  subscriptionIds:
  - mysub
resourceDiscoveryGroups:
- name: service-bus-landscape
  type: ServiceBusQueue
- name: api-gateways
  type: ApiManagement
- name: app-plan-landscape
  type: AppPlan
- name: container-instances
  type: ContainerInstance
- name: container-registry-landscape
  type: ContainerRegistry
- name: cosmos-accounts
  type: CosmosDb
- name: dps
  type: DeviceProvisioningService
- name: event-hubs-landscape
  type: EventHubs
- name: vms
  type: VirtualMachine
service:
  loadBalancer:
    dnsPrefix: promitor-resource-discovery-sample
    enabled: true
telemetry:
  defaultLogLevel: information

scraper values.yaml

azureAuthentication:
  appId: myid
  appKey: mykey
azureMetadata:
  tenantId: mytenant
  subscriptionId: mysub
  resourceGroupName: promitor-spike

resourceDiscovery:
  enabled: true 
  host: promitor-agent-resource-discovery 
  port: 88

metricDefaults:
  aggregation:
    interval: 00:01:00
  scraping:
    # Every minute
    schedule: "* * * * *"
metrics:
  - name: azure_vm_percentage_cpu
    description: "Average percentage cpu usage on an Azure virtual machine"
    resourceType: VirtualMachine
    azureMetricConfiguration:
      metricName: Percentage CPU
      aggregation:
        type: Average
    resourceDiscoveryGroups:
    - name: vms
  - name: azure_vm_network_in
    description: "The number of bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic)"
    resourceType: VirtualMachine
    azureMetricConfiguration:
      metricName: Network In Total
      aggregation:
        type: Average
    resourceDiscoveryGroups:
    - name: vms
  - name: azure_vm_network_out
    description: "The number of bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic)"
    resourceType: VirtualMachine
    azureMetricConfiguration:
      metricName: Network Out Total
      aggregation:
        type: Average
    resourceDiscoveryGroups:
    - name: vms
  - name: azure_vm_outbound_flows
    description: "Outbound Flows are number of current flows in the outbound direction (traffic going out of the VM)"
    resourceType: VirtualMachine
    azureMetricConfiguration:
      metricName: Outbound Flows
      aggregation:
        type: Average
    resourceDiscoveryGroups:
    - name: vms

[tomkerkhove]

Hm odd. Can you share your raw configmaps & services please? I think the service name might be wrong but want to verify that the generated configs are correct.

Thansk for trying Promitor btw and sorry for the hick-up!

[avwsolutions]

Morning Tom, I have found the issues which has caused it wasn't working the correct way. Rather a silly one, but I wasn't that sure about the API connection. After enabling trace logging I came to the conclusion it was something else. After some netstat (available in the containers luckily) I saw some SYNC packages. I would expect that the scraper in this situation will log something like resourceDiscovery not accessible / available. But yes, works now.

This was the configuration

resourceDiscovery:
  enabled: true 
  host: promitor-agent-resource-discovery 
  port: 88     < == must be 8889 since the svc is listening here

problem here was that the svc resource was listening on port 8889 rather than 88. Did some testing with the /api/v1/health API. So after using that port data points where coming in.

I think I was a little bit distracted by the documentation mentioning port 88, but this is the containerport which you don't want to use, since in theory we can have multiple pods I have used the service that was created.

Is port 8889 a know port ? This service was created. Just curious if this is something I can expect. Are the Helm charts somewhere in the repo?

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: promitor-agent-resource-discovery
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2021-01-15T21:11:52Z"
  labels:
    app.kubernetes.io/instance: promitor-agent-resource-discovery
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: promitor-agent-resource-discovery
    app.kubernetes.io/version: 0.1.0
    helm.sh/chart: promitor-agent-resource-discovery-0.1.0
  name: promitor-agent-resource-discovery
  namespace: default
  resourceVersion: "365362"
  selfLink: /api/v1/namespaces/default/services/promitor-agent-resource-discovery
  uid: secrets
spec:
  clusterIP: 10.99.90.200
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: 31370
    port: 8889
    protocol: TCP
    targetPort: 88
  selector:
    app.kubernetes.io/instance: promitor-agent-resource-discovery
    app.kubernetes.io/name: promitor-agent-resource-discovery
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

[avwsolutions]

I will now continue testing the capability how I can further improve it for our use-case. I also found that sometimes a metric value is set to NaN, especially when I want to collect data per 1m instead of 5m. Think this is the Azure API limitation. But I like the way how it is setup in an easy way.

Great work b.t.w. Will share my learnings with my colleagues at Fullstaq

[tomkerkhove]

problem here was that the svc resource was listening on port 8889 rather than 88. Did some testing with the /api/v1/health API. So after using that port data points where coming in.

I think I was a little bit distracted by the documentation mentioning port 88, but this is the containerport which you don't want to use, since in theory we can have multiple pods I have used the service that was created.

Is port 8889 a know port ?

Good point. The runtime docs tend to focus on general port without forwarding while our Helm chart uses 8889 as default port.

You can get more information in our chart README: https://artifacthub.io/packages/helm/promitor/promitor-agent-resource-discovery#configuration

Are the Helm charts somewhere in the repo?

You can find our Helm charts on http://github.com/promitor/charts

I will now continue testing the capability how I can further improve it for our use-case. I also found that sometimes a metric value is set to NaN, especially when I want to collect data per 1m instead of 5m. Think this is the Azure API limitation. But I like the way how it is setup in an easy way.

This is an Azure Monitor limitation indeed, but you can configure the value that is being returned as per our docs.

Would it be useful if this is listed in our FAQ?

Great work b.t.w. Will share my learnings with my colleagues at Fullstaq

Cool stuff, thanks! All feedback is welcome to improve Promitor.

Feel free to let me know if you're using it in production and I'm allowed to list Fullstaq as a user.