Azure Monitor Metrics

[wadhahbenaissa]

hello.
I was wondering what would be the configuration (metrics-decalaration.yaml) that would allow me to get all the metrics of a specific azure resource? for example get all the metrics of all the VMs existing in 1 or multi azure subscriptions?
thanks

[tomkerkhove]

You'll have to combine the upcoming resource discovery in v2.0 (but you can give it a try already with https://github.com/tomkerkhove/promitor/releases/tag/2.0.0-preview-3) and listing all the metrics in your metrics-decalaration.yaml that uses that resource discovery group.

We do not provide a metrics "wildcard" that scrapes all metrics for a given Azure resource, for now. Reason for that is throttling done by Azure, if we do that than you'll hit them (too) easily.

What we recommend is going to https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported, list the metrics you need and include them in your metrics declaration.

Hope this clarifies things?

[tomkerkhove]

Would you mind sharing the Scraper logs please?

[tomkerkhove]

Can you access the metrics directly on the Scraper?

[wadhahbenaissa]

yap....after running the command
kubectl port-forward service/my_service 8080:8888 -n my_namespace
and after accessing http://localhost:8080/metrics, I can see the metrics:

# HELP azure_virtual_machine_percentage_cpu Average pe rcentage cpu usage on an Azure virtual machine
# TYPE azure_virtual_machine_percentage_cpu gauge
azure_virtual_machine_percentage_cpu{resource_group="my_resource_group",subscription_id="my_sub_id",resource_uri="vm1",instance_name="vm1"} 4.303303571428572
azure_virtual_machine_percentage_cpu{resource_group="my_resource_group",subscription_id="my_sub_id",resource_uri="vm2",instance_name="vm2"} 10.973365384615384
azure_virtual_machine_percentage_cpu{resource_group="my_resource_group",subscription_id="mny_sub_id",resource_uri="vm3",instance_name="vm3"} 10.158214285714285
# HELP promitor_ratelimit_arm Indication how many calls are still available before Azure Resource Manager is going to throttle us.
# TYPE promitor_ratelimit_arm gauge
promitor_ratelimit_arm{tenant_id="my_ten_id",subscription_id="my_sub_id",app_id="my_app_id"} 11910

these are some of the logs I'm getting:

[16:02:00 INF] Scraping azure_virtual_machine_percentage_cpu for resource type VirtualMachine
[16:02:00 INF] Found value 3.329230769230769 for metric azure_virtual_machine_percentage_cpu with aggregation interval 00:15:00
[16:02:00 INF] Metrics declaration is using spec version v1
[16:02:00 INF] Found value 7.258269230769231 for metric azure_virtual_machine_percentage_cpu with aggregation interval 00:15:00
[16:02:00 INF] Metrics declaration is using spec version v1
[16:02:00 INF] Found value 2.7315384615384612 for metric azure_virtual_machine_percentage_cpu with aggregation interval 00:15:00
[16:02:00 INF] Metrics declaration is using spec version v1

[wadhahbenaissa]

Hey again Tom.
I was wondering whether you have some updates regarding my last response.
Thanks

[tomkerkhove]

It looks like Promitor Scraper is up and running as it should but for some reason the integration is not correct. I don't have experience with ServiceMonitor so can't talk about that.

From what I'm seeing, it could be your service in the Prometheus config, we provide a guide to follow to integrate it - https://promitor.io/walkthrough/scrape-promitor-with-prometheus-on-azure-kubernetes-service