Scrapper for Azure Microsoft.Cdn/profiles available?

[andreibogdan]

Hi,

Is developed any scrapper Microsoft.Cdn/profiles?

[tomkerkhove]

CDN is not a 1st party scraper yet but you can already use our generic scraper to get started!

I've opened #1664 to track it though. If you want to contribute it based on our guide, feel free to let me know.

[andreibogdan]

yes, the setup is:

- name: azure_profiles_latency
  description: "CDN Latency"
  resourceType: Generic
  azureMetricConfiguration:
    metricName: TotalLatency
    aggregation:
      type: Average
  resources:
  - resourceUri: Microsoft.Cdn/profiles/foobarprofile

[tomkerkhove]

Is that the exact structure or did you remove the spaces here?

Just making sure the YAML is valid 😅

[andreibogdan]

:) te structure is without space. It was a copy/paste issue.

[tomkerkhove]

It looks like I can get it up and running with the following configuration:

version: v1
azureMetadata:
  tenantId: c8819874-9e56-4e3f-b1a8-1c0325138f27
  subscriptionId: 0f9d7fea-99e8-4768-8672-06a28514f77e
  resourceGroupName: promitor-sources
metricDefaults:
  aggregation:
    interval: 00:05:00
  limit: 10
  labels:
    geo: china
    environment: dev
  scraping:
    # Every minute
    schedule: "0 * * ? * *"
metrics:
- name: azure_profiles_latency
  description: "CDN Latency"
  resourceType: Generic
  azureMetricConfiguration:
    metricName: TotalLatency
    aggregation:
      type: Average
  resources:
  - resourceUri: Microsoft.Cdn/profiles/promitor-test

Does your full config look similar?

It doesn't output any metrics, though, but that's because I need to generate some traffic on the endpoint first. This is reflected in the logs:

[06:04:02 WRN] No metric information found for metric TotalLatency with dimension null. Details: No time series was found
[06:04:02 INF] Found value null for metric azure_profiles_latency with aggregation interval 00:05:00

[andreibogdan]

I used the same config as yours with my credentials and I received following error:

07:35:35 INF] Starting validation of Promitor setup
[07:35:35 INF] Start Validation step 1/7: Metrics Declaration Path
[07:35:35 INF] Scrape configuration found at '/config/metrics-declaration.yaml'
[07:35:35 INF] Validation step 1/7 succeeded
[07:35:35 INF] Start Validation step 2/7: Azure Authentication
[07:35:35 INF] Validation step 2/7 succeeded
[07:35:35 INF] Start Validation step 3/7: Metrics Declaration
[07:35:35 INF] Metrics declaration is using spec version v1
[07:35:36 WRN] The following problems were found with the metric configuration:
Warning 9:3: Unknown field 'limit'.
Warning 10:3: Unknown field 'labels'.
[07:35:36 INF] Following metrics configuration was configured:
azureMetadata:
  resourceGroupName: xxxxxxxxxxx
  subscriptionId: xxxxxxxxxxxxx
  tenantId: xxxxxxxxxxxxxxxxx
  cloud:
    name: AzureGlobalCloud
    authenticationEndpoint: https://login.microsoftonline.com/
    resourceManagerEndpoint: https://management.azure.com/
    graphEndpoint: https://graph.windows.net/
    managementEndpoint: https://management.core.windows.net/
    storageEndpointSuffix: core.windows.net
    keyVaultSuffix: vault.azure.net
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping: &o0
    schedule: 0 * * ? * *
metrics:
- azureMetricConfiguration:
    metricName: TotalLatency
    aggregation:
      type: Average
      interval: 00:05:00
  prometheusMetricDefinition:
    name: azure_profiles_latency
    description: CDN Latency
    labels: {}
  scraping: *o0
  resourceType: Generic
  resourceDiscoveryGroups: []
  resources:
  - resourceUri: Microsoft.Cdn/profiles/xxxxxxx
    resourceType: Generic
    resourceName: Microsoft.Cdn/profiles/xxxxxxx
    uniqueName: Microsoft.Cdn/profiles/xxxxxxxx

[07:35:36 INF] Validation step 3/7 succeeded
[07:35:36 INF] Start Validation step 4/7: Resource Discovery
[07:35:36 INF] Metrics declaration is using spec version v1
[07:35:36 INF] Validation step 4/7 succeeded
[07:35:36 INF] Start Validation step 5/7: StatsD Metric Sink
[07:35:36 INF] Validation step 5/7 succeeded
[07:35:36 INF] Start Validation step 6/7: Prometheus Scraping Endpoint Metric Sink
[07:35:36 INF] Validation step 6/7 succeeded
[07:35:36 INF] Start Validation step 7/7: Atlassian Statuspage Metric Sink
[07:35:36 INF] Validation step 7/7 succeeded
[07:35:36 INF] Promitor configuration is valid, we are good to go.
[07:35:36 INF] [Job][5e638988-5200-43d6-bb0e-6dc4ccb000a9-xxxxxx-azure_profiles_latency-Microsoft.Cdn/profiles/xxxxxxxxxx-5103e405-e0e9-4a91-b2da-8f87ac89abc3] is running under time zone: UTC
[07:35:36 INF] Scraping Azure Monitor - 07/02/2021 07:35:36 +00:00
[07:35:36 INF] Scraping azure_profiles_latency for resource type Generic
[07:35:36 INF] Now listening on: http://[::]:8080
[07:35:36 INF] Application started. Press Ctrl+C to shut down.
[07:35:36 INF] Hosting environment: Production
[07:35:36 INF] Content root path: /app
[07:35:38 FTL] Failed to scrape resource for metric 'azure_profiles_latency'
System.ArgumentException: [details] cannot be Null, empty or white-space. (Parameter 'details')
   at GuardNet.Guard.For[TException](Func`1 predicate, TException exception)
   at GuardNet.Guard.NotNullOrWhitespace[TException](String param, TException exception)
   at GuardNet.Guard.NotNullOrWhitespace(String param, String paramName, String message)
   at GuardNet.Guard.NotNullOrWhitespace(String param, String paramName)
   at Promitor.Integrations.AzureMonitor.Exceptions.MetricInformationNotFoundException..ctor(String name, String details, String dimension) in /src/Promitor.Integrations.AzureMonitor/MetricInformationNotFoundException.cs:line 33
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.QueryMetricAsync(String metricName, String metricDimension, AggregationType aggregationType, TimeSpan aggregationInterval, String resourceId, String metricFilter) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 85
   at Promitor.Core.Scraping.AzureMonitorScraper`1.ScrapeResourceAsync(String subscriptionId, ScrapeDefinition`1 scrapeDefinition, TResourceDefinition resourceDefinition, AggregationType aggregationType, TimeSpan aggregationInterval) in /src/Promitor.Core.Scraping/AzureMonitorScraper.cs:line 51
   at Promitor.Core.Scraping.Scraper`1.ScrapeAsync(ScrapeDefinition`1 scrapeDefinition) in /src/Promitor.Core.Scraping/Scraper.cs:line 69

The test that I am doing is via local docker setup and the CLI command looks like:

docker run -p 8999:80 --name promitor-agent-scraper \
    --env PROMITOR_AUTH_APPID='xxxxx-xxxxx-xxxxxxxxxxx' \
    --env-file ${PWD}/azure-mon-auth.creds \
    --volume ${PWD}/metrics-declaration.yaml:/config/metrics-declaration.yaml \
    --volume ${PWD}/runtime.yaml:/config/runtime.yaml \
    ghcr.io/tomkerkhove/promitor-agent-scraper:2.0.0-rc

and I see on the logs these warnings:

Warning 9:3: Unknown field 'limit'.
Warning 10:3: Unknown field 'labels'.

[andreibogdan]

I found the problem. I used promitor-agent-scraper:2.3.0 and everything is working ok until now. :)

[tomkerkhove]

Great to hear! Sorry, I didn't saw your most recent reply yet with 2.0.0-rc.

[tomkerkhove]

If you want to be a listed user, feel free to let me know.

[andreibogdan]

Yeah I want to be a listed user. Thank you!

[tomkerkhove]

I see you work for Adobe so I presume I can use their logo then?

[andreibogdan]

Yes, you can their logo.