Resource Unavailable (HTTP 503) on Metrics Scrape #1214

djsunn · 2020-08-12T14:40:27Z

djsunn
Aug 12, 2020

I'm trying to set up Promitor for a proof-of-concept, but it's failing at with a HTTP/503, resource temporarily unavailable.

The service principal used is heavily over-equipped on the permissions side with the Contributor role.
The same Tenant+Subscription+Service Principal configuration is working fine via azure_metrics_exporter and via the AzureMonitor REST API.

The error seems to occur regardless of the resource type used.
I've tested the following resource types so far:

Generic
VirtualMachine
NetworkInterface

I've also tested multiple tags of the Promitor Linux container image, with the same result:

1.6.0
1.6.1
2.0.0-preview-2-linux

I assume there is a configuration error on my end.
Can anybody help?

Here is the runtime.yaml:

server:
  httpPort: 80 # Optional. Default: 80
metricSinks:
  prometheusScrapingEndpoint:
    metricUnavailableValue: NaN # Optional. Default: NaN
    enableMetricTimestamps: false # Optional. Default: true
    baseUriPath: /metrics # Optional. Default: /metrics
metricsConfiguration:
  absolutePath: /config/metrics-declaration.yaml # Optional. Default: /config/metrics-declaration.yaml
telemetry:
  applicationInsights:
    instrumentationKey: ABC # Optional. Note: Required to be specified when turned on
    isEnabled: false # Optional. Default: false
    verbosity: trace # Optional. Default: N/A
  containerLogs:
    isEnabled: true # Optional. Default: true
    verbosity: error # Optional. Default: N/A
  defaultVerbosity: error # Optional. Default: error
azureMonitor:
  logging:
    informationalLevel: BodyAndHeaders
    isEnabled: true
resourceDiscovery:
  #host:
  #port:

and the metrics-declaration.yaml:

---
version: v1
azureMetadata:
  tenantId: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx"
  subscriptionId: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx"
  resourceGroupName: djsunn-dp200-1
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping:
    # Every minute
    schedule: "0 * * ? * *"
metrics:
  - name: azure_network_interface_bytes_received_rate_discovery
    description: "Number of bytes the Network Interface sent"
    resourceType: NetworkInterface
    azureMetricConfiguration:
      metricName: BytesReceivedRate
      aggregation:
        type: Average
    resources:
      - networkInterfaceName: test-djsunn913

and the full log incl. stack trace:

██████╗ ██████╗  ██████╗ ███╗   ███╗██╗████████╗ ██████╗ ██████╗ 
██╔══██╗██╔══██╗██╔═══██╗████╗ ████║██║╚══██╔══╝██╔═══██╗██╔══██╗
██████╔╝██████╔╝██║   ██║██╔████╔██║██║   ██║   ██║   ██║██████╔╝
██╔═══╝ ██╔══██╗██║   ██║██║╚██╔╝██║██║   ██║   ██║   ██║██╔══██╗
██║     ██║  ██║╚██████╔╝██║ ╚═╝ ██║██║   ██║   ╚██████╔╝██║  ██║
╚═╝     ╚═╝  ╚═╝ ╚═════╝ ╚═╝     ╚═╝╚═╝   ╚═╝    ╚═════╝ ╚═╝  ╚═╝
[12:52:59 INF] Metrics declaration is using spec version v1
[12:52:59 INF] Scheduled scraping job xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-djsunn-dp200-1-azure_network_interface_bytes_received_rate_discovery-test-djsunn913-xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx for resource test-djsunn913 which will be reported as metric azure_network_interface_bytes_received_rate_discovery
[12:52:59 INF] Starting validation of Promitor setup
[12:52:59 INF] Start Validation step 1/7: Metrics Declaration Path
[12:52:59 INF] Scrape configuration found at '/config/metrics-declaration.yaml'
[12:52:59 INF] Validation step 1/7 succeeded
[12:52:59 INF] Start Validation step 2/7: Azure Authentication
[12:52:59 INF] Validation step 2/7 succeeded
[12:52:59 INF] Start Validation step 3/7: Metrics Declaration
[12:52:59 INF] Metrics declaration is using spec version v1
[12:52:59 INF] Following metrics configuration was configured:
azureMetadata:
  resourceGroupName: djsunn-dp200-1
  subscriptionId: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  tenantId: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  cloud:
    name: AzureGlobalCloud
    authenticationEndpoint: https://login.microsoftonline.com/
    resourceManagerEndpoint: https://management.azure.com/
    graphEndpoint: https://graph.windows.net/
    managementEndpoint: https://management.core.windows.net/
    storageEndpointSuffix: core.windows.net
    keyVaultSuffix: vault.azure.net
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping: &o0
    schedule: 0 * * ? * *
metrics:
- azureMetricConfiguration:
    metricName: BytesReceivedRate
    aggregation:
      type: Average
      interval: 00:05:00
  prometheusMetricDefinition:
    name: azure_network_interface_bytes_received_rate_discovery
    description: Number of bytes the Network Interface sent
    labels: {}
  scraping: *o0
  resourceType: NetworkInterface
  resourceDiscoveryGroups: []
  resources:
  - networkInterfaceName: test-djsunn913
    resourceType: NetworkInterface
    resourceGroupName: djsunn-dp200-1
    resourceName: test-djsunn913
    uniqueName: test-djsunn913

[12:52:59 INF] Validation step 3/7 succeeded
[12:52:59 INF] Start Validation step 4/7: Resource Discovery
[12:52:59 INF] Metrics declaration is using spec version v1
[12:52:59 INF] Validation step 4/7 succeeded
[12:52:59 INF] Start Validation step 5/7: StatsD Metric Sink
[12:52:59 INF] Validation step 5/7 succeeded
[12:52:59 INF] Start Validation step 6/7: Prometheus Scraping Endpoint Metric Sink
[12:52:59 INF] Validation step 6/7 succeeded
[12:52:59 INF] Start Validation step 7/7: Atlassian Statuspage Metric Sink
[12:52:59 INF] Validation step 7/7 succeeded
[12:52:59 INF] Promitor configuration is valid, we are good to go.
[12:52:59 INF] [Job][xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-djsunn-dp200-1-azure_network_interface_bytes_received_rate_discovery-test-djsunn913-xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] is running under time zone: Etc/UTC
[12:52:59 INF] Scraping Azure Monitor - 08/12/2020 12:52:59 +00:00
[12:52:59 INF] Scraping azure_network_interface_bytes_received_rate_discovery for resource type NetworkInterface
[12:53:00 INF] Now listening on: http://[::]:80
[12:53:00 INF] Application started. Press Ctrl+C to shut down.
[12:53:00 INF] Hosting environment: Production
[12:53:00 INF] Content root path: /app
[12:53:01 INF] Scraping Azure Monitor - 08/12/2020 12:53:01 +00:00
[12:53:01 INF] Scraping azure_network_interface_bytes_received_rate_discovery for resource type NetworkInterface
[12:53:05 FTL] Failed to scrape resource for metric 'azure_network_interface_bytes_received_rate_discovery'
System.Net.Http.HttpRequestException: Resource temporarily unavailable
 ---> System.Net.Sockets.SocketException (11): Resource temporarily unavailable
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.HttpClientWrapper.GetResponseAsync()
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.AdalHttpClient.GetResponseAsync[T](Boolean respondToDeviceAuthChallenge)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.AdalHttpClient.GetResponseAsync[T]()
   at Microsoft.IdentityModel.Clients.ActiveDirectory.InstanceDiscovery.DiscoverAsync(Uri authority, Boolean validateAuthority, RequestContext requestContext)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.InstanceDiscovery.GetMetadataEntryAsync(Uri authority, Boolean validateAuthority, RequestContext requestContext)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Instance.Authenticator.UpdateFromTemplateAsync(RequestContext requestContext)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.PreRunAsync()
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.RunAsync()
   at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenForClientCommonAsync(String resource, ClientKey clientKey)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenAsync(String resource, ClientCredential clientCredential)
   at Microsoft.Rest.Azure.Authentication.Internal.MemoryApplicationAuthenticationProvider.AuthenticateAsync(String clientId, String audience, AuthenticationContext context)
   at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, String clientId, IApplicationAuthenticationProvider authenticationProvider, ActiveDirectoryServiceSettings settings, TokenCache cache)
   at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, ClientCredential credential, ActiveDirectoryServiceSettings settings, TokenCache cache)
   at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, String clientId, String secret, ActiveDirectoryServiceSettings settings, TokenCache cache)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.AzureCredentials.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperations.ListWithHttpMessagesAsync(String resourceUri, String metricnamespace, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperationsExtensions.ListAsync(IMetricDefinitionsOperations operations, String resourceUri, String metricnamespace, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.ListByResourceAsync(String resourceId, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.Microsoft.Azure.Management.Monitor.Fluent.IMetricDefinitions.ListByResourceAsync(String resourceId, CancellationToken cancellationToken)
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.QueryMetricAsync(String metricName, String metricDimension, AggregationType aggregationType, TimeSpan aggregationInterval, String resourceId, String metricFilter) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 72
   at Promitor.Core.Scraping.AzureMonitorScraper`1.ScrapeResourceAsync(String subscriptionId, ScrapeDefinition`1 scrapeDefinition, TResourceDefinition resourceDefinition, AggregationType aggregationType, TimeSpan aggregationInterval) in /src/Promitor.Core.Scraping/AzureMonitorScraper.cs:line 46
   at Promitor.Core.Scraping.Scraper`1.ScrapeAsync(ScrapeDefinition`1 scrapeDefinition) in /src/Promitor.Core.Scraping/Scraper.cs:line 69

Thanks for your time and best regards!

Answered by tomkerkhove

Aug 13, 2020

So you're saying you get a 503 when calling /metrics? Hm that's odd. Looks like it's bubbling the 503 from Azure AD 🤔

Do you still have it now or was it temporary?

View full answer

tomkerkhove · 2020-08-13T06:04:08Z

tomkerkhove
Aug 13, 2020
Maintainer

So you're saying you get a 503 when calling /metrics? Hm that's odd. Looks like it's bubbling the 503 from Azure AD 🤔

Do you still have it now or was it temporary?

4 replies

djsunn Aug 13, 2020
Author

Calling /metrics works fine.
It returns a HTTP 200 with an empty body, since the metrics can't be scraped due to the "Resource temporarily unavailable" issue.

Is there any further information I could provide?
Are there any settings on service principal or tenant level which could cause this?

tomkerkhove Aug 13, 2020
Maintainer

Oh that makes more sense then.

Did you try it with another resource (type) as well at a later point? I'm asking because it looks like a temporary issue on Microsofts side as we are just calling their API and they give the 503 to Promitor.

The SP should just have (Monitor) Reader so you should be good to go there.

djsunn Aug 13, 2020
Author

I didn't really find the cause yet, but it seems to be some configuration on my linux box which is causing this behavior.
On my Windows box everything just works.
Doesn't really make much sense to me, but it is what it is.

Thanks for taking time to look into my issue.

tomkerkhove Aug 13, 2020
Maintainer

No worries at all and sorry for the lack of help.

Feel free to let me know if you have found the issue on your Linux machine!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resource Unavailable (HTTP 503) on Metrics Scrape #1214

{{title}}

Replies: 1 comment 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Resource Unavailable (HTTP 503) on Metrics Scrape #1214

djsunn Aug 12, 2020

Replies: 1 comment · 4 replies

tomkerkhove Aug 13, 2020 Maintainer

djsunn Aug 13, 2020 Author

tomkerkhove Aug 13, 2020 Maintainer

djsunn Aug 13, 2020 Author

tomkerkhove Aug 13, 2020 Maintainer

djsunn
Aug 12, 2020

Replies: 1 comment 4 replies

tomkerkhove
Aug 13, 2020
Maintainer

djsunn Aug 13, 2020
Author

tomkerkhove Aug 13, 2020
Maintainer

djsunn Aug 13, 2020
Author

tomkerkhove Aug 13, 2020
Maintainer