Storage Account Queue Metrics

[msftkenneth]

It will be painful to implement SAS token for every queue under Storage Accounts as it's not straightforward at all to provision the resources and then update the Promitor declaration afterwards.

Or is it possible to directly obtain the metric data from the corresponding REST API calls?
It seems the pattern is not usual one as it has one more level when comparing to the usual one.
https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported#microsoftstoragestorageaccountsqueueservices

[tomkerkhove]

Would it help if you could authenticate with a storage account key?

Azure Monitor is unfortunately not an option as they don't give metrics on a per-queue level.

[msftkenneth]

What if handling the general one but not the queue-specific one?

/subscriptions/X/resourceGroups/Y/providers/Microsoft.Storage/storageAccounts/Z/queueServices/default/providers/microsoft.Insights/metrics?timespan=2020-12-10T14:45:00.000Z/2020-12-11T14:45:00.000Z&interval=FULL&metricnames=QueueMessageCount&aggregation=average&metricNamespace=microsoft.storage%2Fstorageaccounts%2Fqueueservices&validatedimensions=false&api-version=2019-07-01%22

[tomkerkhove]

That gives you the total message count for all your queues; not an individual one

[msftkenneth]

Is it possible to be automatically discovered if for the total count one?
Or it can only be configured with generic Azure resources?

[tomkerkhove]

Automatically discover the queues in a storage account you mean or what is your question actually?

All services that support resource discovery can be found on https://promitor.io/configuration/v2.x/resource-discovery

[msftkenneth]

Actually I am seeking whether I can get the metric data of the metric 'Queue Message Count' under 'Storage Account - Queue' as I showed above, with using resource discovery.

I tried combination of the metric declaration with using Storage Account type and that specific metric name however failed to obtain the metrics. So I am bringing this here.

[msftkenneth]

FYI, I found out how to configure at the end for that specific metric, using Generic Declaration with ResourceUri.
Like this : "Microsoft.Storage/storageAccounts/[[storageAccountName]]/queueServices/default"

However it cannot be automatically discovered. Hope it can be in the roadmap :)

[tomkerkhove]

Automatic discovery for queues is not planned I think, but feel free to open an issue.

When using Microsoft.Storage/storageAccounts/[[storageAccountName]]/queueServices/default however, it will not give you the metric that you are expecting like I've mentioned here. It gives you the message count for all your queues combined, not for an individual one.

If that is what you can live with, this would work for you.

[msftkenneth]

I think it would be nice if the figure of all the queues combined as some kind of scaling activities (driven from Prometheus or other alerting systems) may rely on this metric as the mentioned metric format is a static one instead of dynamic one. And I guess it's also useful of this idea to match to other resources like Service Bus namespaces and Event Hub namespaces etc.

Besides, SAS token is somehow a potential security risk as the exchange process of generating and transferring the token value cannot be automated at this stage or requires complicated processes, of course it's for the users like me not concerning about the queue-specific count.

[tomkerkhove]

Besides, SAS token is somehow a potential security risk as the exchange process of generating and transferring the token value cannot be automated at this stage or requires complicated processes, of course it's for the users like me not concerning about the queue-specific count.

I agree, tracking in #1418

I think it would be nice if the figure of all the queues combined as some kind of scaling activities (driven from Prometheus or other alerting systems) may rely on this metric as the mentioned metric format is a static one instead of dynamic one. And I guess it's also useful of this idea to match to other resources like Service Bus namespaces and Event Hub namespaces etc.

You will never scale based on an aggregate of all your queues.

If your storage account has following queues:

• Orders: 100 msg
• Shipments: 10000 msgs

The metric you will receive with the above Azure Monitor query will be 10100, not per queue. If this is really what you want, then we can add it but this is fairly useless if you ask me.

So for you the annoying point with the current scraper is that it enforces the queue name or not? Because then you might be interested in #533 or not?

Dynamic discovery for storage queues is being tracked in #1133 btw.

[msftkenneth]

I think it would be nice if the figure of all the queues combined as some kind of scaling activities (driven from Prometheus or other alerting systems) may rely on this metric as the mentioned metric format is a static one instead of dynamic one. And I guess it's also useful of this idea to match to other resources like Service Bus namespaces and Event Hub namespaces etc.

You will never scale based on an aggregate of all your queues.

If your storage account has following queues:

• Orders: 100 msg
• Shipments: 10000 msgs

The metric you will receive with the above Azure Monitor query will be 10100, not per queue. If this is really what you want, then we can add it but this is fairly useless if you ask me.

That case is conceptually for the scaling actions at the subsequent systems to clear the queue (one single queue within that Storage Account) as an example. So I cannot say whether it is generally useful or not anyway. It's just an idea for that scenario.

So for you the annoying point with the current scraper is that it enforces the queue name or not? Because then you might be interested in #533 or not?

Dynamic discovery for storage queues is being tracked in #1133 btw.

Seems quite similar to the new issue I raised just now. 👍

[tomkerkhove]

So if you could get total message count for the whole account + resource discovery, it would be enough for you. Is that correct?