Several questions before running in prod

[icanhazbeer]

Hello,
I have been testing your project the last few days and I have a few questions I was hoping to get clarified. I am heading towards running this in a prod environment and want to make sure my bases are covered.
Please let me know if I have missed something or if something does not make sense.

I am using resource discovery / scraper agent for azure resources.

• is it possible to have more than one metrics-declaration.yaml? this would be useful to have a dedicated file per azure resource
• I want to get each dimension of a multi-dimensional metric. the example here makes no mention of dimensions. is it safe to assume that this example block will pull all dimensions or do they need to be configured explicitly?
• are there any recommendations/best practices for running this in prod?
• if the scraper instance was scaled from 1 to 2, would that double the amount of calls to the azure metrics api? would the metrics be split between the instances or would that need to be configured? is that possible?
• is there a way to declare resourcediscoverygroups in a default block or is it required for each metrics block?

[tomkerkhove]

Hi, great to hear and let me know when you hit PROD - Always happy to list end-users, if I'm allowed to!

In terms of your questions:

is it possible to have more than one metrics-declaration.yaml? this would be useful to have a dedicated file per azure resource

For now only one file is supported, but feel free to open an issue to start tracking it and can see what the demand is on it (for prio reasons). Another route I've been considering is to introduce an operator so that 1 metric = 1 resource in the cluster. Thoughts on this?

I want to get each dimension of a multi-dimensional metric. the example here makes no mention of dimensions. is it safe to assume that this example block will pull all dimensions or do they need to be configured explicitly?

When no dimension is configured, it will not use any at all. If you want to scrape a metric for a given dimension you have to configure it. Note that using multiple dimensions is not supported (and not even in Azure Monitor as well AFAIK).

If you want to have this, you'll need to use multiple metrics, but be careful as this will increase your load on Azure Monitor and you will get throttled a lot faster. I typically recommend to only scrape what you need to avoid hitting limitations.

are there any recommendations/best practices for running this in prod?

Nothing in particular but happy to hear feedback on this. I'm considering adding topology spread constraints & pod disruption budget but not sure what you are mainly looking for?

if the scraper instance was scaled from 1 to 2, would that double the amount of calls to the azure metrics api? would the metrics be split between the instances or would that need to be configured? is that possible?

If you run multiple instances, you'll multiple the amount of calls since every agent will scrape all metrics. I'm planning on improving this as per #1450 though.

is there a way to declare resourcediscoverygroups in a default block or is it required for each metrics block?

Actually no, but that's a good idea, feel free to open an issue for that

[icanhazbeer]

Thanks for replying.

If I can prove that we need this to serious purpose in prod then I think we can make the case for a listing.

For now only one file is supported, but feel free to open an issue to start tracking it and can see what the demand is on it (for prio reasons). Another route I've been considering is to introduce an operator so that 1 metric = 1 resource in the cluster. Thoughts on this?

I think an operator would be a good way to go. How do you mean 1 metric = 1 resource ?
The problem I think is that, while resources may be discovered, we need to actively maintain the config for what we want to scrape. This could be a problem as there may be many eyes on this and having 1 large file that will grow over time will get messy.

Thanks for clarifying about the dimensions. We will need to figure out what we need from the metrics. Any advice on how to manage many updates to the declaration file as it grows?

Just a thought but would there be a way to configure multiple security principal for scraping? For example I could set up a specific principal dedicated to scraping a specific resource?

Spread constraints & pod disruption budget

This would be a good idea I think but I really dont know until I can put a live test together.
I am mainly concerned that as more resources are added, discovered, and eventually scraped, the performance of the deployment/agents may degrade. My thought was to scale the deployment across the cluster, and it sounds like #1450 is the feature I am looking for.
Additionally, might need to consider the possibility to go across subscriptions but this might be a separate deployment.

[tomkerkhove]

Thanks for replying.

If I can prove that we need this to serious purpose in prod then I think we can make the case for a listing.

For now only one file is supported, but feel free to open an issue to start tracking it and can see what the demand is on it (for prio reasons). Another route I've been considering is to introduce an operator so that 1 metric = 1 resource in the cluster. Thoughts on this?

I think an operator would be a good way to go. How do you mean 1 metric = 1 resource ?

The problem I think is that, while resources may be discovered, we need to actively maintain the config for what we want to scrape. This could be a problem as there may be many eyes on this and having 1 large file that will grow over time will get messy.

Imagine that this is your current configuration:

version: v1
# ... omitted ...
metrics:
- name: promitor_demo_appplan_autoscale_instances_current_discovered
  description: "Average amount of current instances for an Azure App Plan with Azure Monitor Autoscale"
  resourceType: MonitorAutoscale
  labels:
    app: promitor
  azureMetricConfiguration:
    metricName: ObservedCapacity
    aggregation:
      type: Average
  resourceDiscoveryGroups:
  - name: autoscaling-rules
- name: promitor_demo_appplan_autoscale_observed_capacity
  description: "Average amount of current instances for an Azure App Plan with Azure Monitor Autoscale"
  resourceType: MonitorAutoscale
  labels:
    app: promitor
  azureMetricConfiguration:
    metricName: MetricThreshold
    dimension:
      name: MetricTriggerRule
    aggregation:
      type: Average
  resources:
  - autoscaleSettingsName: app-service-autoscaling-rules
    resourceGroupName: demo

It would be replaced by a promitor-crd/promitor_demo_appplan_autoscale_instances_current_discovered and promitor-crd/promitor_demo_appplan_autoscale_observed_capacity and no longer through a raw config file

Thanks for clarifying about the dimensions. We will need to figure out what we need from the metrics. Any advice on how to manage many updates to the declaration file as it grows?

Unfortunately the only approach, today, is to restart the agent - Sorry. But I know this is not ideal.

Just a thought but would there be a way to configure multiple security principal for scraping? For example I could set up a specific principal dedicated to scraping a specific resource?

Running multiple deployments with different identities is the only option, sorry.

Spread constraints & pod disruption budget

This would be a good idea I think but I really dont know until I can put a live test together.
I am mainly concerned that as more resources are added, discovered, and eventually scraped, the performance of the deployment/agents may degrade. My thought was to scale the deployment across the cluster, and it sounds like #1450 is the feature I am looking for.
Additionally, might need to consider the possibility to go across subscriptions but this might be a separate deployment.

We recommend doing dedicated deployments as a subscription is typically a security boundary, but, if you use resource discovery you can actually target multiple subscriptions.

[icanhazbeer]

Thanks for clarifying. I like the idea of the operator and I think that is the way to go. Anything to help make the metric declaration mapping easier as well.
I think I can work with different deployments if it gets to that.

Can you clarify a bit on the default aggregation interval? I found the explanation a bit vague. Does this overwrite any default aggregation from the metrics?

metricDefaults.aggregation.interval - The default interval which defines over what period measurements of a metric should be aggregated. a cron that fits your needs.

[tomkerkhove]

That's the default aggregation for all metrics, unless you specify a different one on the metric level.

[icanhazbeer]

Thanks!
I think I have all I need for now.

[tomkerkhove]

Good to hear, feel free to ask more questions if need be or mail me on [email protected]