Monitoring: "No datapoints found"

[smartm0use]

I'm running Prometheus for monitoring electrs but going to http://localhost:9090/graph?g0.range_input=1h&g0.expr=index_height&g0.tab=0 (as suggested here) I get "No datapoints found" error.

What am I doing wrong?

[romanz]

It seems that Prometheus doesn't collect electrs' metrics - see https://github.com/romanz/electrs/blob/d128aef758280c7c38f4d89604c66a7e9f6f2135/doc/monitoring.md

[smartm0use]

It is what I have done (I also reported that bug in the instructions) but I still get that error.
I'm connecting by remote access using <node_onion_address>:4224 instead of localhost:4224.

Could be this the cause of that error?

[romanz]

I don't think... do you see other metrics that prometheus collects by default?
For example: http://localhost:9090/classic/graph?g0.range_input=1h&g0.expr=node_load1&g0.tab=0

[romanz]

My bad, see #831 (reply in thread)

[smartm0use]

It doesn't work, I get the same error.

Here is my configuration at the end of the file located in "/etc/prometheus/prometheus.yml":

scrape_configs:
  - job_name: electrs
    static_configs:
      - targets: ['<address>.onion:4224']

Any way to check if there are errors in establishing connection to electrs? First time I use Prometheus...

[romanz]

Did you restart prometheus (sudo systemctl restart prometheus)?

Could you please share your /etc/prometheus/prometheus.yml

[romanz]

Could you please try using localhost:4224 to see if it works?

[smartm0use]

Sorry, I was wrong, default metrics are working.

Yes, I did restart Prometheus.

Here is the Yml file:

# Sample config for Prometheus.

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'example'

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    # If prometheus-node-exporter is installed, grab stats about the local
    # machine by default.
    static_configs:
      - targets: ['localhost:9100']

  - job_name: electrs
    static_configs:
      - targets: ['<my-address>.onion:4224']

I can't try to use localhost:4224 because electrs in on another machine.

[romanz]

You were right - the '<my-address>.onion:4224' target won't probably work without using a Tor proxy.

[romanz]

https://tor.stackexchange.com/questions/6939/configure-tor-as-proxy#6950

[smartm0use]

I'm already connected under Tor. It is active and running...

[smartm0use]

Do you know any verbose option or setting to see what is the error?

[romanz]

You can set prometheus log level to debug at /etc/default/prometheus:

ARGS="--log.level=debug"

Then, restart prometheus:

$ sudo systemctl restart prometheus
$ journalctl -u prometheus | tail
Jan 28 10:01:34 home prometheus[8410]: ts=2023-01-28T08:01:34.365Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:01:49 home prometheus[8410]: ts=2023-01-28T08:01:49.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:02:04 home prometheus[8410]: ts=2023-01-28T08:02:04.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:02:19 home prometheus[8410]: ts=2023-01-28T08:02:19.365Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:02:34 home prometheus[8410]: ts=2023-01-28T08:02:34.365Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:02:49 home prometheus[8410]: ts=2023-01-28T08:02:49.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:03:04 home prometheus[8410]: ts=2023-01-28T08:03:04.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:03:19 home prometheus[8410]: ts=2023-01-28T08:03:19.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:03:34 home prometheus[8410]: ts=2023-01-28T08:03:34.365Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"
Jan 28 10:03:49 home prometheus[8410]: ts=2023-01-28T08:03:49.364Z caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://localhost:4224/metrics msg="Scrape failed" err="Get \"http://localhost:4224/metrics\": dial tcp 127.0.0.1:4224: connect: connection refused"

[smartm0use]

It was helpfull:

caller=scrape.go:1291 level=debug component="scrape manager" scrape_pool=electrs target=http://<my-address>.onion:4224/metrics msg="Scrape failed" err="Get \"http://<my-address>.onion:4224/metrics\": dial tcp: lookup <my-address>.onion: no such host"

It seems that is adding "http://" prefix to my Onion address...
According to this issue is it a limit of Prometheus?

[romanz]

So it seems... I'm not very familiar with Go, but maybe you can use something similar to https://stackoverflow.com/questions/43682909/connect-to-onion-websites-on-tor-using-python?

[smartm0use]

Sorry, I do not get it. How does this can help me?

[romanz]

I think you should configure Prometheus to use Tor Proxy, maybe via proxy_url:
prometheus/prometheus@7810448

[romanz]

If it doesn't work, maybe you can run Prometheus server on the same machine with electrs?

[smartm0use]

Sorry... how and where do I set proxy_url parameter?
I would avoid to run another server...