Telemetry data recording, visualization and analysis

[amotl]

Dear @jpmens, @ckrey, @linusg and all other OwnTracks fans,

through [1], I found out just the other day that more data can be added to the OwnTracks telemetry payload than just location information.

Now, I am thinking about whether it would make sense to work on a capability for users to record their data arriving on the MQTT bus into an additional storage besides that one maintained by ot-recorder in order to visualize that metrics data easily within Grafana and make the system more interoperable in general.

I was just enjoying a conversation with @dexterbg from Open Vehicle Monitoring System (OVMS) about the same idea within daq-tools/kotori#21, so I wanted to also ask here as well while giving an impulse to maybe connecting both projects in a sensible way.

Likewise to the integration of Kotori/InfluxDB/Grafana with OVMS, I would try to put some time into the respective integration with OwnTracks if this actually resonates with you.

Thanks for listening and with kind regards,
Andreas.

[1] https://github.com/owntracks/ocli#payload

[jpmens]

Generally speaking, OwnTracks JSON can be enriched as long as the values added by the devices (mobile apps and hardware) isn't overwritten.

The question is how would people add such data, particularly with the mobile apps?

Be that as it may, the "storage" backend in OwnTracks can do with a huge load of improvements. I originally wrote it thusly to keep things as lightweight as possible -- we didn't want a SQL database.

I'm sure an optional integration with one of the modern graphing tools would be in order, particularly if it can be integrated with @linusg OwnTracks Frontend. :-)

[linusg]

the "storage" backend in OwnTracks can do with a huge load of improvements. I originally wrote it thusly to keep things as lightweight as possible -- we didn't want a SQL database.

I'm curious - is a SQL DB still off the table, and if so, of what nature would these improvements be?

I'm sure an optional integration with one of the modern graphing tools would be in order, particularly if it can be integrated with @linusg OwnTracks Frontend. :-)

I'm not sure I understand, could you give a more specific example?

---

Of the tools mentioned in this issue so far I know Grafana (a little bit), and if the goal is to get OwnTracks data into Grafana for plotting that could probably be done using a custom data source plugin that either talks to the OwnTracks API or reads and parses the storage files from disk.

On that note, I've been doing a few experiments with Chart.js to plot arbitrary values from the payloads over time: altitude, battery, location accuracy come to mind, and in theory any custom values as well. I'm a bit hesistant about building this feature into the frontend though as I definitely don't want to bloat it. Another issue I had is properly integrating it into the existing UI (basically a header + map, with some modals - where would that graph go?)

Another issue for another day probably.

---

Now, I am thinking about whether it would make sense to work on a capability for users to record their data arriving on the MQTT bus into an additional storage besides that one maintained by ot-recorder in order to visualize that metrics data easily within Grafana and make the system more interoperable in general.

Correct me if I'm wrong but isn't that possible already as multiple programs can connect and subscribe to the same MQTT broker? One being the recorder, and then another custom (e.g. Python) script to optionally pre-process and then write the incoming data into an SQL database. Which can then be read from Grafana and other tools with SQL data source support.

Just an idea though, I'm running the recorder without MQTT in HTTP only mode anyway. There, I said it 😛

[jpmens]

is a SQL DB still off the table, and if so, of what nature would these improvements be?

Not necessarily, although I still think we should remain as lightweight as possible; LMDB comes to mind in terms of performance. For SQL maybe SQLite? I've not given it much thought.

isn't that possible already as multiple programs can connect and subscribe to the same MQTT broker?

Definitely. That's the point.

though, I'm running the recorder without MQTT in HTTP only mode anyway

You'll be the first to get the invoice. I think ckrey was right: we should start charging for HTTP mode. :-)

[amotl]

Correct me if I'm wrong but isn't that possible already as multiple programs can connect and subscribe to the same MQTT broker?

To second @jpmens here: Absolutely. I was more asking for appropriate integration, i.e. making everything ready to work out-of-the-box for platform operators (you?) as well as users running their own infrastructure.

One being the recorder, and then another custom (e.g. Python) script to optionally pre-process and then write the incoming data into an SQL database. Which can then be read from Grafana and other tools with SQL data source support.

This is exactly what Kotori does. It's a bit more than just a custom script but can be installed easily using packages for Debian (other distributions to be added).

I've been doing a few experiments with Chart.js to plot arbitrary values from the payloads over time: altitude, battery, location accuracy come to mind, and in theory any custom values as well.

That's the spirit! If Grafana integration would be appropriate, this could do lengths of more stuff for us than manually programming Chart.js and friends.

I'm a bit hesistant about building this feature into the frontend though as I definitely don't want to bloat it. Another issue I had is properly integrating it into the existing UI (basically a header + map, with some modals - where would that graph go?)

I see your point. Just wanted to bring it up as an idea. I believe a good drill-down way of integrating things would be most appropriate. You might want to have a look at openSenseMap as an example [1].

[1] https://opensensemap.org/explore/5ab9347e850005001b8ea314

[jannickfahlbusch]

If Grafana integration would be appropriate, this could do lengths of more stuff for us than manually programming Chart.js and friends.

I just build a Grafana datasource for Owntracks!

You can find the code here: jannickfahlbusch/grafana-datasource-owntracks (No documentation so far, sorry).

[jpmens]

@jannickfahlbusch thanks for the pointer. As soon as you've time to add some documentation and a nice screenshot or two to your repository, I'll gladly link to it from the Recorder docs. :-)