PostgreSQL gRPC storage plugin

[anayrat]

Hello,

Thanks to SQL analytics capabilities, PostgreSQL can be a good candidate as a storage backend. It allows to perform statistics computation, for example, average duration for a given operation, compare it with current day and yesterday...

I didn't start to write code. I just spent some times to study the model and how gRPC works. I first thought to add a storage backend, then I discovered gRPC plugin.

Here is the model I think should be suitable :

(Don't pay attention to types, I temporary used text in many place).

What do you think of this idea? Is there other people interested with it?

[kkaczkowski]

Can you tell when this functionality will be ready?

[anayrat]

Sure, I did not start to work on this (I do this on my free time).

[jpkrohling]

@anayrat I think it's a great idea. More and more concrete storage providers are compatible with PostgreSQL's wire format, so, adding support for this will automatically add support for a bunch of other solutions, like CockroachDB.

Once you have a proof of concept ready, or even the final code, let us know and we can add a link to it from our documentation.

[jozef-slezak]

Hello, we have implemented this plugin using the design above.

Can we create pull request to this repository?

[yurishkuro]

The main idea of grpc storage plugins is that they don’t need to be in this repo, but exist as external extensions. We can simply add a link to external repo from documentation. Can you point to where the plugin is currently implemented?

[anayrat]

@jozef-slezak great! Where is the repo?

[jozef-slezak]

Hello, during this week we are cleaning up the code before open sourcing it. After that we will put link paste here the link to the repo.

[pavolloffay]

Feel free to submit a PR to our docs that are hosted at https://www.jaegertracing.io/docs/1.17/deployment/#storage-plugin.

https://github.com/jaegertracing/documentation/blob/master/content/docs/next-release/deployment.md#storage-plugin

[pavolloffay]

@anayrat I am just curious, are you using the gRPC plugin with all-in-one or with collector and query in separate processes?

[anayrat]

@anayrat I am just curious, are you using the gRPC plugin with all-in-one or with collector and query in separate processes?

Hello, I do not use either one. I just thought how we can add postgres support as storage plugin and send the database model. Unfortunately I do not have time to spend on this and I do not use jaeger anymore. I can only help on postgres side.

[albertobeas20]

Hello, during this week we are cleaning up the code before open sourcing it. After that we will put link paste here the link to the repo.

Hello, it is possible get the code even if this is not cleaning up?? I am working with this integration too and it could be very useful for me to finished and I need to solve it quickly due to it is priority to my architecture. Thank you very much.

[jozef-slezak]

https://github.com/jozef-slezak/jaeger-postgresql

[Reef3rm4n]

any updates on this ?

[maxekman]

It would be awesome to have a working Postgres storage driver.

@jozef-slezak What is the current status of the driver? Do you think it's usable (-ish) in production?

[deem0n]

We just completed the original code by jozef-slezak and made some fixes. We can see Web UI working with spans stored in the PostgreSQL DB. Please check out our fork at https://github.com/luxms/jaeger-postgresql

[anayrat]

I can help you by review database schema, query and indexes.

[deem0n]

Thanks @anayrat!

Probably we can discuss that on https://github.com/luxms/jaeger-postgresql/discussions ?
BTW, we have plans to use TimescaleDB at some point.

[maxekman]

Really cool, I'll hopefully be able to test this in our staging env next week.

[JaneJeon]

Ooh, super cool. Do we know if we use pg>8 features (mainly for RedShift compatibility) in the implementation?

[ankitnayan]

Has anyone benchmarked the performance with the above schema? Will it be able to handle at least 5K spans/s and handle say, 50 concurrent queries that might fire with alert rules every minute.

[yurishkuro]

Two solutions mentioned, closing as done.

• https://github.com/jozef-slezak/jaeger-postgresql
• https://github.com/luxms/jaeger-postgresql

[ringerc]

Following up on my comment about PostgreSQL - I'm curious about how this performs and its storage efficiency with nontrivial data sets.

PostgreSQL has very high per-row overheads, and its indexes are not great at append-oriented data with pseudorandom keys.

The only way I can imagine this being of much use would be alongside one of the columnar-store extensions. Even then it's not clear to me that it'd be superior to other backends.