perf: fix for cluster aggregation performance

[ssg2526]

The changes include performance improvements for cluster mode aggregations flow. The changes are 3 fold -

• The hashing of each metric is moved to workers. Now each worker hashes the data and adds that in a key named hash in the metrics json.
  • This is distributing the hashing task among workers rather than master doing the CPU task
• The Map which was being built on every request freshly, is made global and only if a new metric comes then that will be put in the map else we will get the value and push the item in the array.
  • The insertion of large keys is very slow in Map as it is maintaining insertion order and this will minimize the insertions
• A major choking point was when master asks workers to provide metrics and workers sending those metrics over IPC causing the choking in IPC which was hindering the request routing from master to workers. The change here is that workers don't send metrics over IPC rather they are writing them in a file and sending master the file name over IPC. Master is then reading the metrics from the files and deleting the files.
  • This is removing the congestion on IPC and requests routing from master to workers not getting hampered.

[ssg2526]

@SimenB @siimon kindly approve for workflows. I have added the prettier fixes because of which the earlier workflows failed.

[zbjornson]

Thanks for working on this!

Do you have any benchmarks you can show? (both the code and the results)

[zbjornson]

I'm really hesitant to involve files, since that suddenly involves the page cache, file system, more syscalls and possibly other surprises like filesystem permissions, running out of disk space and cleaning up. How much of an improvement did this yield? Did you benchmark before or after Node.js v18.6.0, which has this optimization? How much disk I/O load did your system have when benchmarking? I think IPC is "supposed" to be faster than involving the filesystem.

[ssg2526]

The benchmarks that I have performed are on our own application. The screenshots of the results are shared in the
this Issue Link. The benchmarks were done on a 16 core machine and approximately 250-300 metrics (total size ~500 Kbs). At P99 and above levels we are getting ~10x improvement in app performance. The only disk I/Os we have is logging but that is also not happening on /tmp mounted drive on which we are writing our metrics.
The major choking point was with IPC and creating the map and hashing the object. The detailed bifurcation that I have done on my local machine which is 8 core machine is given In the screenshot below. and the code used is in below given zip file. The zip file contains the node modules as well because I have added some logs in prom-client to get the time data of each parts. Here in screenshot if we see the worst IPC time is always higher. I had run multiple iterations and same/similar results are found. The total aggregation time which is 72ms in this screenshot out of which 68ms is taken for hashing and building the map. I haven't tested with node v18.6.0 yet. will test the same code with that as well and share the results in the same thread
cluster_test.zip

[ssg2526]

In these benchmarks (issue link) the scraping interval used was every 5 seconds and the throughput on the app was about 1100-1200 RPS.

[ssg2526]

Hi @zbjornson

• I also tried the code in the zip file with node v18.6.0. I am getting the similar results without any improvements in the logged timings.
• I think we shouldn't use IPC for this communication because that can create a problem whenever the sizes of metrics go beyond a certain point and that will start blocking the requests routing at high throughputs.
• We can ask user to provide permissions for /tmp folder. From prom-client performance POV i didn't see much of a difference with files but again I am only hitting it once every 5 seconds.
• If not files then we should think of other ways of communication to solve this completely like workers calling over tcp on master port

[ssg2526]

Hi @zbjornson @SimenB we went live in production with these changes about 3 months ago and everything is working fine for us. We are not seeing any additional latencies post this change. We serve more than 100,000 RPS at peak for this service. If required I can share the production results as well.
I understand that introducing file system can pose different challenges for the library, But we can explore communication with HTTP or some other mode of communication (Not IPC) when we have larger size of the metrics as the current solution will cause very high tail latencies (>P99) at high throughput and larger metrics sizes.
Requesting your inputs on this, In terms of how we can take this forward as we don't want to diverge from the library maintaining our own custom version.

[mrnonz]

Any concern in filesystem anymore? Should we consider to release it?

[ssg2526]

Any concern in filesystem anymore? Should we consider to release it?

@mrnonz we haven't faced any issues till now related to filesystem. This change would require the users to provide the access to /tmp path for prom-client