Process crashes with: Cannot create a string longer than 0x1fffffe8 characters

[michaelbromley]

What happened?

I attempted to run npx code-pushup and got the following error:

Cannot create a string longer than 0x1fffffe8 characters

What would you expect to happen?

No error, and the command to complete successfully.

What steps did you take?

Set up per the getting started guide, with the following config:

import type { CoreConfig } from '@code-pushup/models';
import eslintPlugin, { eslintConfigFromAllNxProjects} from '@code-pushup/eslint-plugin';

const config: CoreConfig = {
  plugins: [
    await eslintPlugin(await eslintConfigFromAllNxProjects()),
  ],
};

export default config;

Run

npx code-pushup

Code PushUp package version

No response

What operation system are you on?

Windows 11

Node version

20.18.0

"@code-pushup/cli": "^0.53.1",     "@code-pushup/eslint-plugin": "^0.53.1",

Relevant log output

Code PushUp CLI
[ info ] Run collect...

[ warn ] Plugins failed: 
[ warn ] Error: - Plugin ESLint (eslint) produced the following error:
  - node:buffer:834
    return this.utf8Slice(0, this.length);
                ^

Error: Cannot create a string longer than 0x1fffffe8 characters
    at Buffer.toString (node:buffer:834:17)
    at String (<anonymous>)
    at Socket.<anonymous> (file:///<path to project>/node_modules/.pnpm/@[email protected]_@[email protected]/node_modules/@code-pushup/eslint-plugin/bin.js:755:17)
    at Socket.emit (node:events:519:28)
    at addChunk (node:internal/streams/readable:559:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)
    at Readable.push (node:internal/streams/readable:390:5)
    at Pipe.onStreamRead (node:internal/stream_base_commons:191:23) {
  code: 'ERR_STRING_TOO_LONG'
}

[vmasek]

Thanks for the report 🚀 We'll try to reproduce the error in our codebase.

[vmasek]

I think I located the problem to string appending at executeProcess fn and this explanation of the error as system limit of the buffer size in NodeJS (0x1fffffe8 is almost exactly 512MB.)

@michaelbromley Is my assumption you are running it on a significantly large project correct?

[michaelbromley]

Yes, a large multi tenant Nx monorepo.

[vmasek]

It seems like the standard output that we cumulate with intention to print in at once at the end of the task gets too big. I have a Buffer based solution in progress now, I'll ping you when the new version is released to try it out.

[vmasek]

Error itself seems to point to direction of too large JSON being processed from ESLint analysis output. So far we avoided it by splitting projects in big monorepos but since you use eslintConfigFromAllNxProjects in your example, the projects are already auto-split, meaning you probably have one project that outputs data that large.
I've already consulted how we can change the processing from output passing to file/stream reading, so we need to do this bigger change 🏗