Handling very large json array

[PSeitz]

I have a quite large 17gb json file, which is an array of json objects [ {a:2} ... {b:2} ]
When I try to deserialize the whole object, my machine with 96GB RAM gets an OOM, which is i a little odd. Memory consumption for serde_json::from_str(&s) seems to be quite high.

I saw there is an StreamDeserializer, although it seems it handles only data in the form of {a:2} {b:2} ...
https://docs.serde.rs/serde_json/de/struct.StreamDeserializer.html

[oli-obk]

Can you post a backtrace with gdb and the type you're deserializing into?

[PSeitz]

I don't know the structure upfront, so serde_json::Value.

Does RUST_BACKTRACE=full produce a gdb backtrace?

[oli-obk]

@PSeitz no, you need to use gdb in the case of oom or overflows.

serde_json::Value is very wasteful, you might be getting multiple times the size you had on disk. Although a factor 5 increase seems excessive.

[dtolnay]

This example code shows one way to process an array of values without having them all in memory at the same time.

[PSeitz]

The program gets killed by a SIGKILL, is it possible to get a stacktrace there? An attached gdb didn't help.

@dtolnay
Thanks, I saw that before, although the example makes some assumptions about the structure(id and values) and it's not clear to me how to remove those for my case.

[PSeitz]

I'll close this in favor of #404