Retrieving by BM25 becomes slower when there are many queries

[namespace-Pt]

I tried BM25 baseline for MSMARCO passage ranking and succeeded. The retrieving speed is about 0.001s/query according to the terminal output when hits=1000.

But when I was going to retrieve more queries (all the 55k training queries) with the exact same index. I found the speed was becoming slower and slower until the entire program stuck at 49.13% queries. Why this could happen? I don't think it's reasonable that the retrieving speed is dragging down by larger query quantities.

[lintool]

You're probably running out of memory. Since the SearchCollection implementation is multi-threaded, it keeps the hits in memory until all the queries are processed, and the writes out to disk all at once. This simplifies thread synchronization.

Try running on smaller batches of queries.

[namespace-Pt]

You mean that I split the large query file into several small files and run them one by one? Or I can specify some attributes like batch_size directly from command line?

[lintool]

Yes, both. Split into files of, say, 10K queries.

Control number of threads to use with -parallelism - set to 8 by default:
https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/search/SearchArgs.java#L43

This is the number of threads in the thread pool; each query gets its own thread, so there isn't a batch size.

[namespace-Pt]

Thank you.