BM25 ANN Search

[hieudx149]

Hi there,
I have a question about how can use BM25 as a vector searcher (vsearch), because i want sparse search faster, in your guideline about Encode documents with Sparse encoder, you provide a way to encode documents with Sparse encoder, so can we do this with BM25 to obtain jsonl file as the image below:

Then, we have jsonl file, we can do BM25 ANN Search following the steps as Dense Search or we need to more additional steps

[MXueguang]

Hi @r1ckC139 ,
I guess this feature is what you are looking for?
https://github.com/castorini/pyserini/blob/master/docs/usage-indexreader.md#how-do-i-quantize-vectors-of-weights

[hieudx149]

@MXueguang, thanks for quick response
so when i have this document format how can i use it to index and search?

[MXueguang]

it can be indexed as hnsw index by using https://github.com/castorini/pyserini/blob/master/pyserini/index/nmslib.py
Then conduct the search with https://github.com/castorini/pyserini/blob/master/pyserini/search/nmslib/__main__.py