Indexing via python code

[vrdn-23]

Is there a function or method that is exposed in which we can index sparse and dense indexes via python code as opposed to terminal command lines?
I'm trying to run the ingestion process of pyserini in a docker container and from the documentation it looks like the only way to call the indexing process is through the command line since the code for encoding and indexing seem to be wrapped in the __ main __ calls of the classes
I'd be happy to raise a PR if it's something that the team would be interested in!

[lintool]

hi @vrdn-23 I think this is the feature you're looking for? #1344

[vrdn-23]

Hey @lintool,

This wasn't quite the solution that I was looking for, although I do have some questions about that:

• Would this method allow us to add a document to an already existing index?
• Does such a solution exist for the dense indexing?

What I was hoping for was something along the line where I can invoke a method as such:

from pyserini.index.lucene import create_sparse_index
create_sparse_index(collection="JsonCollection", input=... index=... ....)

In the current implementation of Pyserini, the indexing part is covered in the main call, which we can probably refactor using a library called fire (with minimal code overhead).

I'd be willing to work on a PR that can implement this for the sparse and dense indexing as I feel it would be nice for developers and researchers to have the option to consume and ingest indexes via python code and not just terminal cli commands.

[lintool]

Is there a function or method that is exposed in which we can index sparse and dense indexes via python code as opposed to terminal command lines?
I'm trying to run the ingestion process of pyserini in a docker container and from the documentation it looks like the only way to call the indexing process is through the command line since the code for encoding and indexing seem to be wrapped in the __ main __ calls of the classes
I'd be happy to raise a PR if it's something that the team would be interested in!

I don't exactly understand what you mean by this, then... are you talking about something like moving what's in __main__ into a separate function, and then just having main call that function?

re: Fire

This looks like dependency injection? Spring gives me nightmares in Java-land.

My initial response... over-engineering?

But I'm amenable to being convinced otherwise... try me with a PR?

Would this method allow us to add a document to an already existing index?

Potentially, if we exposed the right hooks from the Java end.

Does such a solution exist for the dense indexing?

Nope, but nothing to prevent us from doing it. Dense is easier than sparse because we can remain in Python-land with Faiss. This is probably worth filing an issue.