Add Tokenization Read Limit On Retriever:

[clearsitedesigns]

Let's say you have a ton of data on a topic you use for RAG. When you query a request to the retriever, the tokenizer if you have or even don't have history, can get overloaded.

I recommend at the retriever lever when the long chain gets the data from the data store (chroma). If there is a tokenization max trim function or something like that, otherwise you will run into errors with overloading the tokenizer.

Maybe something like this? I'm not sure if its possible I haven't integrated yet.

results = retriever.similarity_search(query=my_query, k=5, search_kwargs={"max_tokens": 1000})