please collaborate for performance and accuracy for Question Answer system using Flan T5

[pruthviishere]

Hello,
I am newbie in Gen AI, I tried localGPT implementation for flan t5 model specifically for QA .But there are some performance issues.
This is a question answering system specifically designed for low/Mid-end devices. This system has the capability to ingest documents, allowing users to extract information from various sources. I would be grateful if you could take a look at the repository and help improve performance of model.

repo link https://github.com/pruthviishere/QA_FlanT5_Ingestion

The system is designed to run locally and offline, without the need for any Hugging Face API token. Additionally, it utilizes the Flan-T5-large model, which is a zero-shot model known for its efficiency and faster response times compared to other language models.

Thank you for your understanding and interest in my project.

[nekiee13]

I assume that by performance issues you mean speed, because as far as I can see your project runs on CPU only. So, if I'm not wrong on that assumption, by implementing GPU inference, you should get significantly improved performance.

[pruthviishere]

Thanks for response.
Yes, will do GPU inference. I am wondering is there anything other like chunk size, max length parameters or other configuration for low end devices.