RAG Performance & Fairness Evaluation Toolkit (OpenVINO + LangChain)

[pkhara31]

This toolkit enables developers to build, evaluate, and optimize Retrieval-Augmented Generation (RAG) applications with comprehensive quality metrics including accuracy, bias detection, and perplexity analysis plus a racial-bias indicator. This uses RAG pipeline optimized with Intel OpenVINO for enhanced performance on CPU, GPU, and NPU. The pipeline leverages:

Optimum-Intel’s OVModelForCausalLM with the OpenVINO backend for efficient inference.
LangChain for orchestration of document loading, chunking, embedding, retrieval, reranking, and generation.
Goal: Provide a portable notebook-driven workflow for rapid experimentation, model comparison, and validation of RAG systems on custom/private corpora.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[sbalandi]

Hi @pkhara31 , the notebook https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-rag-langchain/llm-rag-langchain.ipynb seems to cover the same tasks you described in comment. Could you please check ? Does this match your idea of ​​the notebook you wanted to add ?
Let's discuss this point and leave only something new in this notebook or update the existing notebook

[pkhara31]

Hi @sbalandi , This NB covers the methodology to evaluate the performance of RAG pipeline by computing the BERT, BLEU, ROGUE, perplexity, racial bias scores.
This notebook also provides ability to scrap web URLs and implement RAG pipeline on the web content as well.
These are the key deltas & additions to the existing notebook on RAG.
Let me know if any more questions!

[openvino-dev-samples]

Thanks for your contribution @pkhara31 Do you think if we can add this NB to RAG notebook, as a separate .ipynb, so users can directly evaluate their RAG system with selected models in your pipeline ?

[openvino-dev-samples]

Another idea is to change this .ipynb to an evaludation_helper.py which can be called inside current NB, and users can evaluate the RAG pipeline without loading the model again. Whats your thought ? @pkhara31

[pkhara31]

The model loading redundancy can be avoided, but I think the evaluation and the ability to apply RAG pipeline on web docs (scraping) should still remain in the helper.

[pkhara31]

May I know the latest update on this?

[sbalandi]

May I know the latest update on this?

Hi, sorry for the delay, I'm in the process of reviewing notebook, I'll write comments within a week