RAGAS with Jithin James, Shahul Es, and Erika Cardenas - Weaviate Podcast #77!

Weaviate Podcast

Hey everyone, thank you so much for watching the 77th Weaviate Podcast on RAGAS, featuring Jithin James, Shahul ES, and Erika Cardenas! RAGAS is one of the hottest rising startups in Retrieval-Augmented Generation! RAGAS began it's journey with the RAGAS score, a matrix of evaluations for generation and retrieval. Generation evaluated on Faithfulness (is the response grounded in the context) as well as Relevancy (is the response useful). Retrieval is then evaluated on Precision (How many of the search results are relevant to the question?) and Recall (How many of the relevant search results are captured in the retrieved results?). Now, the super novel thing about this is that an LLM is used to determine these metrics. So we circumvent painstaking manual labeling effort with the RAGAS score! This podcast dives into the development of the RAGAS score as well as how RAG application builders should think about the knobs to tune for optimizing their RAGAS score: embedding models, chunking strategies, hybrid search tuning, rerankers, ... ?!? We also discussed tons of exciting directions for the future such as fine-tuning smaller LLMs for these metrics, agents that use tuning APIs, and long context RAG! Check out the docs here for getting started with RAGAS! https://docs.ragas.io/en/latest/getstarted/index.html#get-started Chapters 0:00 Welcome Jithin and Shahul! 0:44 Welcome Erika! 0:56 RAGAS, Founding Story 2:38 Weaviate + RAGAS integration plans 4:44 RAG Knobs to Tune 25:50 RAG Experiment Tracking 34:52 LangSmith and RAGAS 38:55 LLM Evaluation 40:25 RAGAS Agents 44:00 Long Context RAG Evaluation