Build a Retrieval-Augmented Generation Pipeline Over Your Own Documents
Ground an open-weight model in your own documents so it answers from what you actually wrote, not from what it guesses, without fine-tuning anything.
Download workflow config (JSON)How the data moves
This is a handoff plan, not an automatic integration. Review outputs before the next step. Unlabeled arrows show order, not a verified data format. Use the capability links to explore tool substitutions.
Step 1
Run a Local Open-Weight Model
Install Ollama and pull an open-weight model that fits your hardware. This becomes the model that actually generates answers, running entirely on your own machine rather than a hosted API.
Step 2
Chunk and Embed Your Documents
Use LlamaIndex to split your source documents into overlapping chunks and generate embeddings for each one. Chunk size is the single most consequential setting here: too large and irrelevant text rides along with the answer, too small and the model loses surrounding context.
Step 3
Store the Vectors in Chroma
Load the embedded chunks into a local Chroma vector database. Chroma's job is narrow and specific: given a query, return the chunks whose embeddings are closest to it, fast.
Step 4
Wire Up the Retrieval-Then-Generation Chain
Connect LlamaIndex's query engine so that every incoming question first retrieves the top-matching chunks from Chroma, then passes them to your local Ollama model as context before it writes an answer. The model never answers from memory alone, it answers from what was actually retrieved.
Step 5
Evaluate Retrieval Quality, Not Just Answer Quality
Run a sample of real questions through Ragas to score whether the retrieved chunks were actually relevant and whether the generated answer stays grounded in them. A pipeline can produce a confident, fluent answer built on the wrong retrieved chunk, and that failure mode is invisible unless you're scoring retrieval specifically.
Tools
Time Required
3-4 hours, one-time setup
Expected Result
A working pipeline that retrieves the most relevant chunks of your own documents for a given question and hands them to a local model as context, instead of relying on the model's training data or a costly fine-tune.
Related Workflows
Build a Multi-Agent Research Pipeline with CrewAI
Set up a CrewAI pipeline where specialized agents handle different research tasks in parallel, one searches papers, one synthesizes findings, one checks contradictions, delivering a comprehensive brief automatically.
View workflowBuild a Production LLM Evaluation and Monitoring Pipeline
Set up systematic evaluation for your LLM application so you know when output quality changes, with automated tests that run on every prompt change and production monitoring that catches regressions.
View workflowBuild an Image Generation API with Replicate
Add AI image generation to your application in an afternoon using Replicate's API, no GPU infrastructure required, pay only per generation.
View workflow