Building a Custom RAG Chatbot with LangChain

Large Language Models (LLMs) are incredibly powerful, but they suffer from two major limitations: they do not know your proprietary, private data, and their knowledge is frozen at the time they were trained. For enterprise applications, these limitations are dealbreakers. The solution is Retrieval-Augmented Generation (RAG). By combining your custom data with the reasoning capabilities of modern LLMs, you can build powerful, context-aware applications. In this technical deep dive, we will explore how to build a custom RAG chatbot using Python, LangChain, and vector databases.

What is Retrieval-Augmented Generation (RAG)?

RAG is an architectural pattern that bridges the gap between an LLM's general knowledge and your specific, domain-level information. Instead of relying on the AI to "remember" facts from its training data—which often leads to hallucinations—a RAG system operates like an open-book exam.

When a user asks a question, the system first queries a database (specifically a vector database) to retrieve the most relevant documents related to the query. It then packages those retrieved documents alongside the original question and sends them to the LLM. The LLM is instructed to answer the question strictly using the provided context, resulting in highly accurate, factual, and traceable answers.

The RAG Architecture Overview

Before we look at the code, it is important to understand the different components that make up a successful RAG pipeline. A typical system involves data ingestion, embedding, vector storage, retrieval, and generation.

Python Code Tutorial: Building the Pipeline

We will use LangChain, the industry-standard orchestration framework for building LLM applications, alongside ChromaDB for our local vector store. This example demonstrates how to load a text document, vectorize it, and create a question-answering chain.

First, ensure you have the necessary libraries installed via your terminal:

pip install langchain langchain-openai chromadb tiktoken

Now, let's write the Python script to execute the RAG workflow.

import os
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# Set up API Key
os.environ["OPENAI_API_KEY"] = "your-api-key-here"

def build_rag_system(file_path):
    # 1. Load the document
    print("Loading data...")
    loader = TextLoader(file_path)
    docs = loader.load()

    # 2. Split the document into manageable chunks
    print("Splitting text...")
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        length_function=len
    )
    splits = text_splitter.split_documents(docs)

    # 3. Create embeddings and store in Chroma Vector DB
    print("Creating vector store...")
    vectorstore = Chroma.from_documents(
        documents=splits, 
        embedding=OpenAIEmbeddings()
    )

    # 4. Set up the retriever
    retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

    # 5. Define the LLM and the Prompt Template
    llm = ChatOpenAI(model_name="gpt-4o", temperature=0)
    
    template = """You are a helpful assistant for question-answering tasks. 
    Use the following pieces of retrieved context to answer the question. 
    If you don't know the answer, just say that you don't know. 
    Keep the answer concise and strictly based on the context.

    Question: {question} 

    Context: {context} 

    Answer:"""
    prompt = ChatPromptTemplate.from_template(template)

    def format_docs(docs):
        return "\n\n".join(doc.page_content for doc in docs)

    # 6. Build the LangChain Expression Language (LCEL) chain
    rag_chain = (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )
    
    return rag_chain

# Execute the system
if __name__ == "__main__":
    # Assume we have a company policy document
    chain = build_rag_system("company_policy.txt")
    
    # Ask a question based on the document
    question = "What is the policy for remote work and required hardware?"
    print(f"\nQuestion: {question}")
    
    response = chain.invoke(question)
    print(f"\nAnswer: {response}")

Understanding the Code

Let's break down the critical steps in the script above.

Chunking (RecursiveCharacterTextSplitter): LLMs have context limits (the amount of text they can process at once). Furthermore, burying the specific answer inside a massive document degrades the model's accuracy. By splitting the document into overlapping chunks of 1000 characters, we ensure that we only send the most highly relevant snippets of information to the LLM. The 200-character overlap prevents critical context from being split in half across two chunks.

Embeddings and Vector Stores: The OpenAIEmbeddings class converts our text chunks into numerical vectors. A vector is simply an array of numbers that represents the semantic meaning of the text. ChromaDB stores these vectors. When a user asks a question, their question is also converted into a vector, and Chroma performs a mathematical operation (like cosine similarity) to find the text chunks with vectors most similar to the question vector.

LCEL Chain: LangChain Expression Language (LCEL) uses the pipe (|) operator to create a declarative chain of operations. In our script, the chain first takes the user's question and passes it to the retriever to get the context. It then passes both the question and formatted context into the Prompt template. The formatted prompt is fed into the LLM, and finally, the output parser extracts the raw string response.

Why RAG Beats Fine-Tuning

A common question among developers is whether they should use RAG or fine-tune a model on their custom data. In almost all enterprise knowledge-retrieval scenarios, RAG is vastly superior.

Fine-tuning is expensive, time-consuming, and difficult to update. If your company policy changes, you would theoretically need to retrain your fine-tuned model to make it "forget" the old policy and learn the new one. With a RAG system, updating knowledge is as simple as deleting the old document vector from your database and embedding the new one. Furthermore, RAG prevents hallucinations by forcing the model to cite specific source chunks, whereas a fine-tuned model relies on probabilistic memory, which is prone to error.

Frequently Asked Questions (FAQ)

Can I run this entirely locally without OpenAI?

Yes. LangChain supports integrations with local models via frameworks like Ollama or HuggingFace. You can replace the OpenAI embeddings with a local open-source embedding model (like `all-MiniLM-L6-v2`) and replace the generation model with a local LLM like Llama-3 running on your own hardware. This guarantees complete data privacy as no information ever leaves your machine.

What kind of databases can be used for Vector Stores?

For prototyping and local development, ChromaDB or FAISS are excellent in-memory choices. For scalable, production-grade applications with millions of documents, you should utilize managed vector databases like Pinecone, Weaviate, Qdrant, or even specialized vector extensions for traditional databases like pgvector for PostgreSQL.

How do I handle complex tables and images in my documents?

Standard text splitters struggle with complex document layouts containing tables and images. To solve this, you need a more advanced ingestion strategy, often referred to as Multi-Vector Retriever patterns or using specialized multimodal LLMs (like GPT-4V) to generate text summaries of images during the ingestion phase, embedding those summaries into the vector store instead.

Is RAG suitable for highly analytical queries?

If you need to ask analytical questions like "What was the total revenue in Q3 across all departments?", standard semantic RAG will fail, because the answer requires aggregating data rather than retrieving a specific text chunk. For structured data analysis, it is better to use a Text-to-SQL architecture, where the LLM writes an SQL query to execute against your relational database, rather than utilizing a vector store.