Definition
- RAG is a pattern that combines a retrieval system (vector search) with a generative model (LLM) to produce answers grounded in specific documents.
- Vector embedding is a fixed-length numeric array (e.g., 1536 floats) that represents the meaning of a piece of text; similar texts have close vectors.
- Vector database stores embeddings and supports similarity search (nearest neighbor lookup): Pinecone, Weaviate, pgvector, Chroma.
- Chunking splits large documents into smaller pieces (500-1000 tokens) with overlap so the retriever can find the most relevant piece.
- Grounding means the LLM answer cites and uses specific document content rather than generating from memory.
Think of the National Bank of India customer support. Without RAG, the support agent (LLM) answers from memory, which might be outdated or wrong. With RAG, the agent first searches the bank's policy database (retrieval), finds the relevant pages about current loan interest rates and minimum balance rules (the most similar documents), puts those pages on their desk (context window), and then answers the customer's question using those specific pages. The answer is grounded in the actual policy, not the agent's memory.
Understanding the Concepts
How does RAG work?
RAG has two phases: indexing (done once) and querying (done per request).
- Indexing phase:
- Load documents (PDFs, Markdown, database records).
- Split documents into chunks (500-1000 tokens).
- Convert each chunk into a vector embedding using an embedding model.
- Store the chunks and their embeddings in a vector database.
- Query phase:
- Convert the user's question into a vector embedding.
- Search the vector database for the top-K most similar chunks.
- Build a prompt with the question and the retrieved chunks as context.
- Send the prompt to the LLM.
- The LLM generates an answer grounded in the retrieved documents.
package com.bank.ai;
import java.util.List;
public class RAGPipelineDemo {
public static void main(String[] args) {
System.out.println("=== Indexing Phase (one-time) ===");
System.out.println("1. Load: loan-policy-2025.pdf, savings-rules.pdf");
System.out.println("2. Chunk: split into 500-token pieces with 100-token overlap");
System.out.println("3. Embed: convert each chunk to 1536-dim vector");
System.out.println("4. Store: insert vectors into pgvector database");
System.out.println();
System.out.println("=== Query Phase (per request) ===");
String question = "What is the current home loan interest rate?";
System.out.println("1. Question: " + question);
System.out.println("2. Embed question: [0.12, -0.45, 0.78, ...] (1536 dims)");
System.out.println("3. Vector search: top 3 most similar chunks");
System.out.println(" Chunk 1: 'Home loan rates effective Jan 2025: 8.5% for...' (score: 0.92)");
System.out.println(" Chunk 2: 'Eligibility for home loan: minimum income...' (score: 0.85)");
System.out.println(" Chunk 3: 'Processing fee: 0.5% of loan amount...' (score: 0.80)");
System.out.println("4. Build prompt with question + chunks as context");
System.out.println("5. LLM generates: 'The current home loan interest rate is 8.5%...'");
}
}
=== Indexing Phase (one-time) ===
1. Load: loan-policy-2025.pdf, savings-rules.pdf
2. Chunk: split into 500-token pieces with 100-token overlap
3. Embed: convert each chunk to 1536-dim vector
4. Store: insert vectors into pgvector database
=== Query Phase (per request) ===
1. Question: What is the current home loan interest rate?
2. Embed question: [0.12, -0.45, 0.78, ...] (1536 dims)
3. Vector search: top 3 most similar chunks
Chunk 1: 'Home loan rates effective Jan 2025: 8.5% for...' (score: 0.92)
Chunk 2: 'Eligibility for home loan: minimum income...' (score: 0.85)
Chunk 3: 'Processing fee: 0.5% of loan amount...' (score: 0.80)
4. Build prompt with question + chunks as context
5. LLM generates: 'The current home loan interest rate is 8.5%...'
How do you implement vector search with pgvector in Java?
pgvector is a PostgreSQL extension that stores vector embeddings and supports cosine similarity search.
-- Enable pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;
-- Create table for document chunks
CREATE TABLE document_chunks (
id SERIAL PRIMARY KEY,
source_file VARCHAR(255) NOT NULL,
chunk_text TEXT NOT NULL,
embedding vector(1536) NOT NULL,
created_at TIMESTAMP DEFAULT NOW()
);
-- Create index for fast similarity search
CREATE INDEX ON document_chunks
USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
-- Search for similar chunks
SELECT chunk_text, 1 - (embedding <=> $1::vector) AS similarity
FROM document_chunks
ORDER BY embedding <=> $1::vector
LIMIT 3;
package com.bank.ai;
public class PgVectorSearchDemo {
public static void main(String[] args) {
System.out.println("pgvector similarity search:");
System.out.println();
System.out.println("Query: 'home loan interest rate'");
System.out.println("Query embedding: [0.12, -0.45, 0.78, ...]");
System.out.println();
System.out.println("Results (cosine similarity):");
System.out.println(" 1. similarity=0.92 | 'Home loan rates effective Jan 2025: 8.5% fixed...'");
System.out.println(" 2. similarity=0.85 | 'Eligibility criteria: min income 50,000/month...'");
System.out.println(" 3. similarity=0.80 | 'Processing fee: 0.5% of sanctioned amount...'");
}
}
pgvector similarity search:
Query: 'home loan interest rate'
Query embedding: [0.12, -0.45, 0.78, ...]
Results (cosine similarity):
1. similarity=0.92 | 'Home loan rates effective Jan 2025: 8.5% fixed...'
2. similarity=0.85 | 'Eligibility criteria: min income 50,000/month...'
3. similarity=0.80 | 'Processing fee: 0.5% of sanctioned amount...'
How do you build the RAG prompt?
The prompt includes a system instruction, the retrieved context, and the user question. Tell the LLM to answer only from the provided context.
package com.bank.ai;
public class RAGPromptDemo {
public static void main(String[] args) {
String systemPrompt = """
You are a National Bank of India customer support assistant.
Answer the customer's question using ONLY the provided context.
If the answer is not in the context, say "I don't have that information."
Cite the source document when possible.
""";
String context = """
[Source: loan-policy-2025.pdf, page 12]
Home loan interest rates effective January 2025:
- Fixed rate: 8.5% per annum for loans up to 50 lakh
- Floating rate: 8.25% per annum (linked to repo rate)
- Processing fee: 0.5% of loan amount, max 25,000 INR
""";
String question = "What is the current home loan interest rate?";
System.out.println("RAG Prompt:");
System.out.println("System: " + systemPrompt.trim());
System.out.println();
System.out.println("Context: " + context.trim());
System.out.println();
System.out.println("Question: " + question);
System.out.println();
System.out.println("Expected LLM answer:");
System.out.println(" The current home loan interest rate at National Bank of India");
System.out.println(" is 8.5% fixed or 8.25% floating per annum.");
System.out.println(" (Source: loan-policy-2025.pdf, page 12)");
}
}
RAG Prompt:
System: You are a National Bank of India customer support assistant.
Answer the customer's question using ONLY the provided context.
If the answer is not in the context, say "I don't have that information."
Cite the source document when possible.
Context: [Source: loan-policy-2025.pdf, page 12]
Home loan interest rates effective January 2025:
- Fixed rate: 8.5% per annum for loans up to 50 lakh
- Floating rate: 8.25% per annum (linked to repo rate)
- Processing fee: 0.5% of loan amount, max 25,000 INR
Question: What is the current home loan interest rate?
Expected LLM answer:
The current home loan interest rate at National Bank of India
is 8.5% fixed or 8.25% floating per annum.
(Source: loan-policy-2025.pdf, page 12)
What are common mistakes with RAG?
Mistake 1: Chunks too large or too small
Wrong: using entire documents as chunks (too large, embedding loses specificity) or single sentences (too small, loses context).
Correct: 500-1000 tokens with 100-200 token overlap.
Mistake 2: Not telling the LLM to stick to the context
Wrong: the LLM answers from its training data, ignoring the retrieved documents.
Correct: the system prompt explicitly says "Answer ONLY from the provided context. If the answer is not there, say so."
What should a senior engineer know?
- Hybrid search. Combine vector (semantic) search with keyword (BM25) search. Some queries are better served by exact keyword matching ("IFSC code SBIN0001234") while others need semantic understanding ("how to transfer money internationally").
- Re-ranking. After retrieval, use a cross-encoder model to re-rank the top-K results. The initial vector search is fast but approximate. The re-ranker is slower but more accurate.
- Evaluation metrics. Measure retrieval quality with recall@k (did the right chunk appear in the top K?) and answer quality with faithfulness (does the answer match the retrieved context?) and relevance (does the answer address the question?).
- Metadata filtering. Store metadata (document date, department, classification) with each chunk. Filter by metadata before vector search to narrow results: only loan policy documents from 2025.
- Spring AI. The Spring AI project provides a RAG framework for Java: document loaders, text splitters, embedding clients, and vector store integrations. It abstracts the plumbing so you focus on the business logic.
How do you handle document updates in a RAG system?
When banking policy documents change, the RAG system must re-index them. Stale embeddings can return outdated information. A pipeline should detect document changes, re-embed affected chunks, and update the vector store.
package com.bank.ai;
public class RagDocumentUpdate {
public static void main(String[] args) {
System.out.println("RAG document update pipeline:");
System.out.println(" 1. Detect changed documents (hash comparison)");
System.out.println(" 2. Re-chunk modified documents");
System.out.println(" 3. Generate new embeddings");
System.out.println(" 4. Upsert into vector store (replace old chunks)");
System.out.println(" 5. Verify retrieval quality with test queries");
}
}
Expected output:
RAG document update pipeline:
1. Detect changed documents (hash comparison)
2. Re-chunk modified documents
3. Generate new embeddings
4. Upsert into vector store (replace old chunks)
5. Verify retrieval quality with test queries
How do you measure RAG retrieval quality?
Retrieval quality determines whether the LLM receives relevant context. Measure recall (did we find the right documents?) and precision (did we avoid irrelevant documents?). Use a test set of questions with known source documents.
package com.bank.ai;
public class RagQualityMetrics {
public static void main(String[] args) {
System.out.println("RAG quality metrics:");
System.out.println(" Recall: Did we retrieve the correct source documents?");
System.out.println(" Precision: Did we avoid irrelevant documents?");
System.out.println(" MRR: How high is the correct document ranked?");
System.out.println(" Answer correctness: Does the LLM answer match ground truth?");
System.out.println(" Faithfulness: Is the answer grounded in retrieved context?");
}
}
Expected output:
RAG quality metrics:
Recall: Did we retrieve the correct source documents?
Precision: Did we avoid irrelevant documents?
MRR: How high is the correct document ranked?
Answer correctness: Does the LLM answer match ground truth?
Faithfulness: Is the answer grounded in retrieved context?
What are common interview follow-up questions?
- What is the difference between RAG and fine-tuning?
- How do you handle documents that are updated frequently?
- What embedding model do you recommend for banking documents?
- How do you evaluate the quality of a RAG system?
- What is the role of chunking strategy in RAG quality?
Visual Overview
RAG Pipeline
=============
Indexing (one-time)
───────────────────
Documents ──> Chunker ──> Embedding Model ──> Vector DB
(PDF, MD) (500 tok) (text -> [1536]) (pgvector)
Query (per request)
────────────────────
User Question
│
▼
Embedding Model ──> Query Vector [1536]
│
▼
Vector DB ──> Top 3 similar chunks
│
▼
Prompt Builder ──> System + Context + Question
│
▼
LLM ──> Grounded Answer (cites documents)
Without RAG With RAG
─────────── ────────
LLM answers from LLM answers from
training memory actual documents
May hallucinate Grounded in data
Outdated info Current info
No citations Cites sources
Key Takeaway
RAG solves the biggest problem with LLMs in banking: hallucination. A plain LLM might confidently state that the home loan interest rate is 7.5% because that was the rate in its training data. A RAG system retrieves the current loan policy document from the bank's knowledge base, includes the relevant section in the prompt, and the LLM generates an answer grounded in the actual 8.5% rate.
The pipeline has two phases. The indexing phase loads documents, splits them into chunks, converts each chunk to a vector embedding, and stores everything in a vector database. The query phase converts the user's question to a vector, finds the most similar chunks, and builds a prompt with those chunks as context. The LLM generates from the context, not from memory.
Chunking strategy directly affects quality. Chunks too large lose specificity in the embedding. Chunks too small lose context. The sweet spot is 500-1000 tokens with 100-200 token overlap. The overlap ensures that a relevant sentence split across two chunks is captured by at least one.
For banking, RAG enables powerful internal tools: a policy search system for compliance officers, a customer support chatbot that answers from the bank's FAQ, and a documentation assistant that helps developers understand legacy codebases.
One sentence to remember: RAG retrieves relevant documents from a knowledge base and includes them in the LLM prompt, so the answer is grounded in real data instead of the model's training memory.
---