Agenda

• What are Embeddings?
• What are Tokens?
• Embeddings vs. Tokens (The Context Gap)
• Common Misconceptions (The Parrot/Carrot Fun Fact)
• Vector Databases
• Vector DB vs. Traditional SQL
• The Problem with LLMs
• RAG: The Solution
• RAG Architecture
• Live Demo & Chatbot Deep Dive

What are Embeddings?

Embeddings are the fundamental building blocks of modern NLP.

• Definition: Numerical representations for words, letters, symbols, or images.
• Structure: They are typically continuous (can take any real number), dense vectors (likely not zero), and serve as a compressed notation to represent data.
• The Goal: To capture semantic meaning.

What are Tokens?

Before we get embeddings, we need Tokens. A computer cannot "read" text; it can only process numbers.

• Definition: The process of breaking down text into smaller units (tokens) and assigning them a static numeric ID.
• Words vs. Tokens: A token is not always a word.
  • Common words ("Apple") = 1 Token.
  • Complex words ("Unfriendliness") = Multiple Tokens ("Un", "friend", "li", "ness").

Embeddings vs. Tokens

The Context Problem: If "Bank" is always token #405, how does the model know the difference between a river bank and a financial bank?

The Process:

1. Each word gets converted into a token (Static ID).
2. Tokens get converted into initial embeddings.
3. The Magic: Through a mechanism called Self-Attention, the model looks at the whole proposition.
4. It updates the embedding to reflect its meaning in that specific phrase.

🧠 Deep Dive: The Self-Attention Mechanism

Note: Skip - Guide to Sources 

The Objective: To calculate a new Context Vector for each token by aggregating information from all other tokens based on relevance.

Technical Breakdown:

• Dot Product (Q·K): Mathematically measures vector alignment. High alignment = High relevance between words.
• Softmax: Normalizes scores into probabilities (0 to 1) so they sum to 100%.
• Weighted Sum: The final vector is not a "replacement" but a blend of the Values (V) of all tokens, weighted by their importance.

Vector Databases

Where do we store millions of these vector lists? Standard databases aren't built for this.

• Definition: Specialized databases designed to store, index, and query high-dimensional vectors.
• How they work: They use indexing algorithms to create a map of the data.
• Scalability: FAISS (In-memory, Fast) vs ChromaDB (Storage, Filtering).

Photo explanation:

Vector DB vs. Traditional DB

Why are they faster than just storing embeddings in a string column in SQL?

In a traditional DB, finding "similar" items would require comparing your query to every single row (Full Table Scan).

The Problem with LLMs

Large Language Models like GPT-4 are powerful, but they have a couple major weaknesses when deployed in business:

• 1. The Knowledge Cut-off: They are frozen in time.
• 2. No Private Knowledge: They don't have access to your internal database.
• 3. Hallucinations: They often confidently invent facts when unsure.
• 4. Security: A public model should not know your private confidential data!

RAG: Retrieval-Augmented Generation

To fix these problems, we don't need a bigger brain; we need a better library.

The Concept

Without RAG (Closed Book Exam): The student (LLM) must answer purely from memory.

With RAG (Open Book Exam): The student (LLM) is allowed to go to the library, find the relevant textbook page (Retrieval), and use that specific information.

How RAG Works (The Pipeline)

How do we technically implement this "Open Book" strategy?

Step 1: Indexing (Preparation)

• Get the data required for the use case, clean it, parse it (hardest part of RAG)
• Break documents into small chunks. (many ways to do that)
• Convert chunks to vectors and store in Vector DB.

Step 2: Retrieval (The Search)

• User asks: "Tell me about topic X."
• System receives question from user, converts it to EMBEDDING and
• System searches VectorDB for documents relevant to the Embedded question

Step 3: Generation (The Answer)

• We retrieve the documents from DB
• And then append the text as context to the llm along with the user question
• LLM generates a factual response.

2D Representation Demo

We use dimensionality reduction (PCA/t-SNE) to visualize these complex vectors on a 2D screen.

RAG Challenges in Production

Building a prototype is easy. Production is hard.

• 1. Handling Unstructured Data (ETL): Garbage In, Garbage Out. Extracting clean text from PDFs, complex tables, and messy HTML is 80% of the work.
• 2. Data Freshness (The Sync Problem): If you update a SQL row, the Vector DB becomes "stale".
  Solution: Cron Jobs or CDC (Change Data Capture) pipelines to re-embed data daily/hourly.
• 3. Query Routing: Does the user need "Technical Support" or "Sales Info"? We need a semantic router (Classifier) to pick the right index.
• 4. Memory & Context: Balancing "Chat History" vs. "Retrieved Documents" within the token limit.
• 5. Security: Especially for more general RAG systems, we need to restrict an user to ask questions about topics he is restricted to seeing.
• 6. Cost: Indexing a lot of documents, keeping a lot of memory, big prompts etc will add the cost quickly.
• 7. Speed: No one wants to wait 2 minutes for a query.

1. Storage & Retrieval Strategy

We use a hybrid approach to balance speed and data integrity.

• Vector Store (FAISS): We use all-MiniLM-L6-v2 for embeddings.
• Fast Retrieval Maps: To ensure O(1) access times, we maintain two in-memory hashmaps:
  • ID_to_Index
  • Index_to_ID
• The ID Hash: A deterministic signature:
  hash(folder_name + filename + sheet_name)
• Data Source: We do not store the actual file content in FAISS. We store metadata pointers. The source of truth is always MinIO.
• Periodic schedules that reindex data based on the hash

2. Live Logic Trace

How the system "reads" the file structure before answering.

{
  "unique_id": "4f257...db",
  "filename": "Risk_Summary_Global_Q3.xlsx",
  "sheet_name": "All Risks - Regions",
  "columns": [
    "0: Risk Group",
    "1: Net Total (USD)",
    "2: Gross Total (USD)"
  ],
  "sheet_summary": "Financial risk summary focusing on market exposure. 
   Identifies significant risk categories...",
  "is_complex": true,
  "last_modified": "2025-12-20 12:00:30",
  "isNewest": true,
  "minio_url": "s3://secure-vault/risk-reports/2025-12-20/summary.xlsx"
}

3. Data Hydration & LLM Handoff

We retrieve based on a similarity search on the file SUMMARY AND the other metadata such as columns, not just keywords.

1. Search: Find top matches in FAISS based on similarity.
2. Fetch: Use the stored URL to pull the binary from MinIO.
3. Parse: Clean the Excel/PDF (remove empty rows/cols).
4. Format: Convert the clean data into Markdown ( very easy to understand for LLMs).
5. Handoff: Pass the Markdown + User Query + Expanded Metadata to the LLM for final answer generation. Expanded metadata: File Name, Folder Location, Accuracy Score etc - it will prevent false positives

4. Intelligent Router (The Intent)

Every user message is categorized into one of 5 specific intents.

3 Functionalities: Content, MinIO, SQL 5 Possible outputs(currently):

5. Memory & Query Rewriting

We maintain a dedicated memory store for each user session. This bridge allows the system to understand context, pronouns, and intent shifts just like a human would.

The Benefit: This decoupling means our Vector Database and SQL engine never have to guess. They always receive complete, standalone instructions.

6. Index Manager & Scheduler

Ensuring consistency between MinIO and FAISS.

• Scheduler: Runs every 30 mins to scan the bucket.
• Validation: Checks last_modified date against the index.
• Update Logic: If the date differs OR the ID is new, we re-process. Otherwise, we skip to save resources.

7. Response Transparency

The response object contains critical metadata for user trust.

8. Security Guardrails

Paranoid security measures for code execution.

• Namespace Execution: Python runs in a restricted scope. No access to globals/OS.
• Defensive prompt engineering: Instruct LLM to redirect to BREACH_INTENT in case user tries to use malicious prompts.
• Restricted Python Built-ins: `os.system`, `subprocess`, `open(write)` are strictly blocked.
• SQL Restrictions:
  • User has SELECT only permissions.
  • Keywords like DROP, TRUNCATE, ALTER are blocked at the prompt injection level.

9. System Benefits

Why this architecture works for Enterprise:

Note: Skip 

10.Sources

• Simple Rag Implementation: Link
• HNSW Algorithm Link
• Embedding model used: Link
• Tokens vs embeddings: Link
• Faiss Database: Faiss Documentation

Note: Skip

11. Useful learning materials

• Generative Ai for beginners: Link
• RAG and Python tutorial: Link
• RAG Techniques(not for beginners) Link
• AI Tool Calling Link