RAG with Vector Databases: Fixing Hallucinations with Embeddings and HNSW

You ask a large language model (LLM) a specific question about your company’s latest product specs. It answers confidently. But it’s wrong. This is hallucination, the Achilles' heel of generative AI. The model isn’t lying; it’s guessing based on static training data that likely doesn’t include yesterday’s update or your internal wiki. To fix this, engineers turn to Retrieval-Augmented Generation (RAG). Think of RAG as giving the LLM an open-book exam instead of making it memorize everything. At the heart of this system sits the vector database, acting as the AI’s long-term memory.

If you’re building AI applications today, understanding how vectors, embeddings, and indexing algorithms like HNSW work together isn’t just academic-it’s critical for shipping reliable products. Let’s break down how these components stop hallucinations in their tracks.

Why Standard LLMs Fail at Specific Knowledge

Large Language Models are incredible pattern matchers. They predict the next word based on billions of parameters learned during training. But they have two glaring flaws. First, their knowledge cutoff date means they don’t know what happened last week. Second, they lack access to private, proprietary data unless you fine-tune them, which is expensive and slow.

RAG solves both issues by retrieving relevant information from an external source before generating an answer. Instead of relying solely on its internal weights, the system fetches facts from a knowledge base-like a document store or database-and injects them into the prompt. This grounds the LLM in reality. If the retrieved text says the product weighs 5kg, the LLM is far more likely to say 5kg than guess 4.5kg.

The Role of Embeddings in Semantic Search

Before we can retrieve anything, we need to find it. Traditional keyword search looks for exact matches. If you search for "fast car," it won’t necessarily find documents about "high-speed vehicles." That’s where embeddings come in. An embedding is a mathematical representation of data converted into a high-dimensional vector-an array of numbers that captures semantic meaning.

Here’s how it works in practice:

• Chunking: You take your domain-specific dataset and split it into manageable pieces, or chunks.
• Vectorization: A specialized model, such as Amazon Titan Text Embedding v2 or the open-source all-MiniLM-L12-v2, converts each chunk into a vector.
• Storage: These vectors are stored in a vector database alongside metadata.

When a user asks a question, the same embedding model converts that query into a vector. The system then searches for vectors in the database that are closest to the query vector. Because similar meanings map to nearby points in vector space, you get results based on intent, not just keywords.

Vector Databases: The Infrastructure Layer

You might wonder why you can’t just use PostgreSQL or MySQL for this. Standard relational databases excel at exact matching but struggle with similarity searches across millions of high-dimensional vectors. Enter vector databases. These are purpose-built systems designed to perform approximate nearest neighbor (ANN) searches efficiently.

Tools like Pgvector extend existing relational databases to handle vector operations, while dedicated platforms like Pinecone or Weaviate offer specialized architectures. Regardless of the tool, the goal is the same: retrieve the top-k most similar vectors in milliseconds. For enterprise apps, latency matters. Users won’t wait five seconds for an answer when they expect instant feedback.

HNSW: Speeding Up Retrieval with Graph Indexing

Scanning every vector in a billion-record database to find the closest match is too slow. That’s why we use indexing algorithms. The gold standard right now is Hierarchical Navigable Small World (HNSW). HNSW creates a multi-layered graph structure that allows the system to traverse data rapidly.

Think of it like navigating a city. The top layer of the HNSW graph has long-range connections between major hubs (coarse details). As you move down layers, the connections become denser and shorter, focusing on local neighborhoods (fine details). When you run a query, the algorithm starts at the top, jumps quickly toward the general area of the target vector, and then drills down through lower layers to pinpoint the exact neighbors.

This approach strikes a balance between accuracy and speed. In a documented case study involving a 1 million-row dataset using Pgvector, building an HNSW index took about 33 minutes. Once built, query times dropped from several seconds to mere milliseconds-a 100x performance boost. This makes real-time RAG applications commercially viable.

There are alternatives, like IVFFlat (Inverted File Flat), which partitions vectors into clusters. IVFFlat can be faster to build and uses less memory, but HNSW generally offers better recall accuracy for complex queries. Many modern systems even combine methods using Reciprocal Rank Fusion (RRF) to blend results from different indexes, improving overall relevance.

Filtering: Adding Business Logic to Search

Semantic similarity alone isn’t always enough. Imagine a multi-tenant SaaS platform. If User A asks a question, you don’t want the system to retrieve context from User B’s private documents. You need filtering.

Vector databases allow you to apply metadata filters alongside similarity searches. For example, you can restrict results to documents tagged with `department: 'HR'` or `date > '2026-01-01'`. This ensures that the retrieved context is not only semantically relevant but also legally and operationally appropriate. Without robust filtering, RAG systems risk leaking sensitive data or providing outdated information.

Implementation requires careful schema design. Your table structure needs fields for the embedding vector, the original content, and rich metadata. For instance, a typical schema might look like this:

CREATE TABLE file_embeddings (
  id SERIAL PRIMARY KEY,
  embeddings vector(384),
  content TEXT NOT NULL,
  metadata JSONB
);

By combining HNSW indexing with metadata filtering, you create a powerful retrieval engine that scales securely.

Practical Implementation Tips

If you’re deploying RAG today, keep these heuristics in mind:

• Consistency is key: Always use the same embedding model for indexing and querying. Mixing models breaks the semantic space alignment.
• Co-locate resources: Store embeddings close to the source data to minimize network latency. If using AWS Aurora, consider generating embeddings directly within the database to reduce data movement.
• Tune your parameters: HNSW has tunable parameters like `m` (number of bi-directional links) and `ef_construction` (size of dynamic candidate list). Higher values improve accuracy but increase index build time and memory usage.
• Monitor recall vs. latency: There is no free lunch. You must decide if you prefer 99% accuracy with 100ms latency or 95% accuracy with 10ms latency, depending on your user experience goals.

The future of this tech lies in mixed-precision approaches and smarter hybrid search strategies. As models evolve, the infrastructure supporting them must remain flexible. But for now, mastering embeddings, HNSW, and filtering gives you the tools to build AI that actually knows what it’s talking about.