Vector store: finding the nearest

The problem

You have vectors - one per fragment from the previous chapter. When a query arrives, you need to find a few fragments closest in meaning to the query vector. You could compare the query against every vector in turn - and with a thousand fragments that works. But with a million fragments, scanning each one on every query becomes too slow.

The solution is a vector store: a specialized storage that holds vectors and can quickly find nearest neighbours without comparing the query against the whole archive. Here is what the store step plus search look like using a vector-database client:

# pip install pinecone openai
from pinecone import Pinecone
from openai import OpenAI

oai = OpenAI()
pc = Pinecone()
index = pc.Index("docs")

# store: put the vector + metadata + chunk text
def embed(text):
    return oai.embeddings.create(
        model="text-embedding-3-small", input=text
    ).data[0].embedding

index.upsert(vectors=[
    {"id": "c1", "values": embed("Refund policy: a product may be returned within 30 days."),
     "metadata": {"source": "faq.md", "section": "refund", "text": "..."}},
    {"id": "c2", "values": embed("The warranty on electronics is 12 months."),
     "metadata": {"source": "faq.md", "section": "warranty", "text": "..."}},
])

# search: top-k nearest to the query vector
res = index.query(
    vector=embed("how to get a refund for a purchase"),
    top_k=3,
    include_metadata=True,
    filter={"section": "refund"},   # metadata filter
)
for m in res.matches:
    print(m.id, round(m.score, 3), m.metadata["section"])

The database stores the vectors itself, searches for the nearest itself, and returns the top-k together with their cosine closeness - you do not have to write a scan by hand.

Why a separate database

A vector database solves two tasks at once: it stores millions of vectors and searches among them for the nearest by meaning (Pinecone, vector database basics). An ordinary database searches by exact equality or by keywords; a vector database searches by geometric closeness in dim-dimensional space, that is, by meaning (Pinecone, what is a vector database).

This is exactly what was missing at the search step: fast semantic search over the whole archive without letter-by-letter comparison.

Approximate nearest neighbour search (ANN)

Exact nearest-neighbour search (kNN) compares the query against all vectors - this guarantees the correct answer but grows linearly with the archive size. With millions of fragments people use approximate search (ANN, approximate nearest neighbour): it almost always finds the same neighbours but many times faster, sacrificing a small fraction of accuracy for speed.

A common ANN algorithm is HNSW (Hierarchical Navigable Small World): a multi-layer graph through which the search "hops" from distant nodes to the nearest in a logarithmic number of steps instead of scanning the whole set (Malkov & Yashunin, 2016, HNSW). Other systems build indexes on top of libraries like FAISS, which is purpose-built for fast search over millions of vectors.

Top-k and metadata filters

In a search you ask not for the single closest vector but for the top-k - several nearest ones (often k=3..10). This gives the model a few spare pieces in case the closest one does not fully cover the question.

Together with the vector, the database stores metadata - source, section, date, access rights. Metadata filters narrow the search to a matching subset in combination with the nearest-neighbour search - rather than scanning the whole index: for example, only documents from this department or only what is fresher than a certain date (Pinecone, metadata filtering). In the code above this is filter={"section": "refund"}.

Scale: millions of fragments without a scan

It is exactly the ANN index that makes semantic search practical on a large archive. Instead of comparing the query against each of millions of vectors, the HNSW graph leads to the answer in a logarithmic number of steps (Malkov & Yashunin, 2016, HNSW). So from 100000+ fragments the top-5 nearest are fetched in milliseconds, not in a full pass over the database.

When you do not need a vector database

A vector database is not always the right choice:

• Small corpus. With a few hundred or a thousand fragments, an exact in-memory scan (kNN) is simple, fast enough, and needs no separate infrastructure.
• Exact match matters more than meaning. If you need to search by exact codes, SKUs, or IDs - an ordinary database or a full-text index is more precise and cheaper.
• You already have a suitable database. Some ordinary DBMSs support vector search as an extension (for example pgvector for PostgreSQL, pgvector) - then a separate specialized database may be redundant.

The rule: take a vector database when the corpus is large AND the search is genuinely by meaning. Otherwise it adds complexity without payoff.

Try it yourself

• Open the ann-topk-drill interaction: find the query node and walk the drawn query path to its top-k neighbours. Drill (semantic zoom) into one of the selected neighbours and look at its cosine and metadata.
• Compare what comes back with and without a metadata filter: in the code above remove filter={"section": "refund"} and see how the top-k set changes.
• Estimate whether a vector database fits your case: how many fragments do you have, and do you search by meaning or by exact match - check against the section "When you do not need a vector database".

What is next

The vectors are in the index and the nearest are found quickly - next you need to turn the user's query into a vector and assemble the top-k on a live request. The next stop: search (searching by meaning at query time).

Back to the route map