Word embeddings

How meaning becomes geometry, and why it matters for retrieval

The most useful trick in modern AI work is the one no one explains. A piece of text, anything from a single word to a paragraph to an entire document, can be converted into a long list of numbers. The numbers do not look meaningful on their own. They are not coordinates anyone would recognize, not measurements of anything tangible, and not human-interpretable in isolation. But the numbers have a remarkable property: two pieces of text that mean similar things produce lists of numbers that are similar to each other. Two pieces of text that mean different things produce lists that are different. The conversion from text to numbers is called an embedding, and the lists of numbers are called vectors.

Embeddings are the unsung infrastructure of almost every modern AI system that does anything more than produce text. Every search engine that finds documents by topic rather than by exact word match is using embeddings. Every retrieval-augmented generation (RAG) system that pulls relevant chunks out of a corpus is using embeddings. Every recommendation system that suggests “items similar to this one” is using embeddings. Every clustering tool that groups documents by theme is using embeddings. The chat interface where you talk to a model is the visible part of the iceberg. Embeddings are most of what is underneath.

This article explains what embeddings are, what they look like, how they are created, what makes one embedding model different from another, what they are used for in practice, and what a practitioner should know to make decisions about using them. The goal is conceptual clarity. By the end of the article, the geometry of meaning should be familiar, and the practical reasons for using embeddings should be obvious.

What an embedding is

An embedding is the numerical representation of a piece of text produced by an embedding model. The piece of text can be any length the model accepts, from a single word to several pages. The output is a vector, which is a fixed-length list of numbers. Different models produce different vector lengths; common sizes range from around 384 numbers per vector to several thousand.

Each individual number in the vector represents some aspect of meaning the model learned during training. The exact meaning of any given number is not something a human can interpret directly. The model decided what each position should represent based on patterns it observed in massive amounts of text, and the result is a coordinate system for meaning that is internally coherent but not labeled in human-readable ways. What matters is the whole vector, not any single number inside it.

The crucial property is that vectors live in a shared space. Every piece of text that gets embedded by the same model lands somewhere in that space. Texts that are about the same thing, or that mean similar things, land near each other. Texts that are about different things land far from each other. The geometric relationships between vectors carry semantic information. Distance in the space is similarity in meaning.

This is the property that makes embeddings useful. Once text is in the space, you can do mathematical operations on it. You can measure how close two pieces of text are. You can find the nearest neighbors of a given piece of text. You can group texts that cluster together. None of this requires the model to read the text again. The text is already represented as a point in space, and the geometry does the work.

The diagram above shows the two key ideas. The top half shows the conversion: text in, vector out. A short phrase like “child welfare policy” goes into the embedding model, and a long list of numbers comes out. The vector itself is not interpretable, but it is a deterministic representation of that phrase in a particular embedding model’s coordinate space.

The bottom half shows what happens when many phrases are embedded by the same model and plotted in the resulting space, simplified down to two dimensions for visual purposes. Phrases about child welfare cluster together in one region. Phrases about law cluster together in another. Phrases about research methods land in a third region, and phrases about cooking land in a fourth. The clusters are visually separate. Within each cluster, the most semantically similar items are closest. Across clusters, the distances are larger.

What is being shown is not literal in any pictorial sense. Real embeddings live in spaces with hundreds or thousands of dimensions, not two. But the principle holds in any number of dimensions. Items that mean similar things land near each other. The simplification to two dimensions is a useful illustration of the geometric structure that exists in the higher-dimensional space.

How embeddings are created

The conceptual story of how embedding models are trained is straightforward, even if the engineering is not.

A neural network is shown enormous amounts of text, on the order of billions of words. It is given various tasks that require it to produce useful representations of the text. One common task is predicting whether two pieces of text are likely to appear together in a document, or whether a given pair was written about the same topic. To do this task well, the network has to learn representations that capture the meaning of the text, because surface features like exact word matches are insufficient. Two passages can be about the same topic without sharing any specific words, and the model has to learn that they are similar anyway.

After training on enough data, the network’s intermediate representations become a useful coordinate system for meaning. Texts about the same things land in similar regions, even when the exact words differ. The trained network is then frozen, and the part that produces these representations becomes the embedding model. New text is fed in, and the output vector reflects what the model learned about meaning during training.

The practical takeaway is that embeddings are not arbitrary. They are the product of training on massive amounts of text by a model designed to extract semantic regularities. The quality of the embeddings depends on the quality and breadth of the training data, the design of the network, and the training task. This is why different embedding models produce different vectors for the same text, and why some models work better than others for some kinds of content.

How embedding models differ

Not all embedding models are the same. Several axes of variation matter.

The first is the size of the vector. Small models produce vectors with a few hundred numbers. Large models produce vectors with several thousand. Larger vectors can capture more nuance but cost more to store and search. The tradeoff is between representational richness and computational expense. For many practical tasks, smaller models are sufficient.

The second is the training corpus. A model trained primarily on web text will represent everyday language well but may struggle with specialized vocabulary in legal, medical, or scientific writing. A model trained on biomedical literature will have rich representations for medical terms but may underperform on general text. Some models are trained on multilingual corpora and produce embeddings that work across languages; others are English-only and perform poorly on non-English text.

The third is the source. Some embedding models are commercial and accessed through APIs, similar to language models. OpenAI, Voyage, Cohere, and others offer paid embedding services with different prices, vector sizes, and quality characteristics. Other models are open-source and can be run locally, including the BGE family, the Sentence-Transformers library, and many models on Hugging Face. Local models avoid per-call costs and keep data on your machine, at the cost of needing the hardware to run them.

The fourth is the unit of input. Some models are designed for short texts (single sentences or paragraphs). Others are designed for longer passages (thousands of tokens at a time). The right choice depends on the granularity of the work. For document retrieval, longer-input models can embed entire chunks; for query matching, shorter-input models often suffice.

The practical consequence is that the choice of embedding model affects results. The same RAG system using two different embedding models will retrieve different chunks for the same query. The same clustering analysis using two different models will produce different cluster boundaries. For serious work, the embedding model is a design decision, not a default.

What embeddings are used for

The geometric property that similar meanings land near each other unlocks several common uses, all of which appear in the bottom of the diagram.

Semantic search is the most direct application. Traditional search matches keywords: a search for “foster care” finds documents that contain the exact phrase. Semantic search converts the query into an embedding, finds documents whose embeddings are close to the query’s embedding, and returns those. The result is search that surfaces documents about the same topic even when they use different words. A query for “out-of-home placement” will find documents about foster care, kinship care, and residential treatment, because all of these mean similar things and embed to nearby vectors.

Clustering is the second application. Given a corpus, embed every document, then run a clustering algorithm on the vectors. The result is groups of documents that the model considers similar, with no predefined labels and no manual tagging. This is a useful way to explore an unfamiliar corpus, identify themes, or discover natural categories that were not anticipated. The clusters are not always meaningful in the way a human would draw them, but they often reveal structure that would otherwise require labor-intensive coding to find.

Retrieval-augmented generation (RAG) is the application most people building AI systems encounter first. The pattern is to break a corpus into chunks, embed each chunk, store the embeddings in a database, and then at query time, embed the user’s question and retrieve the chunks whose embeddings are closest to it. Those retrieved chunks are then fed to a language model along with the question, so the model can generate an answer grounded in the retrieved content. The whole approach depends on embeddings to find the right chunks. Without embeddings, RAG would fall back to keyword search, which misses much of what the user is actually asking about.

Classification is the fourth common use. Given a small set of labeled examples, embed each one. To classify a new item, embed it and find its nearest labeled neighbors. Assign the most common label among the neighbors. This kind of nearest-neighbor classification can be effective for tasks where building a more complex model is not warranted, especially when the number of training examples per category is small.

There are other uses, including deduplication, anomaly detection, recommendation, and analogical reasoning. The pattern across all of them is the same: convert text to vectors, then use the geometry of the vector space to do work that would otherwise require either heavy manual effort or a much larger language model.

Practical examples

A few concrete scenarios illustrate where embeddings show up in practitioner work.

A researcher has a corpus of two thousand abstracts and wants to identify which ones discuss qualitative methods. With embeddings, the researcher can write out a short description of qualitative methodology, embed it, then rank every abstract in the corpus by similarity to that description. The top of the ranked list is highly likely to contain qualitative work. The bottom is unlikely to. The researcher reads through the top portion and confirms or refines from there. This is dramatically faster than reading every abstract.

A legal aid organization has years of intake notes and wants to find all the cases involving immigration matters, including those that did not use that exact phrase. Embeddings allow a search for “immigration concern” to retrieve cases mentioning visa status, deportation, asylum claims, work authorization, and similar topics that share semantic content but not vocabulary.

A faculty group is building a chatbot that answers questions about university policy. The relevant policies are scattered across hundreds of PDF documents. Without embeddings, the chatbot would either need to read every document on every query (slow and expensive) or rely on keyword search (low quality). With embeddings, each policy is broken into chunks and indexed once. At query time, the chatbot embeds the user’s question, retrieves the three or four most relevant chunks, and asks a language model to answer based on those chunks. The retrieval is fast, the cost is small, and the answers are grounded in the actual policy text.

A team analyzing case notes for a quality improvement project wants to discover what topics dominate without imposing categories from the start. They embed every case note, cluster the embeddings, inspect the clusters, and use the clusters as a starting point for inductive analysis. The clusters are not the answer, but they are a useful structure for exploration.

In each case, the embedding model is the engine that makes meaning into geometry. The pipeline above it is the thing that does the work.

What a practitioner should know

A few practical conclusions follow from all of this.

Embeddings are the layer that makes most search and retrieval over text actually work. If you are building or commissioning any system that needs to find relevant documents in a collection, embeddings are almost certainly involved, and the choice of embedding model is part of the system’s design.

Embedding models are not interchangeable. Switching from one to another changes the geometry of the space and therefore changes which documents are considered similar. Pipelines should be built and tested with a specific embedding model in mind, and changing the model later requires re-embedding the entire corpus.

Local embedding models are mature and effective for many tasks. Running an embedding model on your own hardware avoids per-call costs and keeps sensitive data on your machine. For privacy-conscious work, this is often the right choice.

Embeddings are typically created once per document and stored. The cost and time of embedding a corpus happens at index time, not at query time. Once the embeddings are stored, queries are fast and cheap. Updating the corpus means updating only the embeddings of the changed or added documents.

Semantic search retrieves documents about a topic, but it does not understand them. The retrieved documents still need to be read, summarized, or analyzed by something downstream, often a language model. Embeddings handle the retrieval step. The reasoning step is separate.

The quality of embeddings improves with the quality of the input. As discussed in the post on Markdown, clean structured text embeds better than messy PDF extractions. The preparation of source documents matters as much for embedding-based retrieval as it does for any other AI application.

Where this leaves the practitioner

Embeddings are the unobtrusive plumbing of modern AI work. They turn meaning into geometry and make it possible to find, group, and compare text at scale. The technology has matured to the point where embedding any reasonably sized corpus is fast, cheap, and reliable. The choices a practitioner faces are about which embedding model to use, how to preprocess the input, and how to wire the embeddings into the rest of the pipeline.

The mental model is simple. A piece of text becomes a point in a space. Similar texts are nearby points. Different texts are distant points. Almost every useful operation on a corpus, search, clustering, retrieval, classification, comparison, can be expressed as a geometric question in that space. Once that picture is in place, a great many AI workflows that previously seemed mysterious become legible. The work is not magic. It is geometry, applied to meaning, made possible by training models that learned the regularities of language by reading a great deal of it.