Six years ago, I was packing for a sabbatical in China. My plan was to be a visiting professor at Southwest University in Chongqing -- a place that I consider my second hometown. Well, then COVID happened. And the sabbatical I had planned never happened. Instead, I spent the time in lockdown making the best of the experience. I learned Python programming to go deep into text analysis, while converting my backyard into an urban farm. I enjoyed what I created, making the best of the situation. I was satisfied with the sabbatical, but still a bit disappointed.

This year, I have another shot at a sabbatical. This is probably my last one, and I really wanted it to be significant. My plan was to launch my sabbatical with the trip that was lost to COVID. I would return to China and then visit some new places. Everything was perfect. I was wrapping up my academic obligations while dropping into my workshop to make a few things. I love making things, similar to how I work with data. I love the challenge of taking raw materials (e.g., data or wood) and creating something unique (e.g., knowledge or functional art). But a tornado rumbled through Ann Arbor, leaving me with a flooded home that insurance wouldn’t cover. So, my tools, time, and skills shifted from creative projects to tasks that I don’t enjoy.

I finally got my house put back together, giving me a window of time to balance my sabbatical travel plans with getting back to my projects in my workshop. But time was slipping away, and I was losing momentum and interest in my travels, my projects, and life in general. The launch into my sabbatical had become a comedy of errors.

The big question

Everything went on hold while I went to St. Louis to attend a doctoral defense. At the last moment, I reached out to an extremely talented and accomplished professor, Peter Boumgarden. He is well known for his work in organizational strategy and executive leadership. We met over 20 years ago in a structural equation modeling course and have remained loosely connected since. I feel fortunate that he found some time for a one-on-one with me.

After a warm greeting, Peter referred to my writing about AI and asked, “Brian, where do you see all of this going?” I was surprised on two accounts. First, is somebody actually reading what I wrote?! More significantly, the question was presented with intent. What is the intent behind all those keystrokes? My life has been at a pace that I haven’t been able to focus on this type of question. I admitted I didn’t know, but I really wanted this sabbatical to give me time to figure it out.

A framework for understanding

I’m certain that Peter knew I was stuck. He dropped into his expert role, where he excels at being an advisor, coach, and friend. He shared with me an activity he uses in his leadership training to bring some organization to my racing, scattered thoughts. He encouraged me to think about things that I wanted to accelerate in life; to go deeper and be more productive. To think about new activities that are adjacent to my core. And finally, to identify alternatives—activities that are completely different from what I am doing.

At the moment, I wished he could have just given me the answers. But I know that the real work was my own burden. The more I thought about his questions, the more I realized that the measures of success and happiness in my life were entirely wrong. I was benchmarking my own activities in the same way I was benchmarking AI models. But the benchmarks for myself were entirely wrong. I wasn’t approaching my sabbatical trip with intentions that would help set the next period of my life. The creative projects were tasks and deliverables. The pace of my work and my life was no longer manageable, and I was producing simply for the sake of producing.

AI, which is supposed to make life easier and more manageable, was actually making me busier. Even the messages that I received were tokens generated by an LLM. The authenticity was gone. AI is truly a disruptive technology.

Benchmarking the Alternative

I’m still going to use my sabbatical trip to think carefully about what to accelerate and what can be adjacent. But, for now, I have really come to understand what my new alternative is. The creative process is not about optimizing for speed and accuracy, but about discovery, finding inspiration, and exploring the decision space. I need to focus on the process, not the outcome. That is what I was missing.

I sent Peter a text message asking, “What comes after accelerate and adjacent?” A few minutes later, he confirmed it was “alternative.” My alternative for the next stage of my life is really about going back to my desire to create. I started my college experience as an art student, and that is where I want to return. A different style, a different medium, but an opportunity to create.

I returned to my workshop to find my unfinished projects. I dusted them off and quickly saw the imperfections in the projects and in myself.

I was ready to throw them away because I knew they would not have any value. But doing so would be reinforcing my old benchmarks. In the same way I dusted off projects, I dusted off the tools and focused on the process. I discovered some new techniques and was pleased that some imperfections were interesting features. They didn’t turn out to be my best pieces, but I learned a lot. This offered both a sense of closure on my old ways of working and renewed inspiration and motivation.

What’s next?

I still need to give the accelerate-adjacent-alternative framework more thought, but I have made considerable progress in one area. I have been working around the clock in my workshop with a sense of motivation and purpose that has been absent from my work. I’ve taken a few short breaks to finish planning my sabbatical trip, knowing exactly what I want to see and why.

Moving forward, I am committed to matching every bit of academic work with creative activities, with benchmarks focused on process rather than outcomes. With the right process, I know that right outcomes will follow. I’m ready to depart for China. Usually, I bring manuscripts and teaching materials. This time, I have gifts that I have created for my friends. I’m still developing my skills – they aren’t perfect, but I thoroughly enjoyed the process.

Bon voyage!

Embeddings are the unsung infrastructure of almost every modern AI system that does anything more than produce text. Every search engine that finds documents by topic rather than by exact word match is using embeddings. Every retrieval-augmented generation (RAG) system that pulls relevant chunks out of a corpus is using embeddings. Every recommendation system that suggests “items similar to this one” is using embeddings. Every clustering tool that groups documents by theme is using embeddings. The chat interface where you talk to a model is the visible part of the iceberg. Embeddings are most of what is underneath.

This article explains what embeddings are, what they look like, how they are created, what makes one embedding model different from another, what they are used for in practice, and what a practitioner should know to make decisions about using them. The goal is conceptual clarity. By the end of the article, the geometry of meaning should be familiar, and the practical reasons for using embeddings should be obvious.

What an embedding is

An embedding is the numerical representation of a piece of text produced by an embedding model. The piece of text can be any length the model accepts, from a single word to several pages. The output is a vector, which is a fixed-length list of numbers. Different models produce different vector lengths; common sizes range from around 384 numbers per vector to several thousand.

Each individual number in the vector represents some aspect of meaning the model learned during training. The exact meaning of any given number is not something a human can interpret directly. The model decided what each position should represent based on patterns it observed in massive amounts of text, and the result is a coordinate system for meaning that is internally coherent but not labeled in human-readable ways. What matters is the whole vector, not any single number inside it.

The crucial property is that vectors live in a shared space. Every piece of text that gets embedded by the same model lands somewhere in that space. Texts that are about the same thing, or that mean similar things, land near each other. Texts that are about different things land far from each other. The geometric relationships between vectors carry semantic information. Distance in the space is similarity in meaning.

This is the property that makes embeddings useful. Once text is in the space, you can do mathematical operations on it. You can measure how close two pieces of text are. You can find the nearest neighbors of a given piece of text. You can group texts that cluster together. None of this requires the model to read the text again. The text is already represented as a point in space, and the geometry does the work.

The diagram above shows the two key ideas. The top half shows the conversion: text in, vector out. A short phrase like “child welfare policy” goes into the embedding model, and a long list of numbers comes out. The vector itself is not interpretable, but it is a deterministic representation of that phrase in a particular embedding model’s coordinate space.

The bottom half shows what happens when many phrases are embedded by the same model and plotted in the resulting space, simplified down to two dimensions for visual purposes. Phrases about child welfare cluster together in one region. Phrases about law cluster together in another. Phrases about research methods land in a third region, and phrases about cooking land in a fourth. The clusters are visually separate. Within each cluster, the most semantically similar items are closest. Across clusters, the distances are larger.

What is being shown is not literal in any pictorial sense. Real embeddings live in spaces with hundreds or thousands of dimensions, not two. But the principle holds in any number of dimensions. Items that mean similar things land near each other. The simplification to two dimensions is a useful illustration of the geometric structure that exists in the higher-dimensional space.

How embeddings are created

The conceptual story of how embedding models are trained is straightforward, even if the engineering is not.

A neural network is shown enormous amounts of text, on the order of billions of words. It is given various tasks that require it to produce useful representations of the text. One common task is predicting whether two pieces of text are likely to appear together in a document, or whether a given pair was written about the same topic. To do this task well, the network has to learn representations that capture the meaning of the text, because surface features like exact word matches are insufficient. Two passages can be about the same topic without sharing any specific words, and the model has to learn that they are similar anyway.

After training on enough data, the network’s intermediate representations become a useful coordinate system for meaning. Texts about the same things land in similar regions, even when the exact words differ. The trained network is then frozen, and the part that produces these representations becomes the embedding model. New text is fed in, and the output vector reflects what the model learned about meaning during training.

The practical takeaway is that embeddings are not arbitrary. They are the product of training on massive amounts of text by a model designed to extract semantic regularities. The quality of the embeddings depends on the quality and breadth of the training data, the design of the network, and the training task. This is why different embedding models produce different vectors for the same text, and why some models work better than others for some kinds of content.

How embedding models differ

Not all embedding models are the same. Several axes of variation matter.

The first is the size of the vector. Small models produce vectors with a few hundred numbers. Large models produce vectors with several thousand. Larger vectors can capture more nuance but cost more to store and search. The tradeoff is between representational richness and computational expense. For many practical tasks, smaller models are sufficient.

The second is the training corpus. A model trained primarily on web text will represent everyday language well but may struggle with specialized vocabulary in legal, medical, or scientific writing. A model trained on biomedical literature will have rich representations for medical terms but may underperform on general text. Some models are trained on multilingual corpora and produce embeddings that work across languages; others are English-only and perform poorly on non-English text.

The third is the source. Some embedding models are commercial and accessed through APIs, similar to language models. OpenAI, Voyage, Cohere, and others offer paid embedding services with different prices, vector sizes, and quality characteristics. Other models are open-source and can be run locally, including the BGE family, the Sentence-Transformers library, and many models on Hugging Face. Local models avoid per-call costs and keep data on your machine, at the cost of needing the hardware to run them.

The fourth is the unit of input. Some models are designed for short texts (single sentences or paragraphs). Others are designed for longer passages (thousands of tokens at a time). The right choice depends on the granularity of the work. For document retrieval, longer-input models can embed entire chunks; for query matching, shorter-input models often suffice.

The practical consequence is that the choice of embedding model affects results. The same RAG system using two different embedding models will retrieve different chunks for the same query. The same clustering analysis using two different models will produce different cluster boundaries. For serious work, the embedding model is a design decision, not a default.

What embeddings are used for

The geometric property that similar meanings land near each other unlocks several common uses, all of which appear in the bottom of the diagram.

Semantic search is the most direct application. Traditional search matches keywords: a search for “foster care” finds documents that contain the exact phrase. Semantic search converts the query into an embedding, finds documents whose embeddings are close to the query’s embedding, and returns those. The result is search that surfaces documents about the same topic even when they use different words. A query for “out-of-home placement” will find documents about foster care, kinship care, and residential treatment, because all of these mean similar things and embed to nearby vectors.

Clustering is the second application. Given a corpus, embed every document, then run a clustering algorithm on the vectors. The result is groups of documents that the model considers similar, with no predefined labels and no manual tagging. This is a useful way to explore an unfamiliar corpus, identify themes, or discover natural categories that were not anticipated. The clusters are not always meaningful in the way a human would draw them, but they often reveal structure that would otherwise require labor-intensive coding to find.

Retrieval-augmented generation (RAG) is the application most people building AI systems encounter first. The pattern is to break a corpus into chunks, embed each chunk, store the embeddings in a database, and then at query time, embed the user’s question and retrieve the chunks whose embeddings are closest to it. Those retrieved chunks are then fed to a language model along with the question, so the model can generate an answer grounded in the retrieved content. The whole approach depends on embeddings to find the right chunks. Without embeddings, RAG would fall back to keyword search, which misses much of what the user is actually asking about.

Classification is the fourth common use. Given a small set of labeled examples, embed each one. To classify a new item, embed it and find its nearest labeled neighbors. Assign the most common label among the neighbors. This kind of nearest-neighbor classification can be effective for tasks where building a more complex model is not warranted, especially when the number of training examples per category is small.

There are other uses, including deduplication, anomaly detection, recommendation, and analogical reasoning. The pattern across all of them is the same: convert text to vectors, then use the geometry of the vector space to do work that would otherwise require either heavy manual effort or a much larger language model.

Practical examples

A few concrete scenarios illustrate where embeddings show up in practitioner work.

A researcher has a corpus of two thousand abstracts and wants to identify which ones discuss qualitative methods. With embeddings, the researcher can write out a short description of qualitative methodology, embed it, then rank every abstract in the corpus by similarity to that description. The top of the ranked list is highly likely to contain qualitative work. The bottom is unlikely to. The researcher reads through the top portion and confirms or refines from there. This is dramatically faster than reading every abstract.

A legal aid organization has years of intake notes and wants to find all the cases involving immigration matters, including those that did not use that exact phrase. Embeddings allow a search for “immigration concern” to retrieve cases mentioning visa status, deportation, asylum claims, work authorization, and similar topics that share semantic content but not vocabulary.

A faculty group is building a chatbot that answers questions about university policy. The relevant policies are scattered across hundreds of PDF documents. Without embeddings, the chatbot would either need to read every document on every query (slow and expensive) or rely on keyword search (low quality). With embeddings, each policy is broken into chunks and indexed once. At query time, the chatbot embeds the user’s question, retrieves the three or four most relevant chunks, and asks a language model to answer based on those chunks. The retrieval is fast, the cost is small, and the answers are grounded in the actual policy text.

A team analyzing case notes for a quality improvement project wants to discover what topics dominate without imposing categories from the start. They embed every case note, cluster the embeddings, inspect the clusters, and use the clusters as a starting point for inductive analysis. The clusters are not the answer, but they are a useful structure for exploration.

In each case, the embedding model is the engine that makes meaning into geometry. The pipeline above it is the thing that does the work.

What a practitioner should know

A few practical conclusions follow from all of this.

Embeddings are the layer that makes most search and retrieval over text actually work. If you are building or commissioning any system that needs to find relevant documents in a collection, embeddings are almost certainly involved, and the choice of embedding model is part of the system’s design.

Embedding models are not interchangeable. Switching from one to another changes the geometry of the space and therefore changes which documents are considered similar. Pipelines should be built and tested with a specific embedding model in mind, and changing the model later requires re-embedding the entire corpus.

Local embedding models are mature and effective for many tasks. Running an embedding model on your own hardware avoids per-call costs and keeps sensitive data on your machine. For privacy-conscious work, this is often the right choice.

Embeddings are typically created once per document and stored. The cost and time of embedding a corpus happens at index time, not at query time. Once the embeddings are stored, queries are fast and cheap. Updating the corpus means updating only the embeddings of the changed or added documents.

Semantic search retrieves documents about a topic, but it does not understand them. The retrieved documents still need to be read, summarized, or analyzed by something downstream, often a language model. Embeddings handle the retrieval step. The reasoning step is separate.

The quality of embeddings improves with the quality of the input. As discussed in the post on Markdown, clean structured text embeds better than messy PDF extractions. The preparation of source documents matters as much for embedding-based retrieval as it does for any other AI application.

Where this leaves the practitioner

Embeddings are the unobtrusive plumbing of modern AI work. They turn meaning into geometry and make it possible to find, group, and compare text at scale. The technology has matured to the point where embedding any reasonably sized corpus is fast, cheap, and reliable. The choices a practitioner faces are about which embedding model to use, how to preprocess the input, and how to wire the embeddings into the rest of the pipeline.

The mental model is simple. A piece of text becomes a point in a space. Similar texts are nearby points. Different texts are distant points. Almost every useful operation on a corpus, search, clustering, retrieval, classification, comparison, can be expressed as a geometric question in that space. Once that picture is in place, a great many AI workflows that previously seemed mysterious become legible. The work is not magic. It is geometry, applied to meaning, made possible by training models that learned the regularities of language by reading a great deal of it.

Closing this gap is worth a few minutes of attention because tokens are the unit that determines almost everything practical about working with these systems. The cost of a call is computed in tokens. The size of the context window is measured in tokens. The speed at which the model generates a response is measured in tokens per second. The reason a long document costs more than a short one is that it has more tokens. The reason that some languages cost more than others to process is that they tokenize less efficiently. Once you know what a token is, the practical behavior of these systems becomes much easier to predict.

This article explains what tokenization is, how it works conceptually, what tokens look like in practice, why the count varies across kinds of text, and what to do with this information when working with LLMs in real projects. No prior technical background is assumed.

What tokenization is

Tokenization is the step that converts a piece of text into a sequence of small units that the model can process. The units are called tokens. A token is sometimes a whole word, sometimes a part of a word, sometimes a single character or piece of punctuation. The exact rules are determined by a tokenizer, which is a small program that comes paired with each model.

The reason for breaking text into tokens rather than processing whole words is practical. Models work with a fixed vocabulary of possible tokens, typically somewhere between 50,000 and 200,000 entries. Languages have far more words than that, especially when you count inflected forms, technical terms, names, typos, and made-up words. Trying to give every possible word its own slot in the vocabulary would be unworkable. Tokenizers solve this problem by giving common words a single token each, while rarer words get split into smaller pieces that combine to spell them out. This way the vocabulary stays small, but every possible string of text can still be represented.

The practical effect is that common, ordinary text tokenizes very efficiently. Rare or unusual text tokenizes less efficiently. The same idea, expressed in different words, can have meaningfully different token counts.

The diagram above shows what happens when a typical English sentence is tokenized. The four-word sentence “Understanding tokenization is useful.” becomes six tokens. The word “Understanding” is common enough that the tokenizer keeps it as a single token. The word “tokenization” is less common and gets split in two: “ token” and “ization”. The word “ is” is its own token (with the leading space attached). The word “ useful” is its own token. The period is its own token. Six tokens for four words.

This pattern, where common words stay whole and rarer words split into pieces, is the heart of how tokenization works in modern LLMs. There is nothing magical about it. The tokenizer was trained on a lot of text, learned which sequences of characters appear most often, and assigned single tokens to those frequent sequences. Rarer sequences have to be assembled from smaller pieces.

Why token counts vary

The bottom of the diagram shows four short examples that all illustrate how the same kind of pattern produces different counts.

A plain English sentence like “The cat sat on the mat.” takes seven tokens. Each word is common and tokenizes cleanly. The period is one token. This is the efficient case.

A long technical word like “Antidisestablishmentarianism” takes around ten tokens by itself, because the tokenizer has to assemble it from smaller pieces. The word fits in your head as one concept, but the model is reading ten chunks. This is why technical writing in specialized fields tends to use more tokens than everyday prose: there are simply more uncommon words.

A short snippet of structured data like the JSON object {”name”: “Smith”, “year”: 2023} takes around fourteen tokens, even though it has only a small number of words. Punctuation, brackets, quotation marks, and digits each consume tokens. Code and structured data are often more token-expensive than equivalent prose, which is one of the reasons that JSON-heavy workflows can rack up token counts faster than people expect.

A short word in another language, like the Japanese greeting こんにちは, takes around five tokens. The same idea in English (”hello”) would be one or two. This is because most modern tokenizers were trained primarily on English-heavy data, and they encode English very efficiently while encoding other writing systems less efficiently. The practical consequence is that the same task in non-English languages often costs more and consumes more of the context window.

The general rule of thumb is that one token is roughly three quarters of a word in English, or about four characters of text. A 1,000-word document is approximately 1,300 tokens. A 100-page book is approximately 50,000 tokens. These rough conversions are useful for sizing a job before running it.

Why this is worth knowing

Three practical consequences follow from understanding tokens.

The first is cost. Almost every commercial LLM API charges per token, with separate rates for the input you send and the output you receive. A long input is more expensive than a short one. A model that produces a long answer costs more than one that produces a short one. The total cost of a job is the cost per token multiplied by the number of tokens, summed across every call. For a single chat session this is usually trivial. For a pipeline that processes thousands of documents, the math matters, and being able to estimate token counts in advance is the difference between a budget that holds and a budget that does not.

The second is the context window. As discussed in the post on context windows, the window is a fixed budget that has to hold the input, the model’s thinking, and the output. The size of that budget is measured in tokens. A 128,000-token window holds approximately 95,000 words, which sounds like a lot until you start filling it with documents. A research article might be six to eight thousand tokens. Twenty articles is most of the budget. Thirty articles is the whole budget, with no room for the question or the answer. Estimating tokens is how you predict, before running anything, whether your input will actually fit.

The third is consistency. Because tokenization happens silently, two superficially similar inputs can have very different token counts. Switching from one document format to another can change token counts substantially even when the content is the same. Adding markup, adding headers, adding line numbers, all of this affects the count. Pipelines that depend on token-bound processing should be tested with realistic samples, not estimated from word counts alone.

What different tokenizers do differently

There is no single tokenizer used by all models. Each model family ships with its own. OpenAI uses one family of tokenizers, Anthropic uses another, Google uses yet another, and open-source models often use their own. The differences are not large, but they exist. The same text can take 1,200 tokens in one model and 1,250 in another. For most purposes this does not matter. For tight budget calculations, the relevant tokenizer is the one belonging to the specific model being used. Most providers offer a tokenizer tool on their developer site that lets you paste in text and see the exact count.

The differences across tokenizers also help explain why some models handle certain languages better than others. A tokenizer trained on a corpus that included substantial non-English text will tokenize that language more efficiently than one trained almost entirely on English. This can have downstream effects on how well the model performs in those languages, because efficient tokenization means more of the meaningful content fits in the window and more of the model’s attention budget can be spent on the substance rather than on assembling fragmented words.

Practical guidance

A few practical conclusions follow from all of this.

When estimating the size of a job, count tokens, not words. Most provider websites have a free tokenizer tool. Paste in a typical document, get the count, and multiply by the number of documents. This gives you a real number to size against.

When the input feels large, check the count before running anything. If the input plus the expected output exceeds the window, the model will either truncate or fail. Knowing this in advance is much cheaper than discovering it on a long batch run.

When working with structured data or code, expect higher token counts than the equivalent prose. Plan accordingly. If you are processing JSON-heavy material, the cost per item will be higher than the page count suggests.

When working in languages other than English, expect higher token counts and proportionally higher cost. The same task in a non-English language is often two to three times more expensive, simply because the tokenizer is less efficient on that script.

When optimizing a prompt for cost, words matter, but so does formatting. Excess whitespace, repeated headers, and verbose instructions all consume tokens. A concise prompt that gives the model the same information for fewer tokens is a better prompt for production use, holding accuracy constant.

Where this leaves the practitioner

Tokens are the underlying unit of measurement for everything that happens with a language model. They determine cost, they determine what fits in the window, and they determine how the model reads what you sent. None of this is technical in any specialized sense. It is just the layer beneath the words. Once you know that the model is reading tokens, the practical behavior of these systems becomes legible: you can estimate sizes before running jobs, predict where costs will land, anticipate where pipelines will run into limits, and design prompts that respect the format the model is actually receiving.

The text you type and the tokens the model reads are not exactly the same thing. The gap is small but consequential. Knowing that the gap exists is the first step toward working with these systems on the terms they actually operate on.

The context window is the most consequential concept in working with language models, and it is the one most users never have explained to them. Almost every frustration people have with LLM behavior, including answers that miss content from the middle of a long document, sudden refusals on long conversations, costs that climb faster than expected, and outputs that feel vague when the input was specific, traces back to the context window and to how the budget inside it is being spent. Understanding what the window is and what shares it is the conceptual foundation for every other decision about how to use these systems well.

This article explains what the context window is, what fills it, why a bigger window does not solve the problem people think it solves, and what to do instead. The throughline is straightforward: the model can only attend carefully to a focused, manageable amount of material at one time, and the practitioner’s job is to give it focused, manageable material rather than crowded, mixed material.

What the context window actually is

A context window is the maximum amount of text a language model can hold in mind during a single call. Think of it as the model’s working memory for one exchange. Anything outside the window does not exist as far as the model is concerned. Anything inside the window competes for the model’s attention.

The size of the window is measured in tokens. A token is roughly three quarters of a word in English; a separate post on tokens covers this in detail. Common context windows in modern models range from around 128,000 tokens up to 1,000,000 or more, depending on the provider and the specific model. In rough terms, 128,000 tokens is somewhere around 300 pages of plain text. A million tokens is several thousand pages. These are large numbers, and the marketing around them encourages the impression that the practical limit is essentially gone. The marketing is misleading. The window is a budget, not a guarantee, and the budget gets spent on more things than people realize.

The diagram above shows the structure of the window. The single rectangle at the top represents the entire budget. Inside it, three things compete for space. The first is the input, which includes everything you and the system have told the model: the system prompt, any documents you attached, prior conversation history, and your latest message. The second is the thinking, which is the model’s internal reasoning when it is using a reasoning mode. The third is the output, which is the response the model generates back to you.

All three live inside the same window. They share the same budget. If the input takes up most of the budget, less is left for thinking and for output. If the model is asked to think extensively, fewer tokens remain for the answer. If a long conversation accumulates, the room available for the next message shrinks accordingly. The total cannot exceed the limit.

This last point is the one most people miss. The window is not just about how much you can put in. It is about how much can fit in for the entire exchange, including the answer the model has not yet generated.

What fills the input side

The input portion of the window is usually the largest, because it carries everything that has been provided to the model up to the moment of the response. Five things commonly sit here.

The system prompt is the instruction set the platform or the developer has given the model about its role and behavior. Even when you do not see it, it is there, and it consumes tokens. In a chat interface this is usually short. In a custom application it can be quite long, especially when it includes formatting rules, tone guidelines, safety constraints, and example outputs.

Attached documents are the most consequential. Every PDF, Word document, image with text, or pasted block of content occupies a share of the window proportional to its size. A long policy document might be ten thousand tokens. A research article might be six thousand. A scanned manual that has been processed into text might be twenty thousand. The total adds up quickly.

Prior conversation history accumulates as the exchange grows. Every message you have sent and every response the model has produced earlier in the conversation is included in the input on the next call, because that is how the model maintains continuity. A long conversation steadily eats into the available budget.

Tool outputs and retrieved content fill the input on every call when the system is doing retrieval-augmented generation, web search, or tool use. The retrieved chunks, the search results, the tool responses, all of it sits in the input window the next time the model is asked to respond.

The latest user message is the smallest piece, but it is still there.

When you upload a stack of documents and ask a question, the question itself is usually a few dozen tokens. The documents are tens or hundreds of thousands. The model is reading and reasoning over all of it at once, and the question is the small wave at the end of an ocean of input.

What fills thinking and output

The thinking side of the budget exists for reasoning models, which are models that produce internal step-by-step reasoning before generating a final answer. The reasoning itself consumes tokens, and the tokens come out of the same shared budget. Models can be configured to use anywhere from a few thousand to tens of thousands of thinking tokens on a single problem, depending on the difficulty. When a reasoning model gives a careful answer to a hard question, much of the work happens in that thinking region, even though the user never sees it. The thinking is real, and it is paid for in tokens.

The output side carries the model’s generated response. There is usually a separate cap on the output, often somewhere between a few thousand and a few tens of thousands of tokens, because runaway generation would otherwise consume the entire budget. When the output cap is reached, the response is cut off mid-sentence. When the input is so large that little space is left, the model produces shorter responses to fit, sometimes truncating its work without explanation.

The single most useful mental model is that the context window is one container with three rooms inside it, and what you put in one room reduces the space in the others.

Why bigger windows do not solve the problem

The temptation is to assume that as windows grow larger, the problem of fitting documents into them disappears. This is mostly wrong, for two related reasons.

The first is that attention is not uniform across the window. A robust finding across many evaluations is that models attend most carefully to material at the beginning and end of the input, and less carefully to material in the middle. This is sometimes called the lost-in-the-middle effect. Drop a hundred documents into a window and ask a question. The model will tend to surface content from the first few and the last few. Documents in the middle of the pile will be underweighted, sometimes ignored entirely, and the model will not warn you that this is happening. The output will look complete. The synthesis it offers will be incomplete in ways that are difficult to detect.

The second is that even when attention is uniform, the model has to weigh many things against each other when the input is large. A focused input lets the model concentrate. A crowded input forces the model to summarize, generalize, and smooth over differences. The result is a vaguer, less specific answer. This is especially apparent in classification, extraction, and analytic tasks, where the precision of the response is what matters. A model classifying one document at a time is reading that document and producing a label. A model classifying a hundred documents at once is producing a list of labels in a single pass, with much less attention to any one of them.

Larger windows extend the upper limit of what fits. They do not fundamentally change how attention works inside the window. The biggest model with the largest window is still better at one focused question over one focused input than at one diffuse question over a hundred mixed inputs.

The classification pipeline as the practical answer

The right way to process many documents with a model is the pattern described in the post on classification at scale. One document per call, in a loop, with a fresh context window each time. Each call has the full budget available for one focused task. The model is not splitting attention across a hundred documents. It is reading one, classifying it, and finishing. The next call starts clean.

This pattern looks like more work than dumping the corpus into a single chat session, but it produces dramatically better results. The output is structured. Each result is traceable to a specific document. Errors are visible at the level of the individual call rather than smeared across the corpus. The cost is predictable, because the per-call cost is fixed. The pattern scales to corpora of any size, because the constraint is no longer the window; it is just how many calls you are willing to make.

The general principle generalizes. When the work involves reasoning over a large body of material, do not try to fit it all into a single context window. Decompose the work into pieces that fit comfortably, run each piece in its own call, and combine the results in code afterward. The model is the engine. The pipeline is what scales it.

Practical implications

A few practical conclusions follow from all of this.

When using a chat interface for a single question over a single document, you are usually fine. The window is large enough for that, and the model’s attention is concentrated on the small amount of material in front of it. This is the use case the chat interface is best at.

When the work involves many documents, do not paste them all in. Process them one at a time, ideally through a structured pipeline. If the work has to happen in a chat interface for some reason, work on one document at a time and start a fresh conversation between documents, so that prior conversation is not crowding the window.

When a conversation gets long and the model starts behaving oddly, missing earlier context, repeating itself, generating shorter responses, the conversation has likely consumed enough of the window that the model is operating with less room than it should have. Starting a new conversation usually fixes this.

When using a reasoning model on a hard problem, leave room for thinking. A long input plus a long output plus extensive reasoning can collide with the budget in ways that produce truncation or worse performance. Smaller inputs let reasoning models reason better.

When designing a custom system, treat the context window as the most expensive resource in the pipeline. Decide what belongs in the window for each call, and what belongs in retrieval, in code, in storage. The window is for the work the model needs to see right now. Everything else lives elsewhere.

Where this leaves the practitioner

The context window is the underlying constraint that shapes almost every decision about how to use language models in serious work. It is not just an upper limit on how much fits. It is a budget shared by everything the model is doing at once, including thinking and producing the output, and the model attends most carefully when the budget is being used carefully. The interface that invites you to drop in a pile of documents is not optimized for accuracy; it is optimized for the impression that everything is possible. The work, in practice, is to keep the window focused, and to scale by repetition rather than by crowding.

A focused window produces a focused answer. A crowded window produces a vague one. Bigger numbers in the marketing materials do not change this, and the practitioner who internalizes the constraint will produce better results with smaller models than the practitioner who ignores it will produce with larger ones.

Benchmarking is the practice of closing that gap. It is the structured process of evaluating whether a particular task that an LLM is being asked to perform is being performed reliably. The question benchmarking answers is the only question that ultimately matters in production work: when this model returns an answer, how often is the answer correct?

This article describes how benchmarking works, why it has to be human-centered, what counts as a benchmark dataset, what to do when the task is subjective, how many items to evaluate, and why this work belongs at the front of any AI project rather than at the end. The throughline is straightforward. The model’s confidence in its own output is irrelevant. Confidence has to be established by comparison against ground truth, and ground truth has to come from humans first.

The human-first principle

Before an LLM can be evaluated on a task, the task itself has to be well-defined enough that humans can perform it consistently. This is not a stylistic preference. It is a logical prerequisite. If two trained people, working independently from the same instructions, cannot produce the same answer for a given input, then there is no ground truth for the model to be measured against. The disagreement between humans is not a methodological inconvenience. It is a sign that the task itself is not well defined.

This is the inter-rater reliability test in its strongest form, and it is the gate every LLM benchmark must pass before the benchmark is meaningful. Take a sample of inputs. Have two trained people, each working independently from the same written instructions, produce outputs. Compare them. Where they agree, the task is well-formed for those inputs. Where they disagree, you have learned something important: the instructions are unclear, the categories overlap, the task is genuinely subjective, or the inputs themselves are ambiguous. Each of these problems has to be addressed before the LLM enters the picture. If the humans cannot agree, the model has nothing to be measured against, and any “evaluation” of the model reduces to looking at its output and deciding whether it sounds plausible.

Benchmarking, in this sense, is not really about the model. It is about the task. The benchmark dataset documents what the task is by exhibiting examples of correct outputs. The model is tested against that documentation. Without the documentation, the model is being asked to do something that has not been specified, and the project rests on faith.

The diagram above shows a small benchmark in operation. The leftmost columns hold the independent classifications produced by two human coders. Where they agree, the gold-standard label is obvious. Where they disagree (row 002 in the example), the coders meet, discuss, and produce a resolved label that becomes the gold standard for that document. The next column shows what the LLM produced. The notes column captures whether the model agreed with the gold standard. The match column tallies the result. At the bottom, two summary statistics matter equally: the inter-rater agreement between the humans (which tells you whether the task is well-defined) and the LLM accuracy against the gold standard (which tells you whether the model is performing it reliably).

Building a gold standard

A benchmark dataset, sometimes called a gold standard, is a small labeled corpus where the correct answer for each item is known. The point of the dataset is to give you a fixed reference against which the LLM’s output can be compared. The work of building one is also the work of figuring out what the task actually is, which is part of why it is worth doing carefully.

For a classification task, the procedure is direct. Take a sample of items from your full corpus. Have two trained coders, each working independently from the same written instructions, classify every item in the sample. Compare their classifications. Where they agree, that classification becomes the gold label. Where they disagree, the coders meet, discuss the disagreement, and either reach consensus or refer the item to a third coder. The goal is not to suppress disagreement. It is to surface it, because each disagreement points to a gap or ambiguity in the task definition that needs to be resolved before the model is trusted with it.

The output of this process is two artifacts. The first is the labeled dataset itself, which becomes the benchmark. The second is a refined version of the instructions, which becomes the basis of the prompt sent to the LLM. The instructions you write at the start are almost never the instructions you end with. The act of trying to apply them produces clarification and revision, and the revised instructions are what should drive the prompt.

For extraction tasks (pulling specific fields out of documents), the same logic applies but the artifact is a list of correct extractions per document. For tagging tasks, the artifact is a set of correct tags per document. For summarization or other generative tasks, the gold standard is harder to construct, because two people will rarely produce identical summaries. This is where subjectivity enters the picture and benchmarking becomes more difficult.

The problem of subjective tasks

Some tasks do not have a single correct answer. Two skilled professionals reading the same case note can produce different but equally defensible summaries. Two judges scoring the same essay on a holistic rubric can land on different but reasonable numbers. Two reviewers labeling the “tone” of a passage can disagree in ways that reflect their training and not their carelessness. The output of these tasks is interpretive judgment, not classification or extraction.

Benchmarking subjective tasks is harder, but not impossible. The trick is that you are not benchmarking against a single correct answer. You are benchmarking against the range of acceptable answers that humans produce. If three trained reviewers give summaries that all hit the same key points but use different language, the model’s summary needs to hit the same key points; it does not need to use the same language. The metric becomes coverage of required elements, or scoring against a rubric, rather than exact match.

The harder version of the problem is when a task is fundamentally subjective and the model’s output is being used downstream as if it were objective. This is a real validity concern that benchmarks alone cannot resolve. If a system treats the model’s classification of “tone” or “intent” as if it were a fact, but the underlying construct is one where humans disagree substantially, then the system is reporting one model’s interpretation as ground truth. The fix is not better benchmarking. The fix is honest documentation of what the field actually represents, what its reliability is, and how it should and should not be used.

The practical heuristic carries through every case: if you cannot get two trained humans to agree, do not pretend the LLM has produced a fact. Subjective outputs can still be useful, but they need to be described accurately as what they are.

How many items to evaluate

A common question is how large the benchmark dataset needs to be. The answer depends on three factors: the number of categories or fields involved, the variation present in the corpus, and the precision required in the estimate of accuracy.

For a binary classification with two roughly balanced categories, fifty to one hundred items will give a reasonable first read on the model’s performance. With a sample that size, it is possible to distinguish, say, 70 percent accuracy from 90 percent accuracy with confidence. It is not possible to distinguish 90 percent from 92 percent.

For a multi-category classification, a useful rule of thumb is that each category should be represented by at least twenty items, and ideally fifty. A four-category scheme therefore needs eighty to two hundred items in the benchmark. This matters because performance varies by category. A model might do well on three categories and poorly on the fourth, and the benchmark needs enough examples per category to reveal that.

For extraction tasks, fewer items can suffice if each item contains many fields to extract. A document with twenty extracted fields supplies twenty data points per document, so fifty documents in that case is one thousand evaluations.

For corpora with high internal variation, the benchmark sample needs to capture the variation. A sample drawn only from one segment of the corpus will tell you about the model’s performance on that segment, not on the corpus as a whole. Stratified sampling, where the sample is drawn proportionally from known subgroups, is appropriate when the subgroups are known and meaningful.

The general principle is that the benchmark should be large enough that real changes in the model’s performance show up above the noise of the metric. A benchmark of ten items gives almost no resolution. A benchmark of one thousand items gives fine resolution but takes substantial human effort to build. The right size is usually somewhere in between, and it is reasonable to start small, build the pipeline, and expand the benchmark as the project warrants.

Iterating on the benchmark

Benchmarking is not a single test that the model passes or fails. It is an iterative process that runs alongside the development of the prompt, the pipeline, and any post-processing logic. The first benchmark run almost always reveals something that needs to change: a category definition that is too vague, a prompt instruction the model is misinterpreting, an output format that produces parsing errors, an edge case that was not anticipated. Each fix gets re-tested against the same benchmark, and the score moves up or down accordingly.

The benchmark dataset itself should be held constant across iterations. If the dataset changes between runs, the comparison becomes unreliable; what looks like model improvement might just be the difference between two datasets. Lock the benchmark in at the start, save it to disk, and re-run the model against it after every meaningful change. The accuracy number that comes back tells you whether the change helped, hurt, or had no effect.

This is the point in the workflow where prompt engineering becomes legible. Without a benchmark, prompt revisions are guesswork. With a benchmark, every revision is testable. Change the wording of an instruction, re-run the benchmark, see whether accuracy moved. Decisions about which prompt to deploy in production can rest on data rather than on impressions.

A good practice is to track multiple metrics, not just overall accuracy. Per-category accuracy reveals which categories are hardest. A confusion matrix reveals which categories are getting confused with which others. Time per call tracks practical viability. Cost per call tracks economic viability. The dashboard built for the benchmark is, in effect, the dashboard for the project, and the metrics that matter are the ones that connect to decisions you will actually make.

Front-loading benchmarking

The most consequential point about benchmarking is when in the project it should happen. The default pattern in AI work is to build the pipeline first and benchmark it at the end, after the prompt is written, the loop is implemented, the data is processed, and the results are reported. This is exactly backwards. By the time the pipeline exists, decisions have already been made about category definitions, output schemas, prompt structure, and processing logic that the benchmark might invalidate. Reworking those decisions after the pipeline is built is expensive and demoralizing, and people often resist doing it.

The right order is to write the benchmark first. Take a small sample of the corpus before any pipeline exists. Have humans label it. Inspect the labels and refine the task definition. Write the prompt. Run the prompt against the benchmark. Iterate until the model is performing acceptably on the benchmark. Then, and only then, scale up to the full corpus. The pipeline that runs at scale should be the same pipeline that performed well on the benchmark, not a different one assembled hastily and validated afterward.

This is what front-loading evaluation looks like in practice. The benefit is that the most consequential decisions, the ones that are most expensive to change later, get tested against evidence early. The cost is the discipline of doing the slow work before the fast work, which is unfamiliar in many AI workflows. The slow work pays for itself within a single project.

For high-stakes work, benchmarking is not optional under any circumstances. If an LLM is being used to produce outputs that affect a person’s case, a patient’s care, a client’s outcome, or a research finding, the model’s outputs need to be benchmarked against human ground truth as a matter of professional responsibility. The fact that the model produces fluent text and looks confident is not evidence that it is correct. The benchmark is.

Where this leaves the practitioner

The throughline is simple. The model cannot evaluate itself. The platform cannot evaluate the model. The tool’s marketing materials cannot evaluate the model. Only a benchmark, built carefully against human-validated ground truth, can establish whether a particular task is being performed reliably. Until that benchmark exists, the output is plausible-sounding text, and no one knows whether it is correct.

This is the work that distinguishes serious AI projects from speculative ones. Building a benchmark is not glamorous. It involves writing instructions, sampling documents, asking colleagues to do tedious labeling, computing agreement metrics, and revising the task definition multiple times. The resulting dataset is small. The resulting accuracy number is one number on one task. But that one number is the foundation on which everything else rests. Every meaningful task an LLM is asked to perform deserves one, and critical decisions deserve nothing less.

The same pattern is happening behind a great many other tools that practitioners use without thinking about. When a citation manager pulls metadata about a paper from CrossRef, that is an API call. When a legal research platform retrieves a docket entry from PACER, that is an API call. When a workflow automation tool moves a file from Google Drive to Box, those are API calls. When a custom RAG system retrieves a document chunk and then asks a model to summarize it, those are two API calls in sequence. Modern software is held together by API calls, and the AI ecosystem is held together by them more tightly than most.

This article explains what an API call is, what travels in the request and what comes back in the response, what authentication and rate limits and pricing actually mean, and which kinds of services that practitioners encounter are accessed this way. The goal is not to teach you how to write the code. The goal is to make the layer beneath the tools you already use legible, so that when you evaluate, commission, or build something new, you can think about it accurately.

What an API is

API stands for Application Programming Interface. The phrase is unhelpful in the abstract, so a better way to think about it is this. Every meaningful piece of software, including the services run by OpenAI, Anthropic, Google, the US government, your university library, and most cloud providers, has two ways to be used. The first is the visual interface, the website or app where a human clicks buttons and reads pages. The second is a programmatic interface, where another piece of software can ask the same service to do the same things, but through a defined set of structured exchanges instead of a visual layout.

The API is that programmatic interface. It is a published contract. The service operator says, in effect, here are the operations we offer, here is how you ask for them, here is what you have to send, and here is what you will get back. Anyone with credentials can write software that follows the contract, and the service will respond accordingly. The visual interface and the API often sit on top of the same underlying system. The website you use is, in many cases, just one client of the same API that other developers can also use.

This is why understanding APIs is useful even if you never write any code. The website is one way to use a service. The API is another. Almost any tool that integrates with a service is using its API. Almost any custom system you might commission is built on top of one or more APIs. Almost any AI workflow that does anything more than answering a single question involves at least one.

The request and the response

The basic shape of an API call is a single round trip. Your code sends a request to the server. The server processes it. The server sends back a response. Both the request and the response are blocks of structured text traveling over the internet, almost always over an encrypted connection called HTTPS.

The diagram above shows the full pattern. On the left is your code. This can be a Python script you wrote, an app installed on your computer, an AI agent acting on your behalf, a workflow tool like Zapier or n8n, or even a chat interface that calls the API for you. Anything that can make a network request can act as a client. On the right is the API server. This can be OpenAI, Anthropic, Google, PubMed, CrossRef, OpenAlex, PACER, CourtListener, Google Drive, Slack, Box, or any of thousands of other services. The arrows in the middle show the two halves of the exchange: the request going out, and the response coming back.

Each half has a defined structure, summarized in the bottom panels of the diagram. A request contains four pieces of information. The endpoint is the specific URL the request is being sent to. Each operation a service offers has its own endpoint, and the URL identifies which one you are calling. The method is the type of operation, drawn from a small set of conventions: GET to retrieve, POST to send or create, PUT to update, DELETE to remove. The headers are short labeled fields that carry metadata about the request, the most important of which is the API key that proves you are authorized to make the call. The body is the actual data you are sending, almost always formatted as JSON. When you send a chat message to a language model, the message is in the body.

A response also contains a small set of components. The status code is a three-digit number that summarizes what happened: 200 means success, 401 means you are not authenticated, 429 means you have hit a rate limit, 500 means something went wrong on the server. The headers carry metadata about the response, including information about how many requests you have left in your current rate limit window. The body contains the result, again almost always formatted as JSON. When the model finishes generating its answer, the answer is in the body. Your client code reads it and does whatever it does next: displays it, saves it, passes it to another step in the workflow.

The whole exchange typically takes anywhere from a few hundred milliseconds for a simple query to several seconds for a long language model response. Multiply that across the dozens or hundreds of calls a complex workflow can make, and you start to understand why latency, retries, and error handling are real concerns in any system built on APIs.

Authentication and API keys

Almost every API requires authentication. The service has to know who you are, partly to enforce permissions and partly to bill you correctly. The dominant mechanism for non-interactive software is the API key, a long opaque string that functions as a password for your account. You generate a key in the service’s website, copy it, and store it somewhere your code can read it. Every request your code makes includes the key in one of the headers, and the server uses the key to look up which account is calling and what that account is allowed to do.

API keys are sensitive. Anyone who has the key can make requests as if they were you, including requests that cost money or that touch private data. Treat keys the way you would treat a password, and ideally a password that gives access to a credit card. Keys should never be embedded in code that gets shared or committed to a public repository. They should be stored in environment variables, in dedicated secrets managers, or in encrypted configuration files. When a key is exposed, it should be rotated, which means invalidating the old key and generating a new one.

Some services use slightly more elaborate schemes. OAuth, the authorization protocol behind login flows like “sign in with Google,” is used when a service needs to act on behalf of a specific user without holding that user’s password. Bearer tokens are short-lived credentials issued after an authentication step. Service accounts are credentials issued to non-human users for backend automation. The details vary, but the underlying logic is the same: the request has to carry proof of identity, and the server has to verify it before doing the work.

Rate limits and cost

Two practical constraints shape how APIs are used in practice. The first is the rate limit. Almost every service caps how many requests a single account can make in a given window of time, sometimes per second, sometimes per minute, sometimes per day. The cap exists to prevent abuse, to manage server load, and to enforce pricing tiers. When you exceed the cap, the server returns a 429 status code instead of doing the work, and your code has to wait and retry. Building a workflow that respects rate limits is part of building a workflow that runs reliably.

The second is cost. Most useful APIs are billed, sometimes per call, sometimes per unit of data, sometimes per token in the case of language models. The costs are individually small and aggregate quickly. A single call to a frontier language model might cost a fraction of a cent. A pipeline that processes ten thousand documents might cost tens or hundreds of dollars. A poorly designed automation that loops without a stopping condition can cost meaningfully more than that before anyone notices. Pricing pages on API services exist for a reason. Reading them before building anything that runs at scale is part of the work.

This is also one of the reasons that local, open-weight models matter. A language model running on your own computer does not require an API call at all. The request never leaves the machine, no key is needed, no rate limit applies, and no per-token billing accumulates. For sensitive data and for high-volume workflows, this can be the difference between a viable system and an unviable one. The tradeoff is that the local model is whatever model you can run on the hardware you have, which is usually smaller and slower than the frontier hosted models. Knowing where the line falls between calls that should go out to a hosted API and work that should stay local is one of the practical judgments that practitioners building AI systems have to make.

Categories of APIs that matter to practitioners

The set of services accessed through APIs is enormous, but a few categories matter directly to social work and legal practitioners. The list below is illustrative rather than exhaustive.

Language model APIs are the most familiar. OpenAI, Anthropic, Google, and the smaller providers all offer APIs that let you submit a prompt and receive a generated response. OpenRouter is an aggregator that exposes most of these models behind a single API and a single key, which is useful for development and for situations where you want to switch models without rewriting code. Every chat interface you have used is wrapping an API of this kind.

Document processing APIs convert files from one form to another. Services like Mathpix and various OCR providers convert scanned documents to text. Services like Adobe Document Services and several open-source equivalents convert PDFs to structured representations. These APIs sit at the front of any pipeline that has to ingest documents that did not start out as plain text.

Academic and research APIs are essential for any work involving the scholarly literature. PubMed offers an API to query the biomedical literature. CrossRef provides bibliographic metadata for nearly every published article with a DOI. OpenAlex offers a more comprehensive open index of scholarly works, citations, authors, and institutions. ORCID provides researcher identifiers. Semantic Scholar offers an API for paper search and citation analysis. A literature review tool that does anything more than text search is using one or more of these.

Legal and court APIs are increasingly important and unevenly available. PACER provides federal court docket and document access, with its well-known constraints on cost and ergonomics. CourtListener, run by the Free Law Project, offers a more developer-friendly API to a substantial body of federal case law and dockets. Various state court systems provide APIs of varying quality. Westlaw and Lexis offer commercial APIs for their large legal research products. The CAP (Caselaw Access Project) database from Harvard provides historical case law. Tools that automate legal research, build litigation timelines, or extract data from court records depend on these.

Government and public data APIs are a substantial and underused category. Data.gov aggregates federal datasets with API access. The Census Bureau offers extensive demographic APIs. The CDC, NIH, and HHS provide health data APIs. The Department of Health and Human Services offers various administrative data interfaces. State governments offer increasing numbers of open-data APIs for everything from licensing data to inspection records. For social work researchers and policy analysts, these are where much of the underlying data of practice actually lives.

Cloud storage and document APIs allow programmatic access to files stored in Google Drive, Microsoft OneDrive, Dropbox, Box, and similar services. These APIs are how integrations move documents into and out of cloud storage, how RAG systems ingest organizational corpora, and how automated workflows save outputs.

Communication and productivity APIs connect to the tools where work happens. Slack, Microsoft Teams, Gmail, Outlook, Google Calendar, and similar services all expose APIs. Workflow automation, AI assistants that read or send messages, and integrations of any kind use them.

Translation, geocoding, and specialized data APIs fill in specific needs. Google Translate and DeepL provide translation through APIs. Google Maps, Mapbox, and OpenStreetMap provide geocoding and routing. ClinicalTrials.gov provides trial registration data. The list extends in every direction the work extends.

The practical observation across all of these is that almost any external system you interact with for work has an API behind it. The visual interface is one front door. The API is the other.

Why this matters for AI workflows

Two implications follow from all of this for anyone building or commissioning AI systems on top of organizational work.

The first is that an AI workflow is rarely a single API call. It is a small graph of them. A typical RAG system involves at least three: an embedding call to convert the user’s question into a vector, a database call to retrieve matching document chunks (often itself an API call), and a model call to generate the final answer over the retrieved chunks. A document processing pipeline might add an OCR call, a PDF-to-Markdown call, and a metadata extraction call before any of that happens. A research synthesis tool might add a CrossRef call, an OpenAlex call, and a PubMed call. Each call has its own latency, its own cost, its own failure mode. Building a system that runs reliably means thinking about all of them, not just the model call at the end.

The second is that the format of what travels in the request and response is structured data, almost always JSON. This is why the post on plain text formats matters here, and why JSON in particular deserves its own treatment. The model does not receive your documents in their original form. It receives JSON-shaped text, prepared by whatever code you or your tool has written to package the prompt. The quality of the preparation determines the quality of the result. This is also why the conversion of source documents to clean Markdown matters: the cleaner the input, the cleaner the JSON, the better the model’s output, the more useful the response that comes back through the same arrows the diagram shows.

Where this leaves the practitioner

You do not need to write API code to be fluent in what an API is. You need to know that every external service your tools touch is being touched through an API, that every call carries an authenticated request and returns a structured response, that calls cost money and have rate limits, and that the AI ecosystem is built on this pattern from end to end. With that mental model in place, conversations about what a tool does, what data it sends out, what data it brings back, and what it costs to run become specific and grounded rather than vague.

The chat interface is the simplest version of all of this. You type a question. The interface sends an API call. The response comes back. You see the answer. Every more complex system you build or commission is the same pattern, repeated and arranged. The arrows do not change. What changes is what the arrows carry, and what the practitioner has decided to do with it.

For casual one-off questions, this is acceptable. For any system you are building seriously, where the same documents will be queried repeatedly, where the answers need to be reliable, or where the corpus needs to be analyzed at scale, the conversion step is too important to leave to chance. The standard practice in serious AI work is to convert source documents into Markdown once, inspect the result, fix what needs fixing, and then build the rest of the system on top of those clean Markdown files. The conversion process is its own topic and deserves a separate post. This article is about what Markdown is, why it works the way it does, and why it has become the canonical preparation format for AI work.

The argument is structural. Markdown is a plain text format that captures the logical organization of a document, headings, paragraphs, lists, tables, emphasis, and links, using a small set of punctuation conventions. The structure is encoded in the text itself, in a way that humans can read and machines can parse without ambiguity. That property, structure-as-text, is what makes Markdown the format AI systems handle most reliably. Understanding why requires looking at what Markdown actually is and what its tags are doing.

What Markdown is

Markdown is a lightweight markup language designed in 2004 by John Gruber, with substantial input from Aaron Swartz, with a single goal: to let people write text that reads naturally as plain text but can be converted into formatted output, primarily HTML, by a parser. Before Markdown, formatting plain text either meant writing raw HTML, with all of its angle brackets and closing tags, or using one of several earlier conventions for marking emphasis and structure that had grown up informally over decades of email and Usenet posts. Markdown formalized those conventions into a small, consistent grammar.

A Markdown file is a plain text file. The contents are characters. There is no hidden formatting, no embedded objects, no proprietary structure. By convention the file uses a .md or .markdown extension, but as discussed in the post on plain text, the extension is a label. The bytes inside a .md file are the same kind of bytes as in any other plain text file. What distinguishes a Markdown file is that the characters inside follow the conventions that signal structure to a reader and to a parser.

The conventions are deliberately minimal. A line that begins with one or more pound signs is a heading. A line that begins with a hyphen is a list item. Text surrounded by single asterisks is italicized. Text surrounded by double asterisks is bold. A line that begins with a greater-than sign is a blockquote. Text wrapped in backticks is treated as code. A blank line ends a paragraph. These rules can be learned in fifteen minutes and remembered indefinitely. The aesthetic principle behind the design was that the markup should be unobtrusive enough that the document remains pleasant to read in raw form, even by people who have no idea what Markdown is.

The result is a format that occupies a useful middle ground. Plain text without structure is universally readable but has no way to express headings or lists or tables in a form that downstream tools can act on. HTML expresses everything but is verbose and hostile to read in raw form. Markdown lands between the two. It looks like ordinary writing with a few small marks, and it parses cleanly into a structured document tree.

What the tags are doing

The conceptual move at the heart of Markdown is that the tags do not describe what the text should look like. They describe what the text is. A pound sign at the start of a line does not say “make this bigger and bolder.” It says “this is a top-level heading.” The visual representation, font size, weight, color, spacing, comes later, when something renders the document. Different renderers can produce different visual results from the same Markdown source. The structure stays the same.

This distinction between semantic markup and visual formatting is the central reason Markdown is useful for machine processing. A Word document or a PDF generally records visual choices. The font is twenty-four point. The text is centered. The color is dark blue. From these visual choices, a parser has to infer what the structural intent was. A line that is twenty-four point and bold and centered was probably intended to be a top-level heading, but the document does not say so directly. Visual heuristics are required, and they fail in exactly the cases where the formatting is unusual or inconsistent.

Markdown reverses the relationship. The structure is stated explicitly, and the visual representation is whatever the renderer decides. A heading is a heading because the document says so, not because of how it looks. A list item is a list item because the document says so. The text “important” inside double asterisks is bold because it has been marked as emphasized, not because someone reached for the bold button. When a parser, or a model, or any downstream tool reads the document, the structure is unambiguous. There is no inference step. The intent is in the text.

This is what people mean when they call Markdown a structured format. The structure is not an interpretation. It is a declaration.

The full set of common elements

The following elements are part of the original Markdown specification or part of the widely adopted GitHub Flavored Markdown extension. Together they cover almost everything a document needs. Each element is a tag, in the sense that it marks a region of the document as being a particular kind of thing.

Before working through the elements individually, it helps to see them in two views. The first shows the raw Markdown source, the characters as they appear in the plain text file. The second shows the rendered output, what a reader sees after the file has been processed by a Markdown parser. The same content appears in both. The difference is the representation.

The raw view. This is what the Markdown file actually contains. Pound signs mark headings, asterisks mark emphasis, hyphens mark list items, square brackets and parentheses mark links, backticks mark code, and so on. Every character shown here is literal text in a plain text file. It can be opened in any text editor and edited by hand. There is no hidden formatting. The structure is declared by the tags themselves.

The rendered view. This is what a reader sees after a Markdown parser has processed the file and produced formatted output, typically HTML. The pound signs are gone, replaced by typographically styled headings. The asterisks are gone, replaced by bold and italic text. The hyphens are gone, replaced by bullet points. The brackets and parentheses are gone, replaced by a clickable hyperlink. None of this styling is in the source file. It is applied by the renderer at the moment of display.

The point of showing the two views together is to make the central property of Markdown visible. The structure of the document is in the source. The styling is added at render time. The same source can be rendered with different styles, by different tools, on different platforms, and the structural relationships, what is a heading, what is a list, what is a link, remain identical. This is what makes Markdown durable as a working format and reliable as an input to any system that reads documents.

With the two views in mind, the individual elements are easier to read.

Headings are lines that begin with one to six pound signs followed by a space, with the number of pound signs indicating the level. A single pound sign is the top-level heading; six pound signs is the deepest. Headings establish the hierarchical outline of the document. Almost every other structural property of a Markdown file follows from how the headings are arranged.

Paragraphs are runs of text separated by blank lines. The blank line is the entire syntax. Wrap a paragraph however you like across lines or write it as a single long line; the parser treats it as one paragraph until it sees a blank line.

Emphasis is marked with surrounding characters. Single asterisks or underscores produce italics. Double asterisks or underscores produce bold. Triple asterisks produce both. Double tildes, in the GitHub Flavored variant, produce strikethrough. Emphasis applies to the text between the markers.

Unordered lists are lines that begin with a hyphen, asterisk, or plus sign followed by a space. Each line becomes a list item. Indentation creates nested sub-lists, with the convention that two or four spaces denote a level of nesting depending on the parser.

Ordered lists are lines that begin with a number, a period, and a space. The numbers themselves do not have to be sequentially correct in the source; most parsers renumber them when rendering. This is intentional, because it allows the writer to insert or rearrange items without renumbering the whole list by hand.

Links are written as [link text](url). The square brackets contain the text the reader sees, and the parentheses contain the destination. There is also a reference style, where the bracketed text is matched to a definition placed elsewhere in the document, which keeps long URLs out of the body of the prose.

Images are written as , with the leading exclamation point distinguishing them from links. The alt text is the textual description of the image, used for accessibility and for situations where the image cannot be displayed. In Markdown files prepared for AI systems, the alt text is often the only representation of the image the model will see, which makes it more important than people sometimes realize.

Inline code is text wrapped in single backticks. It is rendered in a monospaced font and treated as literal characters, not as Markdown syntax. This is useful for referring to file names, command names, function names, or any string that should not be interpreted.

Code blocks are larger sections of code, marked with three backticks on their own line at the beginning and end. An optional language identifier after the opening backticks tells the renderer how to apply syntax highlighting. The contents of the code block are treated as literal text, even if they contain characters that would otherwise be Markdown syntax.

Blockquotes are lines that begin with a greater-than sign followed by a space. They are typically rendered with an indent and a vertical bar in the margin. Nested blockquotes use multiple greater-than signs.

Tables, in the GitHub Flavored extension, are written using vertical bars to separate columns and a row of hyphens to separate the header from the body. The visual alignment of the bars in the source does not have to be perfect; the parser handles whitespace gracefully. Tables in Markdown are limited compared to the tables in a word processor, but for tabular data they are perfectly serviceable.

Horizontal rules are produced by a line containing three or more hyphens, asterisks, or underscores. They are used to separate sections of a document where a heading is not appropriate.

Footnotes, in some Markdown variants, are written using a reference notation similar to links, with the footnote text defined separately in the document. Pandoc Markdown and several other extended dialects support footnotes; the original specification did not.

HTML passthrough is a feature of Markdown that is worth mentioning. Anywhere Markdown does not give you a way to express what you need, you can fall back to writing literal HTML, and most parsers will pass it through to the rendered output. This is an escape hatch, and it should be used sparingly, but it means that Markdown is never a strict ceiling on what the document can contain.

The complete set of elements is small. A writer can hold all of it in mind. A parser can handle all of it deterministically. A model trained on web text has seen all of it billions of times.

Why this format suits AI work

The argument for Markdown in AI workflows has several interconnected parts, and they are worth working through individually because each one matters on its own.

First, the structure is preserved through the conversion. When a PDF or a Word document is processed into plain text, the structural information, what was a heading, what was a list, what was a table, is usually lost or attenuated. The text comes out as one undifferentiated stream. A parser can sometimes reconstruct headings from font sizes, but the reconstruction is fragile. Markdown does not require reconstruction. The structure is in the text. A heading is still a heading after every step of the pipeline.

Second, the structure is queryable. Once a document is in Markdown, a script can find every section, every list, every table, every code block, every link, by looking for the corresponding markup. The same script applied to plain text would have to guess. The same script applied to a PDF would have to depend on the original parser’s interpretation of the visual layout. Markdown allows direct, deterministic access to the parts of the document.

Third, the structure aligns with how retrieval-augmented generation systems chunk documents. Almost every RAG pipeline divides a long document into smaller chunks before indexing. The naive approach is to split on a fixed character count, which produces chunks that often begin and end mid-sentence and that mix unrelated topics. A more thoughtful approach is to split on structural boundaries: at headings, at major section breaks, at paragraph boundaries when no heading is available. Markdown makes those boundaries explicit. A chunker that respects Markdown headings produces chunks that correspond to coherent sections of the document, which produces better retrieval, which produces better answers. The quality of the chunks is the quality of the system.

Fourth, language models have seen a great deal of Markdown during training. The web is saturated with it. Every README file on GitHub is Markdown. Most documentation sites are generated from Markdown. Stack Overflow posts use a Markdown variant. A large fraction of the technical writing the model encountered while training was written in this format. The result is that Markdown is, in a real sense, a format the model speaks fluently. When a model produces output, it produces Markdown by default. When a model receives input in Markdown, it parses the structure correctly and uses it. The match between input format and model expectations is part of why the format performs as well as it does.

Fifth, Markdown is diffable and versionable. Two Markdown files can be compared line by line. Changes to a single character are visible. Version control systems handle Markdown perfectly. This matters when a corpus is being maintained over time, when documents are being edited collaboratively, or when an audit trail is required. None of these properties hold for binary formats in any practical sense.

Sixth, Markdown renders to almost anything. The same source file can be converted to HTML for the web, to PDF for printing, to a Word document for distribution, to a slide deck through tools like Marp, to a static website through tools like Jekyll, Hugo, or MkDocs, to an academic paper through Pandoc or Quarto. The rendering is downstream of the source. The source remains the canonical version. This is the inverse of how most institutions handle documents, where the Word file or the PDF is treated as the master and any other version is a derivative. Treating the Markdown as the master and rendering to other formats as needed is a more durable arrangement, particularly for content that will be queried by AI systems, which will always prefer the structured source over the rendered output.

The conceptual foundation

The deepest reason Markdown is useful is that it forces a particular kind of clarity about what a document is. A document, in the structural sense, is a tree. The top is the document itself. Below the top are major sections, each marked by a top-level heading. Below each major section are subsections, each marked by a deeper heading. Below the subsections are paragraphs, lists, tables, and code blocks, the leaves of the tree where the actual content lives. The tree is what gives the document coherence. A reader navigates the tree to find what they are looking for. A model can do the same when the tree is explicit.

Many documents that exist in binary formats have lost their tree, or never had one. A PDF that consists of scanned pages has no tree at all; it is a stack of images of paper. A Word document with inconsistent heading styles has a corrupted tree, where what looks like a heading visually is just bolded body text without the heading designation. Converting these documents to Markdown forces the question of what the tree actually is. The act of producing clean Markdown is, in part, the act of reconstructing the document’s structural intent. When the conversion is done well, the result is a document whose organization is recoverable not just by the human eye but by any program that reads the file.

This is also why Markdown is the right preparation format for bespoke systems. A bespoke system, by definition, is one you have built yourself for a particular corpus and a particular purpose. The behavior of the system depends on the quality of its inputs. When you control the format of the inputs, you can be specific about what the system can rely on. You can write a chunker that splits on level-two headings because you know your documents have level-two headings. You can write a retrieval index that prioritizes section titles because you know the section titles are explicit. You can write evaluation tests that examine specific tables because you know the tables are addressable. None of this is possible when the inputs are PDFs and the conversion happens behind the scenes on every query. The work of preparing documents into Markdown is the work of giving your system something to stand on.

Where this leaves the practitioner

Markdown is a small format. You can learn the syntax in an afternoon. You can write a Markdown file in any text editor. You can read one without any tools at all. The technical surface area is intentionally minimal, which is why it has lasted for two decades and why it has spread to nearly every part of the technical ecosystem.

Its importance is out of proportion to its complexity. Markdown is the format that documentation is written in, that AI models were trained on, that retrieval systems chunk, that version control systems track, and that nearly every modern technical workflow assumes at some layer. For practitioners building AI systems on top of organizational documents, Markdown is the format the source materials should end up in before anything else is built. The conversion from PDF or Word into Markdown is its own work, with its own decisions and its own pitfalls, and that work deserves separate treatment. But the conceptual point is settled. The format that makes the rest of the pipeline tractable is Markdown, and the reason it does so is that Markdown carries the structure of a document in the text itself, where every program, every model, and every reader can see it.

The motivation is simple. Almost every AI application built on top of an organization’s existing materials, whether a retrieval-augmented chatbot, a search interface, a classification pipeline, or an analytic dashboard, depends on a layer of structured information that sits between the raw documents and the model. That layer is metadata. If the metadata is thin, inconsistent, or absent, the downstream system will struggle no matter how capable the underlying model is. If the metadata is thoughtful and well-structured, even modest models can do useful work.

What metadata is in this context

Metadata is data about data. The textbook definition is correct but unhelpful for a practitioner. In the context of AI applications built on document collections, metadata is the structured set of fields you attach to each document so that the document can be located, filtered, grouped, summarized, and reasoned about without having to read the document in full each time.

A legal brief has a caption, a court, a docket number, a filing date, a party list, a procedural posture, and a set of issues. A scientific article has authors, an affiliation list, a publication year, a journal, a DOI, a methodological orientation, a population studied, and an outcome focus. A policy document has an issuing body, an effective date, a jurisdiction, a topic area, and a relationship to prior policy. A set of meeting minutes has a date, a body, attendees, agenda items, decisions made, and action items assigned. Each of these is metadata. None of it is the document itself. All of it is what makes a collection of documents searchable as a corpus rather than as a pile.

The point of generating metadata is to take an unstructured or semi-structured collection and produce a structured representation that can sit in a database or a flat file and answer questions deterministically. How many briefs were filed in the Eastern District in 2023. Which articles in the corpus involve administrative data. Which faculty meetings discussed the new evaluation policy. None of these questions should require an LLM to answer. They should be answered by querying a well-built metadata table.

What metadata actually looks like

The abstraction becomes much easier to grasp once you see the structure. A metadata record is, at its core, a list of named fields with values. Below are realistic examples for four common document types. The exact fields depend on what you intend to do with the corpus, and a real project would refine these substantially. The point here is to show the form.

A scientific article record might look like this:

{

  "doi": "10.1080/02615479.2023.2189456",

  "title": "Predictors of foster care reentry among adolescents",

  "authors": ["Smith, J.", "Lee, K.", "Anderson, R."],

  "journal": "Children and Youth Services Review",

  "year": 2023,

  "volume": 145,

  "pages": "106789",

  "study_design": "secondary_analysis",

  "data_source": "state_administrative",

  "population": "adolescents_in_foster_care",

  "open_access": true

}

A state policy document record might look like this:

{

  "document_id": "FFPSA-MI-2022-001",

  "issuing_body": "Michigan Department of Health and Human Services",

  "jurisdiction": "Michigan",

  "title": "Family First Prevention Services Act State Plan",

  "effective_date": "2022-10-01",

  "supersedes": "FFPSA-MI-2021-003",

  "topic_area": "child_welfare_prevention",

  "page_count": 87,

  "summary": "State plan describing evidence-based prevention services for candidates for foster care, including service array, evaluation strategy, and workforce supports."

}

A faculty meeting minutes record might look like this:

{

  "meeting_id": "FAC-2024-09-14",

  "body": "School of Social Work Faculty Meeting",

  "date": "2024-09-14",

  "attendees_count": 32,

  "agenda_items": ["budget update", "curriculum review", "promotion guidelines"],

  "decisions": ["approved revised PhD comprehensive exam policy"],

  "action_items": ["curriculum committee to circulate draft by 10/15"],

  "summary": "Discussion centered on revised promotion guidelines and the proposed comprehensive exam policy, which was approved with minor amendments."

}

A legal brief record might look like this:

{

  "case_caption": "People v. Johnson",

  "court": "Michigan Court of Appeals",

  "docket_number": "365432",

  "filing_date": "2024-03-15",

  "filing_party": "appellant",

  "procedural_posture": "appeal from summary disposition",

  "issues": ["due process", "evidentiary ruling", "ineffective assistance"],

  "page_count": 42,

  "summary": "Appellant challenges the trial court's grant of summary disposition, arguing improper exclusion of expert testimony and ineffective assistance of trial counsel."

}

Several things are worth noticing about these examples.

First, every record uses the same structure within its corpus. Every scientific article record has the same fields. Every policy document record has the same fields. The structure is what makes the collection queryable. A field that exists for some documents and not others is either a sign that the corpus is mixed or a sign that a missing value should be recorded explicitly rather than left out.

Second, the fields are short and discrete. Authors are a list of names. The year is a four-digit integer. The jurisdiction is a single value drawn from a controlled list. Issues are a list of short tags. The summary is a single paragraph. Nothing in the metadata is the document itself. The metadata is meant to be small and fast to query.

Third, the fields differ by document type because the questions you would ask differ by document type. You ask about journals, populations, and study designs in a scientific corpus. You ask about jurisdictions, effective dates, and superseding relationships in a policy corpus. You ask about courts, dockets, and procedural posture in a legal corpus. Designing the metadata schema is partly an exercise in anticipating the questions the corpus will need to answer.

Where metadata lives

Metadata can be stored in several formats. The format you choose depends on how you plan to use it, but the underlying content, the field-value pairs, is the same across formats.

JSON is the dominant format for AI applications and for any workflow that involves nested fields, lists, or programmatic access. The records above are written in JSON. Every record is an object containing fields and values, and a corpus is typically stored as a list of such objects or as one JSON file per document. JSON handles lists (multiple authors, multiple agenda items) and nested structures (parties with sub-fields) naturally, which is why it has become the default for AI pipelines. A subsequent post will cover JSON in more depth.

CSV is appropriate when every field is simple and the metadata is genuinely tabular. A corpus of scientific articles where every record has a title, year, journal, and DOI fits cleanly into a CSV. The moment you need lists, nested objects, or variable-length fields, CSV starts to strain. Workarounds, such as comma-separated values inside a single cell or multiple rows per document, undermine the very querying benefits the format is supposed to provide.

Database tables are the right home for metadata that will be queried frequently or by multiple applications. A relational database, including lightweight options like SQLite, gives you indexing, fast filters, and the ability to join metadata with other tables. Most production systems that begin with JSON or CSV metadata eventually migrate to a database when query load grows.

YAML appears occasionally, usually for metadata that is hand-edited or read by humans alongside machines. It is more forgiving to write than JSON but is rarely the right choice for large generated corpora.

Document-embedded metadata also exists, in the form of PDF document properties, Word file metadata, EXIF tags on images, and so on. This is real metadata, but it is generally a starting point rather than a destination. For AI applications, you will almost always extract embedded metadata into one of the structured formats above so that it can sit alongside the metadata you generate yourself.

The format is a logistical choice. The fields, their definitions, and the consistency of their application across the corpus are the substantive choices. A well-designed metadata schema can be ported between formats with minimal loss. A poorly designed one will be useless in any format.

Extraction versus inference

The conceptual move that matters most is the distinction between metadata that can be extracted directly from a document and metadata that requires inference.

Extraction is straightforward. The author names are on the first page. The filing date is in the caption. The journal title is in the citation. The agenda is at the top of the minutes. When a model is asked to extract this kind of metadata, the task is to locate and copy text that already exists in the source. Reliability is high. Validity is largely a matter of whether the model has correctly identified the boundaries of the field. Errors tend to be specific and detectable, and they can usually be corrected with better prompting, better preprocessing, or fallback rules.

Inference is a different kind of task. If the metadata field is something like “primary methodological approach” or “policy domain affected” or “tone of the deliberation,” the model is no longer copying text. It is reading the document and producing a judgment that may or may not appear anywhere in the source verbatim. This work can still be useful, but it is a categorically different task with categorically different evaluation requirements. Inferred fields need reliability and validity analyses appropriate to classification or labeling work. They need ground-truth comparisons, inter-coder agreement checks, and explicit decision rules. They cannot be treated as if they were extractions even when they end up in the same JSON object.

The trouble starts when these two kinds of work are blurred. A pipeline that mixes extraction fields and inference fields without acknowledging the distinction will produce a metadata table that looks uniform but is not. Some of the fields will be near-deterministic and some will be classification outputs with unmeasured error rates. Downstream queries against that table will return results that appear authoritative but are partly the product of unvalidated inference. The fix is not to avoid inference. The fix is to label the two kinds of fields differently, evaluate them differently, and document their different epistemic statuses.

There is a gray area, and it is worth naming. Some fields look like extractions but require a small amount of normalization or interpretation. A jurisdiction field might be extracted from the caption but standardized against a controlled list. A publication year might be extracted but reconciled across the print and online versions. The summary fields shown in the examples above are inferred, not extracted, even though they may appear to live comfortably alongside the extracted fields. These cases sit at varying distances from pure extraction, and each introduces decisions that should be documented and tested. Treating them carelessly is how downstream errors compound.

The principle of a single table

Metadata generation only works when the collection is coherent. The simplest way to think about this is to imagine that the output will populate a single table in a database. Every row is a document. Every column is a field. Every document in the collection has the same fields, defined the same way, generated by the same process.

This means that a metadata workflow should not mix fundamentally different kinds of documents. A collection of state policy documents can be processed together. A collection of case notes can be processed together. A collection of conference abstracts can be processed together. The fields appropriate for each will overlap in places and diverge in others, but each is a coherent corpus on its own terms. Trying to build a single metadata pipeline that ingests case notes and policy documents and meeting minutes in one pass produces a table whose columns are either too generic to be useful or too sparse to support querying.

The discipline here is conceptual before it is technical. Before any extraction or inference is run, the practitioner has to decide what the corpus is, what the unit of analysis is, and what fields are meaningful for every document in the corpus. If a field is meaningful for some documents and not others, the corpus probably contains more than one collection, and they should be processed separately and joined later if needed.

Why retrieval is the central use case

The primary reason to invest in metadata is retrieval. When a user asks a question of a document collection, the worst possible answer is one that runs the full text of every document through an LLM and hopes that the right material surfaces. That approach is slow, expensive, environmentally costly, and unreliable on collections of any meaningful size. A well-built metadata layer makes most retrieval queries deterministic. Filter by jurisdiction. Filter by date range. Filter by topic tag. Return the matching documents. The LLM is reserved for the work that actually requires it, which is usually summarization or synthesis over a small, already-relevant set of documents.

Metadata can also include short generated summaries as fields, as the examples above illustrate. A two-sentence summary of each policy document, attached as metadata, allows a search interface to scan summaries rather than full texts. Summaries are inferred fields and need to be evaluated as such, but once produced and validated, they make retrieval dramatically faster and more useful. The pattern is consistent: structured metadata at the front, full-text or vector search behind it, model reasoning at the end and only over a manageable subset.

The economic and environmental argument matters here as well. An LLM call against every document in a corpus, on every query, is wasteful by every measure. Metadata-first retrieval pushes the model to the smallest part of the workflow where it adds genuine value. The rest is database work. That is the right division of labor.

Where this leaves the practitioner

Before building any AI application on top of a document collection, decide what the collection is, what fields are meaningful across every document in it, which of those fields can be extracted and which require inference, what format will hold the records, and how each kind of field will be evaluated. Once those decisions are made, the metadata layer follows. Once the metadata layer is in place, the rest of the system has something to stand on.

Most projects skip this step and pay for it later. The chatbot underperforms. The search returns nothing useful. The dashboard double-counts records. None of these failures are model failures. They are failures of the layer beneath the model, and that layer is metadata.

Two clarifications about my teaching philosophy. First, while I understand the interest for publishing papers, that is not the focus of my teaching. Doing good research is what makes publishing easy. My approach to AI will be about solving problems, supporting learning, and creating effective research workflows. Second, the goal is not to outsource thinking to AI. The opposite. AI can make tasks easier and faster – in doing so, this frees up space for expert thinking.

The framing throughout is practical. Each post will address a foundation concept, often within the context of an actual skill or activity. Building this type of understanding makes everything else easier. Everything I cover will be things that I actually use to build solutions.

Here is the rough plan for the series. Topics may shift based on what readers ask for, and the order is not fixed.

• Prompt engineering, context management, and harness engineering. These three terms are often used interchangeably and they should not be. Prompt engineering shapes what you ask. Context management shapes what the model sees alongside the prompt. Harness engineering shapes the surrounding system that runs the model and orchestrates the work. Knowing where you are working is essential.

• The context window. What it is, why it matters, why “the model has a long memory” is a misleading way to describe it, and how to think about what fits inside it for a given task.

• Giving models useful “skills.” Skills are reusable capabilities that extend what a model can do reliably for specific tasks. The quotation marks are deliberate; the term is imprecise but useful.

• Right-sizing models. Larger is not better by default. Choosing the smallest model that performs the task at the required quality is a discipline, not a compromise.

• APIs, MCPs, and CLIs. Three different ways of connecting models to other systems, with three different sets of tradeoffs. Understanding the differences clarifies what your tool should and should not try to do.

• Curating data. The corpus is the system. Document quality, scope, and organization determine whether anything built on top of the data will work.

• Benchmarking. How to evaluate whether a model or workflow is actually doing the job, with methods that go beyond impressionistic testing.

• NLP tasks. Classification, extraction, and other classic tasks where small, well-tuned models often outperform general-purpose chatbots, and where the structure of the task matters more than the brand of the model.

The series will also work through agentic systems, including OpenClaw, which I use in my own daily workflow.

If you are coming to one of the workshops in China, or following along from elsewhere, write to me with topics you want covered first or in more depth. The list above is a starting point, not a syllabus.

The critiques of AI are familiar by now. Large language models hallucinate. They flatter users rather than challenge them (sycophancy). They were trained on scraped work without consent. They consume water and electricity at scale. They sit atop exploited annotation labor. They route private data through the servers of massive corporations with little meaningful oversight. They encode and amplify disparities that social work has spent decades trying to name.

Each of these points contains something true. I have written about several of them myself. I am not asking anyone in the field to pretend the harms are not real.

I want to open a different question, and I have not yet seen the field engage it directly.

If these are the standards you are putting forward, are they fixed standards? Is the position a complete moratorium on AI in social work practice, research, and education? Or are there conditions under which use is acceptable, and if so, what are those conditions, and who gets to decide?

I ask because the current discourse collapses two very different things. One is an ideological stance against a category of technology. The other is a strategic framework for deciding when, how, and for whom a tool is appropriate. These are not the same, and the field needs to be honest about which one is actually on the table.

Consider four cases. I want to be plain about something before I lay them out: these are not hypotheticals constructed to win an argument. All four are drawn from work I am actively doing or have done, with real collaborators and real populations. I raise them because I have not found a principled way to resolve them inside the concerns colleagues are currently raising, and I would rather surface the tension than pretend it does not exist.

Case one: the asylum file that no one has time to write

Asylum seekers in the United States are among the most vulnerable people any social worker will encounter. The paperwork is dense, the evidentiary standards are exacting, and the volume of need dwarfs the supply of attorneys, accredited representatives, and social workers available to help. People with legitimate claims are denied because no one had the capacity to thoroughly document country conditions, organize the narrative, or assemble the exhibits before the filing deadline.

Suppose a qualified representative uses a large language model as a drafting aid. The representative verifies every factual claim and every citation against primary sources, reviews the country conditions evidence for accuracy, and takes full professional responsibility for the final filing. The tool accelerates the work; it does not replace judgment. The result is a stronger, more complete draft in a fraction of the time, produced under the same standard of care the representative would apply to anything leaving the office.

Is using it a violation of the standards you have set out? If the answer is yes, what is your response to the person whose claim goes unfiled because the alternative was nothing? If the answer is no, then the moratorium is not actually a moratorium, and we should say so plainly.

Case two: research that would otherwise not happen

There are research questions in social work that require processing tens of thousands of administrative records, case narratives, or published abstracts. Done manually, the work takes months or years. Much of it never gets done at all, which means the knowledge that would protect children, inform substance use treatment, or evaluate policy effects simply does not exist.

A well-designed NLP pipeline can complete the same work in days, under conditions that make the results defensible: validation against hand-coded gold standard samples, reported error rates, sensitivity analyses for construct drift, and domain-expert review of outputs before anything enters the analytic record. The energy cost is real. It is also a small fraction of the cost of a year of human labor, travel, server time, and institutional overhead to do the work the slow way, assuming the slow way is even possible. More often than not, the realistic alternative is not slower research. It is no research.

Under your standards, is this acceptable? If not, what is the plan for the questions that will go unanswered? If yes, what are the conditions: open-weight models only, local deployment only, documented energy accounting, public release of the pipeline, something else?

Case three: AI that predates generative AI

Many of the tools social work researchers and practitioners have used for years are AI in the technical sense: supervised classifiers, topic models, word embeddings, clustering algorithms, and rule-based NLP. They run locally on a laptop. They do not flatter the user. They do not scrape the internet. They do not consume meaningful water. They were built by researchers using published methods.

Are these also prohibited? If the concern is generative models specifically, the critique needs to say so. If the concern is the word “AI,” then the critique is about a label rather than a practice, and much of the field has been inside the boundary for a long time without anyone noticing.

Case four: a small model running on your laptop

Suppose the model is small enough to run entirely on your own computer, with the network cable unplugged and the wifi turned off. Nothing touches a cloud server. No query is logged by a corporation. No data is ever exposed to a third party. Suppose it was distilled from a larger model, which means most of the training cost has already been paid, and the efficient version inherits that capability with a fraction of the energy footprint during inference. Suppose it handles a specific, bounded task: summarizing a case narrative, drafting a first pass at an outreach letter, classifying records against a codebook.

No data leaves the laptop. No API calls, not even occasional ones. No water consumed per query. The user controls the weights and can audit the behavior. The marginal environmental cost of running it is closer to that of running a spreadsheet than to that of hitting a cloud API. A client’s information stays within the room where it was collected.

Is this acceptable? If the objection is that the small model inherits from a larger one with scraped training data, that history is real, but the same logic applies to every search engine, every compiler, and every piece of software built on public code. If the objection is environmental, the local offline model is the most defensible version of the technology that currently exists. If the objection is data privacy, offline operation directly addresses it. If the answer is still no, then the category being rejected is broader than generative AI, and the field should say so plainly.

The AI is already in the room

The field already operates inside AI systems. GPS routing, spam filters, autocomplete, photo search, fraud detection, voice transcription, medical imaging triage; the infrastructure of ordinary professional life has absorbed machine learning without a moratorium and largely without debate. I note this not to accuse anyone of inconsistency, but to establish the baseline. The question is not whether to participate in AI. That boundary was crossed years ago. The question is which deliberate additions the field will engage with, and on what grounds.

The deskilling concern

A related concern deserves direct treatment: that routine use of these tools will erode the capacities of the practitioners who rely on them. I take the concern seriously, and I think it is misdirected. Deskilling happens when we stop teaching skills. A student who never learns to read a country conditions report, trace a citation, or evaluate a source will be deskilled regardless of what tools are in the room. A student who learns those skills first, and then learns to use tools under supervision, is not. The question is not whether AI is present in the workflow. It is whether the curriculum continues to develop the underlying capacities that let a practitioner supervise the tool rather than defer to it. That is a pedagogical responsibility, and it is ours. Framing deskilling as a property of the technology lets the curriculum off the hook for a problem that belongs to the curriculum.

The conference we flew to

The field has already worked out, for air travel, exactly the kind of framework I am asking for here. Every year, social work researchers fly on fossil fuels to conferences, including conferences where emerging technology ethics are on the program. We do not treat the practice as an ethical crisis, nor as ethically neutral. We weigh purpose against cost, reduce harm at the margins through shared travel, virtual options, fewer trips per year, and carbon accounting in program notes, and we accept that some trips are worth taking and others are not.

That is the reasoning structure I am asking the field to apply to compute. Not blanket acceptance, and not blanket refusal. A clear accounting of which uses are worth which costs, under what conditions, with what mitigations. The framework is mature in one domain, and it translates directly to the other.

The positionality question

There is a version of this critique that functions less as a standard and more as a personal exemption. An established researcher with tenure, a mature research program, and a professional network built over decades can plausibly continue the rest of their career without ever opening a language model. The workflow is intact. The citations still flow. The pipeline was built in a different era and still runs. A career can be built and maintained on those terms, and I know several people doing exactly that.

That is a defensible personal choice. It is not a field-wide standard, and it should not be presented as one.

The students I teach do not have that option. The practitioners inside organizations that are already integrating these tools, with or without their input, do not have that option. The self-represented litigant who cannot afford an attorney does not have that option. The asylum seeker whose case is sitting in a pile does not have that option.

If your ethical framework works for you because your career was already built, the field still needs to reckon with the fact that it does not work for the people whose careers are being built right now, inside a profession that is being reshaped, whether social work engages thoughtfully or not. Knowledge work is moving into an AI-infused environment. That is not a prediction. It is the current operating reality across every adjacent field, and social work will be no exception.

Opting out personally is a choice available to some. It is not an answer to the question the field is facing. It is a way to avoid answering it.

What I am actually asking

I am not asking anyone to drop their concerns. I am asking for specificity.

If the answer is that no use is acceptable, I want to understand how that position holds up against the asylum, research, classifier, and local model cases. What happens to the people and the questions on the other side of that line?

If the answer is that some uses are acceptable, then the work begins. What are the conditions? Open weights? Local deployment? Energy budgets? Human review requirements? Sycophancy mitigation in supervisory and ethics-consultation contexts through prompt design, model selection, and workflow constraints that preserve the challenge function? Population-specific carve-outs? Disclosure standards that go beyond the current boilerplate? A tiered framework that distinguishes research from practice from administration? Who decides, and how do disagreements get adjudicated?

A third option exists in principle: case-by-case review under institutional ethics processes, with the burden on the proposer and no pre-specified decision rule. I would welcome that framework. It is not the framework currently being advanced, and until it is, the conversation tends to settle into either blanket objection or unspecified conditions. Either of the first two can be argued, refined, and held accountable together. A blanket position is harder to work with, because the moment it encounters a hard case, it either bends or it leaves the people the case was about without a plan.

Every critic of generative AI who publishes in a commercial journal is already using a tool built inside an exploitative system. The field’s response to academic publishing has not been refusal; it has been naming the harm, pushing for open access (while recognizing the unanticipated problems that reform has introduced, from predatory journals to article processing charges that disadvantage unfunded scholars), and continuing to use the infrastructure while working to change it. That is the posture I am asking us to bring to AI.

I am open to being convinced that the line should sit in a particular place, and I am open to revising my own current uses in light of a better argument. What I am not able to work with is a position that refuses to draw a line at all, because the people in the cases above cannot wait for one that never gets drawn.

This post proposes five standards that any environmental discussion of AI in social work should meet. I will state them up front and then walk through each one.

1. Specify which model is running, where, and for what task.

2. Distinguish inference costs from training costs, and acknowledge embodied hardware costs honestly.

3. Treat right-sizing as a core environmental practice.

4. Apply the environmental justice lens to the actual system in question.

5. Anchor quantitative claims to sources that describe the specific system, not to aggregate statistics applied indiscriminately.

None of this is a defense of frontier AI. It is the opposite. The environmental stakes warrant analysis rather than slogans.

Specify the model, the location, and the task

The word “AI” in most social work writing functions as if it referred to a single technology with a single environmental cost. It does not, and the problem runs deeper than most critics acknowledge.

Read the field’s recent writing on the environmental consequences of AI, and a pattern becomes clear. The concerns are almost always about large generative models accessed through hyperscale APIs. That narrow category is then quietly relabeled “AI,” and the conclusions are generalized to everything the term might cover. The relabeling is the move that does the damage, because everything the term covers is a much larger set.

Encoder-only models such as BERT and ModernBERT do classification and extraction work without generating a single token. Embedding models turn documents into vectors for retrieval and similarity search, and a typical embedding call is a fraction of the workload of a generative query. Classical methods, including logistic regression, random forests, and gradient-boosted trees, have been in routine use in social work research for decades and remain the right tool for many tasks a researcher might want to perform. All of these are AI by any reasonable definition. Their environmental profiles differ from those of frontier generative models in ways that a serious analysis cannot ignore. A researcher classifying case notes with a fine-tuned encoder on a laptop is not doing the same thing, environmentally or technically, as a practitioner running a long-context prompt through a frontier API, and a critique that cannot tell them apart is not yet describing the technology it claims to assess.

The second layer of specification sits inside generative AI itself. Consider where the field stood at the beginning of April 2026. Google released Gemma 4, a family of open-weight models that anyone can download and use for free. The smallest members of that family are designed to run on a phone. A social worker with a recent Android device or a current iPhone can install the Google AI Edge Gallery app, download a model, switch to airplane mode, and have a useful conversation with an assistant that never contacts a server. The model sits on the phone the way a podcast episode sits on the phone.

That is not the same technology as a frontier model accessed through ChatGPT, Claude, or Gemini. Frontier models live in hyperscale data centers and depend on warehouses full of specialized chips, dedicated power substations, and industrial cooling systems. A small open-weight model running on a device you already own relies on hardware already sitting on your desk or in your pocket. The two cases differ substantially in the energy they draw and the infrastructure they require. Treating them as the same thing when writing about environmental impact is a category error, and it is the second one the field routinely makes after collapsing all of AI into generative AI in the first place.

Any environmental argument should therefore be able to answer two questions rather than one. What kind of model is running, generative or otherwise? And if it is generative, where is it running and on what scale of infrastructure? A critique that cannot answer either has not yet reached the point where environmental analysis can begin.

Distinguish inference from training, and acknowledge embodied costs

The strongest version of the opposing view deserves a direct response. Small local models exist only because someone, somewhere, trained large ones first. The hardware that runs them carries embodied carbon from manufacturing, regardless of where inference happens. Both observations are correct, and neither is what I am contesting.

The claim I want to make is more specific. Training costs are real, but they are amortized across every subsequent use of a model, and open-weight releases distribute that amortization across a much larger and more diverse user base than a proprietary API does. A model trained once and downloaded by millions is a different environmental object than a model trained once and served exclusively through a metered endpoint.

Embodied hardware costs are also real, and this is where the argument needs care. A laptop already sitting on a practitioner’s desk carries no marginal embodied cost when it is used for local inference. A workstation or small GPU server purchased specifically for local inference does carry such a cost, but the relevant comparison is not local hardware against nothing. It is local hardware against the share of data center capacity that a comparable workload would consume over the same period. A single workstation, manufactured once and used for years across many workloads in an agency or a school, sits in a different category from hyperscale infrastructure that depends on continuous buildout, dedicated power substations, industrial cooling, and sustained water withdrawals. Treating the two as environmentally equivalent because both involve hardware is the same flattening move I am arguing against throughout this piece.

An honest environmental discussion should make these distinctions clear rather than collapsing them into a single, undifferentiated cost.

Right-sizing is an environmental practice

Some tasks genuinely require large generative models running on large infrastructure. Long-context reasoning across complex documents, high-quality code generation, and several multimodal tasks currently depend on frontier systems. Those uses carry real environmental costs and should be discussed accordingly.

Many tasks do not, and the right-sizing argument has two axes rather than one.

The first axis is whether the task needs a generative model at all. A great deal of the text work that fills a social work research pipeline or an agency workflow is classification, extraction, retrieval, or similarity search. Labeling case notes by service type, pulling structured fields from intake forms, finding the policy documents relevant to a question, or grouping records by topic are all tasks an encoder-only model or an embedding pipeline can perform at a fraction of the computational cost of a generative query. For many of them, a logistic regression or a gradient-boosted tree trained on a modest labeled sample will match or exceed the accuracy of a large language model while running on a laptop in seconds. Reaching for a frontier generative model to do work that a fine-tuned encoder or a classical classifier would handle is not a neutral choice. It is a more expensive choice, both financially and environmentally, and often less accurate.

The second axis sits inside generative AI. When a task does call for a generative model, the question becomes which one. Summarizing a short document, drafting a routine email, tidying up meeting notes, or rewriting a paragraph for clarity can be handled by small open-weight models running locally. When practitioners and educators reach for the largest available frontier model to do work that a small-parameter model on a laptop could adequately complete, the cost is not only financial. It is environmental and avoidable.

Right-sizing, which is the practice of matching the tool to the task across both axes, is therefore a core environmental position. An honest environmental argument in social work should name it as such. It should point readers toward classical methods where they remain appropriate, and toward Ollama, LM Studio, and the Google AI Edge Gallery for the cases where a small generative model is the right choice. The individual case matters, but the institutional case matters more. The environmental argument for right-sizing becomes substantial when a school of social work, a child welfare agency, or a legal aid organization adopts local and task-appropriate tools as the default rather than routing every problem to a frontier API. A critique that warns about the carbon cost of AI while ignoring the existence of tools that run on a laptop does not give readers the information they need to act.

Apply the environmental justice lens to the actual system

The environmental justice framing matters. Data center siting decisions, water withdrawals in drought-stressed regions, and the uneven distribution of environmental harms across communities are legitimate concerns that belong in this conversation. The lens, however, only works when applied to the specific system under discussion. Invoking environmental justice while failing to distinguish between a cloud-based frontier model queried through a hyperscale API and a small open-weight model running offline does not advance the cause of environmental justice. It borrows the moral authority of that framing without conducting the environmental analysis it requires.

A properly specified environmental justice analysis would name the data center, identify the community affected, describe the water and power draws of that specific facility, and connect those harms to the particular model families and workloads the facility supports. That is a harder piece of writing than a general invocation, and it is also the one the field actually needs.

Anchor the numbers

Quantitative claims about energy, water, and carbon should be tied to sources that describe the specific system in question. Aggregate statistics about “AI” as a category are not usable evidence for a claim about a particular workflow. If the claim is that a specific task on a specific model carries a specific cost, the citation should support that level of specificity. If the only available numbers are aggregate, the claim should be scaled back to what those numbers can actually support.

What this looks like in practice

A presentation, LinkedIn post, or paper that meets these five standards will look noticeably different from most of what is currently being discussed. It will name models. It will distinguish inference from training. It will treat local, open-weight alternatives as part of the environmental conversation rather than as a technical footnote. It will apply environmental justice framing to specific facilities and specific workloads. It will use numbers that actually describe the system under discussion.

That is a higher bar, and it should be. The communities whose water and air are affected by data center expansion deserve analysis at that level. So do the practitioners trying to decide what tools to adopt and what defaults to set for their organizations. Meeting the bar is the work. The standards above are where it starts.

Here is a common pattern I see in my work with organizations looking to integrate AI into administrative workflows. The majority of the resources are devoted to preparing existing documents and data for machine consumption, because none of it was designed with that possibility in mind.

I call this the “retrofit tax.” This is the cost an organization incurs when it tries to bolt AI onto a workflow built exclusively for human readers or web displays. The tax is not inevitable. It is a design choice, usually made by default rather than deliberately, and it is one of the most expensive decisions organizations are making right now without realizing it.

Working example: the policy manual

Consider something that exists in nearly every organization: a policy manual. It might govern human resources procedures, academic standards, compliance requirements, or clinical protocols. These documents have been written, formatted, and distributed for human consumption long before anyone was thinking about large language models. They are optimized for someone who will read them sequentially or look up a specific section by scanning a table of contents.

Now consider what happens when someone proposes a reasonable AI application for that same content. A chatbot that answers employee questions about leave policies. A compliance checking tool that compares operational procedures against the written standards. A system that generates training materials from the policy content. A search interface that lets staff ask natural-language questions and get accurate, cited answers.

Each of these applications requires the AI system to consume policy content programmatically. And this is where the retrofit tax hits. The policy manual was published as a PDF. The PDF preserves how the document looks, but it does not reliably preserve the structure of the information. Section boundaries are visual, not semantic. Cross-references say “see Section 4.3” in prose rather than linking to a defined location. Definitions are scattered across chapters. Tables render as images. Headers that look hierarchical to a human eye may be indistinguishable from body text to a parser.

So the AI project becomes a data preparation project. You build extraction pipelines to pull text from PDFs. You write custom logic to identify where one section ends, and another begins. You design chunking strategies that attempt to keep related content together despite formatting that works against you. You build a retrieval-augmented generation system and spend weeks tuning it because the source material keeps producing inaccurate or incomplete results.

By the time the chatbot works adequately, you have spent more on document preparation than on the AI system itself. And every time the policy manual gets updated, much of that preparation work has to be repeated.

What does AI-ready actually require?

Here is what surprises most people when I walk them through this: making a document AI-ready does not require sacrificing anything about the human reading experience. It typically requires only a lightweight formatting layer on top of good content practices that most style guides already recommend.

An AI-ready policy manual at the source has a few characteristics. The content is authored in a structured plain-text format, such as Markdown, rather than directly in a formatted output like a PDF, a complex word-processing template, or an HTML web page. Heading levels are consistent and hierarchical, so a machine can reliably determine that Section 3.2.1 falls within Section 3.2, which falls within Section 3. Cross-references point to defined anchors rather than relying on page numbers or visual proximity. Definitions are collected in a consistent location or tagged inline so they can be extracted programmatically. Metadata about each section, its scope, effective date, and applicability is maintained alongside the content rather than embedded in introductory prose.

From this single structured source, a lightweight formatting layer produces whatever output the organization needs. A styled PDF for printing. A web page for the intranet. A formatted document for distribution. The human-facing outputs look exactly as they would under the traditional approach. But the source material is also immediately consumable by an AI system, with no extraction, conversion, or custom parsing required.

This is not a novel architecture. It is how modern documentation systems have worked for years. Software companies author documentation in markdown and generate websites, PDFs, and API references from the same source. The principle of separating content from presentation is well established in technical fields, even if it remains the exception rather than the rule in most other industries. Academic publishing, for instance, still largely relies on PDF as both the authoring target and the distribution format, which is one reason that mining the scholarly literature at scale remains so difficult. The point is that the tools and methods for doing this well already exist. They simply have not been applied to most administrative and organizational documents, because until recently, there was no compelling reason to do so.

The compounding cost of ignoring this

The retrofit tax is not a one-time expense but something that compounds. Each new AI use case applied to the same poorly structured source material requires its own preparation pipeline. A chatbot needs one chunking strategy. A compliance tool needs another. A training material generator needs a third. Each time the source document is revised, every downstream pipeline needs to be updated and retested.

Organizations that structure their documents for AI consumption at the source pay the design cost once. Every subsequent AI application draws from the same clean, structured material. The second use case is dramatically cheaper than the first. The tenth is nearly free.

Organizations that build strictly for human consumption and then retrofit pay the preparation cost repeatedly. Each AI application is its own project. Each revision cycle multiplies the maintenance burden. The cumulative cost over three to five years dwarfs whatever modest investment would have been required to structure the content properly at the outset.

The opportunity in every redesign

Most organizations are not going to stop what they are doing and restructure every existing document. That would be its own kind of expensive retrofit. But documents and processes get redesigned all the time. Policy manuals are revised. Handbooks are rewritten. Reporting systems are updated. Submission processes are overhauled.

Each of these redesigns is an opportunity to build in AI readiness at negligible incremental cost. The question to ask during any redesign is not “Are we building an AI tool right now?” It is “are we structuring this so that an AI tool could work with it later, if we choose to build one?” The answer almost always requires nothing more than choosing a structured source format, maintaining consistent organization, and separating content from presentation.

The organizations that build this thinking into their design processes will find that AI integration becomes routine and affordable. The organizations that continue to design exclusively for human consumption will keep paying the retrofit tax, project after project, year after year. The difference is not in the technology. It is in the decision about when to think about it.

Last week, our team released the SSWR Conference History Database, a structured dataset of 23,793 presentations from the Society for Social Work and Research Annual Conference spanning 2005 to 2026. I described it as AI-ready, meaning the data shipped with precomputed word embeddings, clean metadata, and everything an AI assistant would need to work with.

I was wrong. Or at least, I was incomplete.

The dataset was AI-ready for people who already had the tools. If you knew how to install Python, connect to a database, and write queries against a vector store, the data was yours. But that is not the case for most social work researchers. It is not most doctoral students. It is not that most of the people who attend SSWR every January present their work and never see it cited again.

So our team went back and built what should have existed from the start: a browser-based search interface that requires no software installation, no technical knowledge, and no account. You open a web page. You type a query. You get results with one-click APA citations.

What it does

The search system offers five modes. Keyword search does what you would expect: exact term matching in titles and abstracts. Author search finds researchers by name using fuzzy matching to handle spelling variations. Institution search lets you look up presentations by university or organization.

The two modes that required more engineering are semantic search and hybrid search. Semantic search converts your natural-language query into a mathematical representation, a word embedding, and compares it against pre-computed embeddings for all 23,793 abstracts. If you search “how communities recover after natural disasters,” it surfaces papers about post-disaster resilience even if those exact words never appear. Hybrid search combines keyword precision with semantic breadth using a rank fusion algorithm, giving you the benefits of both approaches in a single result list.

Every result includes the year, methodology, authors with their institutional affiliations at the time of presentation, a link to the original Confex page when available, and a button that copies a formatted APA 7th edition citation to your clipboard. You can filter by year range and methodology type. You can download the results as a CSV file.

This is not generative AI

Before the objections arrive, let me be direct about what this system does and does not do.

It does not generate text. It does not summarize. It does not hallucinate citations or fabricate abstracts. Every result is a real presentation from the SSWR archives, displayed exactly as it was submitted. The system uses word embeddings, which are mathematical representations of meaning developed through natural language processing research, to find conceptual matches between your query and existing abstracts. The embedding methodology is explained in a paper we published last year in the Journal of the Society for Social Work and Research. The technology predates the current wave of generative AI by years.

I raise this because I know the reflex. Any mention of AI in a research context now triggers reasonable concern about environmental costs, hype, and whether we are solving real problems or chasing trends. Those concerns are legitimate when applied to generative AI systems that consume enormous computational resources to produce text of uncertain reliability.

They do not apply here. A semantic search query against this database costs approximately two one-hundredths of a cent. The entire system runs on free-tier infrastructure. The embeddings were computed once. The ongoing computational footprint is negligible. The project is funded entirely from personal savings accumulated during my previous career as an independent entrepreneur, when my brothers and I specialized in selling rocks. I wish that were a joke.

The real problem this solves

Here is what should bother us. The Society for Social Work and Research has held annual conferences for nearly 30 years. Researchers have traveled across the country, paid registration fees, booked hotel rooms, prepared presentations, and shared their findings. The collective investment in this scholarship is staggering, easily hundreds of millions of dollars when you account for the research itself, the travel, the institutional support, and the carbon emissions from nearly three decades of flights and hotel stays.

And almost none of this research is cited. Conference presentations occupy a category sometimes called gray literature, not because the work lacks rigor, but because it lacks organization and indexing. A presentation delivered in 2012 about community-based interventions for opioid use disorder effectively vanishes once the conference ends. The abstract might exist somewhere on a Confex server, but no one can find it through a standard literature search. No one builds on it. The investment largely evaporates.

We have had over twenty years to organize this body of work, and we have not done it. The programs were printed, distributed, and recycled. The digital records were scattered across annual websites with no unified search, no consistent metadata, and no way to trace a researcher’s contributions across years or connect related work across sessions.

Using AI to solve this organizational problem does not qualify as hype. It qualifies as overdue infrastructure.

What AI-ready actually means

When we first released the dataset, AI-ready was defined in technical terms: structured data, clean embeddings, and documented schemas. I now think the definition needs to be broader. A dataset is not truly AI-ready if using it requires specialized software. It is AI-ready when the people who need it can reach it.

For the SSWR Conference History Database, that means a doctoral student writing a literature review can search twenty years of presentations in a single query to find who has been working in their area, instead of clicking through twenty-one separate annual conference programs on Confex. A researcher writing a grant proposal can identify potential collaborators and check whether the question has already been explored at SSWR. A faculty member can trace how a research topic has evolved across two decades of conferences, or discover that a colleague at another institution has been working on the same problem from a different angle. And anyone who needs to cite a conference presentation gets a formatted APA 7th edition reference copied to their clipboard with one click, rather than reconstructing it by hand from an old program book.

I don’t agree that AI is just hype…

On April 24, 2026, a federal rule takes effect that will reshape how public institutions deliver digital content. The Department of Justice’s ADA Title II Digital Accessibility Rule, published in April 2024, establishes the first legally binding technical standard for web and mobile accessibility across state and local government. The standard is WCAG 2.1 Level AA. The scope is broad: every website, every app, every online form, every PDF, every court document, every video, every digital service provided by a covered entity must comply.

The first compliance deadline applies to jurisdictions serving populations of 50,000 or more. That includes most state agencies, public universities, court systems, transit authorities, and municipal governments. Smaller jurisdictions have until April 2027. The penalties are real: up to $75,000 for a first violation, $150,000 for subsequent violations, assessed per instance. A single inaccessible e-filing portal or batch of untagged court forms could generate substantial liability.

This is not new territory in principle. The ADA has been the law since 1990. Section 508 has required federal digital accessibility for decades. WCAG guidelines have existed since 1999. What is new is that the DOJ has, for the first time, specified an enforceable technical standard for state and local government and attached a deadline to it. The DOJ’s compliance guidance outlines the steps covered entities should be taking now. The era of vague obligations and good-faith efforts is ending.

The document problem

Most institutions I talk to understand, at least in broad terms, that their websites need to be accessible. Fewer have reckoned with the document problem. Public institutions produce and host enormous volumes of PDFs, Word documents, slide decks, and scanned forms; court opinions, administrative paperwork, policy documents, research publications, meeting minutes, and budget reports. The overwhelming majority of these documents are inaccessible. They lack tagged headings, meaningful reading order, table structure, alt text for images, and navigable form fields. A screen reader encounters most of them as a wall of undifferentiated text, or worse, as a series of images with no text content at all.

Manual document remediation is expensive. Industry estimates range from $50 to $150 per document, depending on complexity, and a single document can take a trained specialist 10 to 15 hours to complete. For an institution with thousands of documents, the math is prohibitive. This is why, despite decades of legal obligation, most public-facing documents remain inaccessible. The cost of compliance, on a document-by-document basis, has exceeded the institution’s willingness to pay.

The new rule changes that calculus. The penalties for non-compliance now carry dollar amounts, and the DOJ has signaled its intent to enforce. Institutions that have been deferring remediation no longer have that option.

“Just stop using PDFs” is not a solution

A response I have encountered with increasing frequency is the suggestion that institutions should simply move away from PDFs. Use native HTML. Publish in accessible formats from the start. Stop producing PDFs altogether.

This sounds reasonable if you are a legal compliance team or a computer scientist advising from outside the workflow. It does not sound reasonable if you are an educator whose entire field runs on PDFs. Scientific publishing produces formatted PDFs. That is the output. Journals publish PDFs. Preprint servers distribute PDFs. Course reading lists are PDFs. Faculty share PDFs. Students read PDFs. The suggestion that higher education should abandon the format in which virtually all of its scholarly literature is produced and distributed is not a practical recommendation; it is an abstraction offered by people who do not deliver the education.

The same holds across government. Courts produce PDFs. Agencies publish PDFs. Legislative bodies distribute PDFs. The format is embedded in institutional workflows at a level that will not change by April 24, or any time soon. A useful solution has to meet institutions where they are, not where a compliance consultant wishes they were.

A solution that is all LlamaParse and a bit of vibe coding

I built a tool using LlamaParse that converts inaccessible documents to fully accessible, WCAG 2.1 AA-compliant HTML. It handles headings, tables, equations, figures, alt text, reading order, skip navigation, landmark regions, and auto-generated tables of contents. Every output file passes an eight-point accessibility audit at a 100% rate.

I want to be precise about what I actually built, because the honest accounting matters. The tool is 95% LlamaParse and 5% my own work. LlamaParse, developed by LlamaIndex, is an API service that extracts structured content from documents (e.g., text, tables, equations, figures, headings) and returns it as a clean, organized Markdown file. That extraction is the hard part. It is the part that has taken the LlamaIndex team years of hard work to refine, making them the industry leader in this space.

LlamaParse does it well, and it does it at scale. But LlamaParse does not produce accessible HTML. Its output formats are markdown, JSON, and plain text; structured representations of a document’s content, but not WCAG-compliant web documents. That gap between structured markdown and accessible HTML is where my vibe coding comes in. And because LlamaParse handles over 130 file types (e.g., PDFs, Word documents, PowerPoints, Excel files, images, etc.), this approach is not limited to PDF remediation. Virtually any document format an institution produces can be routed through the same pipeline to produce accessible HTML output.

My post-processing pipeline is simple: five Python scripts that convert LlamaParse’s markdown output into accessible HTML. One script handles math equations, routing them through MathJax for screen-reader navigation. One restructures tables with proper headers and captions. One wraps figures with accessible markup. One repairs heading hierarchies and adds navigation IDs. One adds the accessibility scaffolding: skip links, ARIA landmarks, language attributes, and the table of contents. Then an audit script checks the output against WCAG criteria.

I wrote those scripts with Claude as my coding partner. I am not a software engineer, so you might cringe at my vibe-coding efforts. That’s fine. I welcome you to help address this larger problem I am trying to solve. This isn’t about flexing. It’s about solutions.

The cost comparison

The numbers are worth stating plainly.

LlamaParse uses a credit-based pricing model. The free tier includes 10,000 credits per month; at the standard extraction rate of one credit per page, that is 10,000 pages at no cost. Paid tiers start at $50 per month for 40,000 credits. The effective cost per page ranges from roughly $0.001 to $0.003, depending on the extraction mode, well under half a cent in every case.

For context, I built a cost comparison that I keep coming back to: one meeting; four staff members, one hour, salaries averaging $60K annually, costs an organization about $170, and converts zero documents. The same $170 applied to this tool would convert tens of thousands of pages to accessible HTML. And the free tier alone covers 10,000 pages per month before an institution spends anything.

The scripts used to convert markdown to accessible HTML are available on GitHub. A user brings their own LlamaParse API key, free to obtain from LlamaIndex, with a generous free tier. You upload a PDF or a batch of documents, and get back self-contained HTML files that work in any browser with any screen reader. I am in the process of setting up the repository and making it publicly available online, and I will share the link when it is ready.

Not perfect, but effective and sustainable

I want to be honest about limitations. This is not a 100% perfect solution. Edge cases exist. Complex multi-column layouts, heavily designed documents, unusual table structures, and scanned images without OCR will produce imperfect output. Neither an automated system nor a human specialist working at speed can handle every document flawlessly. But, so far, the PDFs, PPTs, and other docs have all passed the accessibility test.

And when I post the accessible HTML files to Canvas, the University of Michigan’s learning management system, all the HTML files (as pages) pass the accessibility test.

But the question is not whether the tool is perfect. The question is whether it is effective, cheap enough to deploy at scale, and sustainable over time. On all three counts, the answer is yes. A tool that correctly converts the vast majority of documents at half a cent per page, achieves a 100% pass rate on accessibility audits, and can be run by anyone with a free API key is a practical response to a problem that has resisted solutions for decades. The remaining edge cases can be identified and addressed through human review at a fraction of the cost of full manual remediation.

Why this matters beyond compliance

I want to be clear that I view this primarily as a civil rights issue, not a compliance issue. Approximately 42.5 million Americans live with a disability that affects their use of digital content. When a court system publishes opinions as inaccessible PDFs, it is not a technical shortcoming; it is an access-to-justice failure. When a university posts research behind inaccessible formatting, it excludes the people that research is meant to serve. The standards to fix this have existed for more than 25 years. The publishers and institutions with the resources to implement them have chosen not to do so.

What has changed is that the cost barrier has collapsed. The argument that accessibility is too expensive to implement at scale no longer holds. A tool built in two days by a non-engineer, powered by an extraction API that costs less than half a cent per page, produces accessible output that passes every audit check. The technology to do this is available now at a price point that makes inaction a choice rather than a constraint.

The April 24 deadline is four weeks away. For institutions still weighing their options, the tool will be freely available and open source. The harder question, why accessible documents were not the default from the beginning, is one that the deadline does not answer.

• GitHub repo

• Streamlit app

I have tried to solve this problem for years. Barcode scanners. Phone-based cataloging. Tagged spreadsheets. Each iteration of AI seemed to solve a different piece of the puzzle: storing data, retrieving data, and moving from keyword matching to semantic search. But none of those pieces ever assembled themselves into a working system.

What finally brought them together was understanding how skills work in agentic AI. Not a specific platform. Not a specific model. The concept of a skill itself: what it is, what it contains, and why it is the fundamental unit of useful AI work?

What a skill actually is

In the context of agentic AI, a skill is a structured set of instructions that teaches an AI agent how to perform a specific task using specific tools. It is not a prompt. It is not a chatbot personality. It is a complete operational manual for a defined piece of work.

A skill typically contains three things. First, the instructions: when to activate, what questions to ask, what logic to follow, and how to handle edge cases. Second, the tools: the external capabilities the agent needs to execute the task, such as a database, a file system, a web browser, or an API. Third, the output format: what the result should look like and how it should be delivered back to the user.

This is the critical distinction between agentic AI and conversational AI. A language model without skills can discuss tool organization strategies, recommend database schemas, and explain SQL syntax. A language model with the right skill can actually build the database, populate it from a photograph, query it in natural language, and return the answer to your phone. The skill is the bridge between knowing and doing.

Skills are not specific to any one platform

An important clarification: the concept of a skill is not unique to any single AI provider or application. OpenAI, Google Gemini, and Anthropic’s Claude have all adopted skills within their ecosystems. This is not a niche feature confined to one environment the way a Custom GPT is specific to OpenAI’s platform. Skills are the emerging standard for how agentic systems organize and execute structured work, across providers and across applications.

I use skills in OpenClaw, which is where I built the system I describe below. But I also use skills in Claude Desktop and Claude Code, each for different purposes. The underlying architecture is the same everywhere; a structured process description paired with tools, given to a language model that can decide when and how to invoke them.

If you already have a well-defined process description for a task, you already have the raw material for a skill. The agentic systems you may already be using support this capability. It is a matter of learning how to provide those instructions in the format the platform expects.

One practical note on cost: it is difficult to find an entirely free path through this ecosystem. OpenClaw itself is free and open-source, but it requires a language model to run, and capable models generally involve some expense, whether through API fees, a subscription, or the hardware to host a local model. Claude Desktop and Claude Code require subscriptions. A fully zero-cost setup is possible in narrow cases, but most people working with these systems should expect some cost. That is worth knowing upfront.

How skills have tools, and why that matters

The word “skill” can sound abstract until you see what sits inside one. A skill is not just prose instructions. It bundles tools; executable capabilities that the agent calls during the task. This is what gives an agent the ability to act on the world rather than merely describe it.

In the system I built, the skill is called tool-tracker. It lives in a single directory on my server and consists of four files: a SKILL.md instruction file and three Python scripts. Those scripts are the tools. One initializes the SQLite database and creates the tables. One inserts new tool records with location history logging. One handles search and filtering across multiple parameters. The agent also has direct SQL access for anything the scripts do not cover.

When I send a message to Telegram saying “where’s my level?”, the agent reads the SKILL.md file to understand how my location system works, calls the query script with the right parameters, receives the formatted results, and sends them back to me with the location and a photo of the drawer. The skill told it what to do. The tools let it do it. The model decided which tool to invoke based on my natural language request.

This is the anatomy of every agentic workflow: instructions, tools, and a reasoning engine that connects them. The skill is the package that holds the first two together. In my case, the agent harness, OpenClaw, provides the third.

What I built, step by step

I want to describe this concretely, because the details reveal how the process actually works. But I want to emphasize this. This entire workflow was developed in natural language. I didn’t write a single line of code. Do I trust it? Yes. Does it work? Yes.

Defining requirements through conversation

I started by opening a conversation with my OpenClaw agent, which I access through Telegram. I told it what I wanted: a database to manage physical tools spread across two locations, with the ability to query by name, category, or location, and to add new entries using photos and text. I asked it to plan the system before building anything.

The agent asked clarifying questions. What are the two locations? What fields do I need per item? Am I entering data from my phone or a desk? Do I need to track who has something checked out, or just where it lives? What database format do I prefer? (Interestingly, I have been using OpenClaw to brush up on my language skills in preparation for a trip to China. Chuck, my OpenClaw, periodically drops some Chinese words to freshen my skills!)

I answered each question in plain language. Home and workshop. Name, category, function, location. Phone, through Telegram. Just a location for now. SQLite is fine. The agent asked one more question that mattered: when a tool is “at home,” do I just need to know it is at Location A, or do I need to specify where at home? I told it I needed a second and possibly third level of storage definition. It immediately proposed a three-level hierarchy: Location, then Area/Zone, then Specific Spot. Home → Basement → Drawer 10. Workshop → Main Bench → Metal File Drawer.

Then it proposed a database schema. Three tables. A main tools table with dual-location tracking, independent zone and spot fields for each location, condition monitoring, consumables, photo paths, and timestamps. A tool_needs table for tracking missing consumables and maintenance items, linked by foreign key. A tool_location_history table for an audit trail of every movement, with cascading deletes. It showed me the schema, explained the design decisions, and asked if I was ready to build. The entire schema was developed through the converstion I shared.

| Field      | Type                | Notes                                             |
| ---------- | ------------------- | ------------------------------------------------- |
| id         | INTEGER PRIMARY KEY | Auto                                              |
| name       | TEXT                | Tool name                                         |
| category   | TEXT                | Functional category (drill, saw, measuring, etc.) |
| location_a | INTEGER             | 1 if present, 0 if not                            |
| loc_a_zone | TEXT                | e.g., "Garage", "Kitchen"                         |
| loc_a_spot | TEXT                | e.g., "Red Chest", "Junk Drawer"                  |
| location_b | INTEGER             | 1 if present, 0 if not                            |
| loc_b_zone | TEXT                | e.g., "Main Bench"                                |
| loc_b_spot | TEXT                | e.g., "Metal File Drawer"                         |
| photo_path | TEXT                | Path to photo file                                |
| notes      | TEXT                | Optional                                          |
| created_at | TIMESTAMP           |                                                   |
| updated_at | TIMESTAMP           |                                                   |All of this happened in natural language, in Telegram, in a few minutes.

Building the skill

The agent then built the skill: the SKILL.md instruction file, the database initialization script, the tool insertion script, and the query script. The SKILL.md is over 100 lines of structured instructions that tell the agent when to activate, how the location hierarchy works, what follow-up questions to ask based on tool type (power tools get battery questions; cutting tools get blade questions; drills get bit questions), how to detect missing consumables and automatically add them to the needs list, and what SQL patterns to use for every type of search.

My workflow for developing skills like this involves Claude Code, Anthropic’s command-line coding agent, which I use to generate and refine the SKILL.md files and supporting scripts. OpenClaw itself runs on a small, locally hosted model on my VPS; that is sufficient for executing well-defined skills at low cost. But writing the skills themselves benefits from a more capable model that gets the logic right on the first attempt. Other people take different approaches. Some give their OpenClaw instance access to a large external model for everything. OpenClaw can even autonomously write its own skills. There is no single correct workflow.

Populating the database with photos

With the skill built, I started cataloging. I opened a drawer of pliers, took a photograph, and sent it to Telegram with the message.

The agent used visual examination of the photograph to identify eight distinct tools: large blue-handled diagonal cutting pliers, curved-jaw locking pliers, a smaller pair of diagonal cutters branded NUINN, yellow-handled needle-nose pliers, and four others. It estimated sizes, noted visible rust on one pair, and listed all eight with descriptions. Before inserting them into the database, it asked two follow-up questions: which zone at home (basement, garage, closet?), and whether any consumables were needed, such as replacement handles, lubricant, or new jaw inserts.

It inserted all eight tools, linked them to the photograph, logged their initial locations in the history table, and confirmed the entries. I moved to the next drawer and repeated the process.

Quick fixes are also very simple.

Now, populating the database, I just take a picture of the tool and tell its location. The entire cataloging process took less time than it took to write this post. That ratio, years of friction reduced to minutes of construction, is the clearest measure I have of what this technology is worth when it is applied with discipline.

Querying the system

The database now supports natural-language queries via Telegram. “Where’s my level?” returns the location (Ann Arbor, Basement, Red Tool Box Drawer 2) with a photo. “Show me all my cutting tools” returns seventeen results grouped by location. “What’s in Drawer 9?” returns the box cutters, blades, and knife stored there. “What tools need maintenance?” returns the two items flagged for attention. “What do I need to buy?” queries the needs table and returns outstanding consumables.

When I move a tool between properties, the agent updates the location flags, records the movement in the history table with a timestamp, and confirms the change — I just need to send a short message to make the update. The audit trail means I can reconstruct where any tool has been and when it moved.

All of this runs through natural language in Telegram. I want to manage the entire production workflow right from the messaging app I already use throughout the day. The system meets me where I am.

And it is extensible

So, I have this database and set of skills all hooked up. I can now quickly add other automations and features. Since I am in the mood for house maintenance / DIY, I should probably add a few reminders to myself.

The anatomy of an agentic workflow

What I have described is not a chatbot. It is a structured workflow that includes a SQL database, Python scripts, vision-based data extraction, and natural language query processing, all orchestrated by a skill that tells the agent how to use each tool. The components are worth naming explicitly:

1. The skill (SKILL.md) contains instructions on when to activate, what to ask, what logic to follow, how to format the output, and which tools to call.

2. The tools (Python scripts and direct SQL access) are the executable capabilities: database initialization, record insertion with transaction management, multi-parameter search, and filtering.

3. The agent harness (OpenClaw) provides the runtime environment: routing messages from Telegram, loading the skill at session start, managing sessions, and letting the model decide which tool to invoke based on the user’s request.

4. The model is the reasoning engine that reads the skill, interprets the user’s natural language, decides which tool to call, processes the results, and formulates the response.

This four-part structure: skill, tools, harness, model, is the anatomy of every agentic workflow, regardless of platform. The specific technologies I chose are less important than the pattern.

The framework for building any workflow

The pattern is the same regardless of what you are building:

Define the problem concretely. What is the system supposed to do, stated in terms a person could understand and execute?

• Decompose the task into discrete steps. Each step should be something a language model can perform reliably when given the right prompt and the right tools. If a step is too complex or ambiguous for a trained professional to execute consistently, it is too complex for a model.

• Identify the tools each step requires. Does the step need database access? File system operations? Web search? Vision? Each tool becomes part of the skill’s architecture.

• Build each step as a skill. Give it clear inputs, clear instructions, and a clear output format. Test it. Refine it.

• Connect the skills through the agent harness. The harness is the orchestration layer. The skills are the capabilities. The model is the reasoning engine that decides what to do.

None of this requires programming once the system is set up. The technical difficulty of building these workflows in the backend is remarkably low. The hard part is the thinking that precedes it: defining the requirements clearly enough that each step becomes a repeatable, reliable operation a language model can perform with the right prompt and the right tools.

That is the part most people skip. And it is the part that determines whether the result is a useful system or a frustrating toy.

My most successful vibe coding projects succeeded not because the coding went well, but because the design thinking did. Frankly, the coding is the part that the AI handles. The part it cannot handle, at least not without sustained human judgment, is figuring out what to build, for whom, under what constraints, and at what scope.

Design thinking comes first

Before any code exists, there is a phase of problem definition that most people skip or underestimate. What is the actual problem? Who experiences it? What does a solution look like in their daily workflow? What are the constraints: technical, institutional, political, and budgetary? This is design thinking, and it is more demanding than people expect.

The difficulty is not in answering these questions. It is in constraining the answers. Scope discipline is the single most important skill in this process. The instinct is always to build the comprehensive system, the one that handles every edge case and serves every user. That instinct will sink your project before it starts.

I always push toward the MVP; the minimum viable product. What is the smallest thing you can build that will make a meaningful difference for this problem and give you a return on the time invested? Do not touch a coding tool until you can answer that question with specificity. You can use AI during this design phase; I have built a design thinking skill specifically for Claude that walks through these questions systematically. But the thinking is yours. The AI helps you structure it; it does not replace it.

From design to PRD

Once the design thinking is solid, the next step is producing a product requirements document. The PRD is a detailed written description that captures everything from the design phase: the problem definition, the target users, the workflows, the constraints, the scope boundaries, and whatever technical specifications you can identify. If you are early in your development experience, you may not be able to confidently select specific technologies; that is fine. The PRD remains the bridge between your thinking and the AI’s code.

This is the document you hand to your coding agent. Not a vague prompt. Not a wish list. A structured description of what needs to exist and why.

Building the proof of concept

With a PRD in hand, this is where a tool like Claude Code enters the process. But even here, the approach matters. You are building a proof of concept, not a production system. The goal is to take the simplest possible approach while remaining flexible enough for further development. Get the core idea working first. It may require refactoring later. That is expected and acceptable.

The critical point about this phase is that you are not problem-solving on the fly. Because you did the design thinking and wrote the PRD, you already know exactly where you are going. Your role during the build is not to generate solutions; it is to steer. You are continually directing the AI back toward the PRD, keeping it on task, and preventing it from drifting into unnecessary complexity. AI coding agents will happily solve problems you did not ask them to solve, add features you did not request, and make architectural decisions that complicate everything downstream. Your job is to read everything the system is doing, direct it to explain its choices, and course-correct when it veers off the plan.

This is where people misunderstand what “no coding experience” actually means in practice. You do not need to know how to write code. But you do need to understand what is being built. The AI’s role is not just to code for you; it is to help you comprehend what it is producing so that you can make informed decisions about direction. Ask it to explain. Ask it why it chose a particular approach. Ask it what alternatives exist. You are the project manager, the product owner, and the quality control system. The AI is the builder. If you abdicate the steering role, you end up with a product that technically runs but does not solve the problem you started with.

And yes, you do have to look at the code. Not to write it, but to verify it. Here is a concrete example that has bitten me repeatedly: when I am building an application that calls a language model, I specify the exact model I want in the PRD. Say I want Gemini 3.1 Pro, because it is the latest and most capable version for my use case. The coding agent writes the application, and everything looks correct in its summary. But when I inspect the actual code, it has substituted Gemini 2.5 or another older version because the agent’s training data predates the model I specified. It does not know the newer model exists, so it silently replaces it with one it recognizes. This is not a rare edge case. It has happened to me many times, across different coding agents and different model families. If I had not looked at the code, the application would have run fine, just with the wrong model powering it, producing different results than I intended. You do not need to understand every line. But you need to find the line that specifies which model is being called and confirm it matches what you asked for.

The emphasis on simplicity supports this dynamic. The less complex the build, the easier it is for you to follow what is happening and maintain control of the process. Until you have experience navigating that complexity, start with the smallest functional version of your idea and validate that it works before expanding.

Then you vibe

The “vibing” part, the part the term actually describes, comes last. It is what happens once you have a functional product and begin interacting with it, making adjustments, tweaking the interface, refining the behavior. This is genuinely enjoyable and genuinely productive, but it represents the final phase of a four-stage process, not the whole process.

So the actual sequence is: design thinking, then PRD, then building, then vibing. The term “vibe coding” captures only the last stage.

The essential toolkit

For those entering this space, the number of tools can feel overwhelming. But the core stack is smaller than it appears, and each tool occupies a distinct role.

Plain text and Markdown. Understanding how to work with plain-text files, specifically Markdown formatting, is foundational. This is how you will communicate with AI systems, structure documentation, and organize your thinking.

VS Code. You need an editor for working with Markdown and other plain text files, and VS Code is the standard. It is free, renders Markdown previews as you write, and integrates directly with GitHub and AI coding tools like Claude Code. If you are writing PRDs, editing configuration files, or reviewing code that an AI agent has generated, this is where that work happens.

LlamaParse. If Markdown is the foundational format, the immediate question is how to get your existing documents into it. Most professional documents, PDFs, Word files, and PowerPoints are not structured for AI consumption. LlamaParse, from LlamaIndex, is the best tool I have found for converting them into clean Markdown while preserving tables, headings, and structure. It offers 10,000 free pages per month, which is more than sufficient for development work.

GitHub. Version control is not optional. Your projects will evolve, break, and need to be rolled back. GitHub handles this, and AI coding tools integrate directly with it.

OpenRouter. This is my preferred service for model selection during development. It gives you access to a wide range of models through a single interface, which means you can quickly test different models for your specific application without managing multiple API accounts.

Claude Code. This is my primary development tool, and I use it for all my building work. It is the coding agent that turns a well-written PRD into a working application. Cursor and Google Antigravity are also strong options in this space; all three represent the new category of AI coding agents that can plan, execute, and iterate on code from natural language instructions.

Streamlit. When I need a simple interactive application, Streamlit is the fastest path. It lets you build functional web interfaces in Python with minimal frontend work.

Supabase. This one sounds more complicated than it is. Supabase is just a cloud database for working with data. When your project needs to store, retrieve, or manage structured information, Supabase provides a managed PostgreSQL instance with an accessible interface and good documentation.

This may sound like a lot of tools. But each one represents a major category: text formatting, editor, document conversion, version control, model access, coding agent, application framework, database, and together they form a toolkit that can address a remarkably wide range of problems. Build familiarity with these tools and their competitors, and you have a foundation that scales with your ambitions. And here is a practical tip: if you have a project in mind, drop this entire tool stack into your coding agent and ask it which ones are relevant to what you are trying to build. Let the AI help you navigate the choices.

The real skill

The term “vibe coding” is catchy, and the vibing part is real. But the skill that determines whether your project succeeds or fails is not the ability to prompt a coding agent. It is the ability to clearly define a problem, ruthlessly constrain your scope, and communicate your design in writing before the first line of code is written. The coding agent is powerful. What it needs from you is clarity about where to direct that power.

I use it every day. Both of those things can be true at the same time.

This post focuses on one of many problems I have built OpenClaw workflows to address. I am starting here because solving this single problem has been worth every resource I have invested in the system as a whole. That is my own personal return on investment. Everything else Chuck does for me is a bonus.

You may not feel the need to have an AI agent in your life that manages this kind of work. You may already have analog systems that handle it. If so, you are a better person than I am. You have the strength of Chuck Norris, and I do not. I named my OpenClaw after Chuck Norris. It hasn’t slept since.

Brittle systems and broken workflows

I teach in social work and have become involved in the inaugural AI Law & Policy Clinic at the University of Michigan’s Law School (more on this activity is forthcoming). I consult. I help manage a research lab. I have multiple active manuscripts, contracts, and tool-development projects at any given time. The technology landscape across these domains is advancing at a pace that makes it genuinely difficult to stay current. I become aware of new tools, models, frameworks, and techniques in a dozen different contexts throughout any given week. Some arrive through information flows I have deliberately cultivated: r/LocalLlama on Reddit, specific YouTube channels, targeted newsletters. Some surface in conversation with colleagues or students. Some appear on LinkedIn. Some arrive by email.

The problem is not finding information. The problem is that I find it in fragmented contexts, at inconvenient moments, and then lose track of it. I read about a promising small model while standing in line. A student mentions a tool I have not heard of during class. A colleague forwards a link I mean to investigate later. None of these things gets captured in any systematic way, and by the following week, half of them have dissolved into the general noise. I am facing information overload.

I have tried to solve this before. Note-taking apps. Bookmarking systems. Manual spreadsheets. Automated email filters. Every one of these solutions was brittle. They required me to stop what I was doing, switch contexts, and perform a deliberate act of information management at the exact moment I was least likely to do it. The systems worked for a week, maybe two, and then the friction won. I stopped maintaining them. The spreadsheet went stale. The bookmarks accumulated unread. The carefully tagged notes became a graveyard of good intentions.

This is the pattern that matters: the workflows I need are not complex, but they are repetitive, they span multiple input channels, and they break the moment they depend on me to perform a manual step at the point of capture. What I needed was a system that meets me where I already am, in my messaging apps, in my email, and handles the organization in the background without requiring me to change my behavior.

The setup

My OpenClaw instance runs on a cloud VPS hosted on Hostinger. A VPS is a virtual private server; essentially, a small, lightweight computer in the cloud that stays on all the time. It sounds complicated to set up, but I used Claude Code to configure the entire OpenClaw framework, which handled the installation, dependencies, and configuration files. The VPS gives me a dedicated, isolated environment where my agent runs independently of my personal devices.

The agent needs a language model to process requests and generate responses. I use OpenRouter as my model provider because it lets me select from dozens of different models, including free tiers, and swap between them without reconfiguring anything. My goal is to find the smallest model that performs adequately for each workflow. Tasks like classifying links, summarizing tools, and generating structured digests are not computationally demanding. A capable small model handles it well. The commitment to finding the smallest viable model is not just about cost. It reflects a genuine concern for environmental efficiency. If a 7-billion-parameter model can classify and summarize a forwarded link as well as a model 100 times its size, there is no justification for the larger one. With OpenRouter, I can test this directly; swap models, compare outputs, and settle on the lightest option that meets the standard.

Meet Chuck

I recently renamed my OpenClaw agent Chuck, after the late Chuck Norris, who passed away a few days ago. I grew up watching his films and TV shows. The internet honored him with the Chuck Norris Facts meme that proves he is immortal. Renaming the agent felt like my own small tribute.

In case you don’t know about Chuck Norris, here are a few facts that are easily verifiable.

Chuck Norris doesn’t read books. He stares them down until he gets the information he wants.

Chuck Norris doesn’t sleep. He waits.

Chuck Norris counted to infinity. Twice.

When Chuck Norris enters a room, he doesn’t turn the lights on. He turns the dark off.

A persistent agent that never sleeps, processes everything you throw at it, and organizes information by force of will? The name was obvious.

How it works: the technology digest

The technology digest is one of several workflows I have built with Chuck, but it is the simplest to explain and the one that illustrates the core pattern. My design principle is straightforward: get the content to Chuck, and let Chuck do all the work.

I interact with Chuck primarily through Telegram, a messaging app I chose specifically to keep these workflows completely separate from my personal and professional communications on WhatsApp. Within Telegram, I have organized Chuck’s functions into separate channels: one for the technology digest, one for annual evaluation tracking, one for travel planning, and others as I continue to extend the system. Everything is compartmentalized by domain.

For the technology digest, I can send a voice note describing something I encountered. I can forward a link. I can type a quick message. Whatever is easiest in the moment is what I do, and Chuck processes it. For information that arrives by email, I use AgentMail, a Y Combinator-backed platform that provides dedicated email inboxes for AI agents. When I receive an email about a new tool or resource, I forward it to Chuck’s AgentMail address, and it enters the same processing pipeline.

In the backend, Chuck maintains a lightweight SQL database that classifies and stores everything I send. I have never been formally trained in SQL, but I use it constantly now because the barrier to working with structured data has effectively collapsed. Chuck built the database schema, maintains it, and queries it; all orchestrated through natural language. I did not write a line of code for any of this. Instead, I organized the flow of information, defined the decision rules, and let the AI system build the pipeline — this is just straight-up task decomposition.

Chuck then generates a weekly email digest, formatted and tailored to my interests, that summarizes everything I have collected. It does not just list the items. It knows what I care about, understands my workflows, and explains why each tool or resource might be useful in my specific context. If something falls out of favor or proves irrelevant, I tell Chuck to exclude it. The system learns from these corrections over time.

Here is what the Table of Contents looks like:

And here is an example entry in my personalized digest.

Beyond the digest

The technology digest was the first workflow, but Chuck now handles several others. Each follows the same pattern: a repetitive task that spans multiple inputs, requires classification and storage, and historically consumed far more of my time than its intellectual complexity warranted.

Annual evaluation tracking. Every year, I spend an unreasonable amount of time reconstructing my activities for annual review. Historically, this meant combing through a year’s worth of email, calendars, and documents to assemble a record of publications, presentations, committee service, teaching activities, consulting engagements, and everything else that constitutes an academic workload across social work, law, and consulting. Chuck now maintains a separate database for this. When I complete a manuscript review, give a talk, join a committee, or finish a consulting project, I forward a brief note or message to Chuck through the appropriate Telegram channel.

Chuck classifies the item as research, teaching, or service; saves the artifact; and generates a summary. When annual review season arrives, the record is already assembled. This is not a task that requires advanced judgment. It requires consistent, low-friction capture and classification – exactly the kind of work that an LLM following simple rules handles well. (If any colleagues have an analog system that is as efficient as Chuck, I would love to hear about it.)

Travel planning. Another channel, another workflow. Chuck manages itineraries, tracks confirmations, and organizes logistics. Again, the same pattern: information arrives from multiple sources, needs to be consolidated and structured, and benefits from having a persistent agent maintaining the record.

Each of these workflows runs through its own Telegram channel, which keeps the domains cleanly separated. I am not building one monolithic system. I am building a collection of small, focused workflows that each solve a specific problem.

Extending to students

Once I had the technology digest running for myself, the next step was obvious. My law students are entering a technology landscape that is equally overwhelming and far less familiar to them. They need the same kind of curated, structured awareness of what tools exist and why they matter, but written in plain language and calibrated to their level of experience.

I am now working with Chuck to generate a second newsletter, adapted from the same underlying database, that provides a plain-language summary targeted to students who are just getting started with these technologies. The same information, rewritten for a different audience, was automatically produced by the same pipeline. I orchestrated the entire thing with natural language instructions. I realize that some of my technologies of interest will not be relevant to law students, so I am helping Chuck understand how to classify them using a simple set of rules. I am also thinking these technologies are relevant to social work students, but that would require a different classification approach. I can build all of these rules by simply having a conversation with Chuck – no coding required.

Why the security risk is manageable

The security warnings about OpenClaw are legitimate. But risk is a function of what you expose, and I have been deliberate about minimizing exposure.

Everything Chuck produces lives in accessible, portable formats: Markdown files and SQL databases. I can back up the entire system, inspect any file in a text editor, or migrate the data to another platform if needed. Nothing is locked inside a proprietary format. Nothing is irreplaceable.

More importantly, none of these workflows involves sensitive data. If someone compromised my Technology Digest database, they would find a curated list of AI tools and model releases. If they accessed my annual evaluation tracker, they would find a record of talks I gave and papers I reviewed. These are not confidential client records or financial data. The consequence of a worst-case breach is that someone sees my reading list. I would be delighted if people found some of these things useful.

I have backups for everything. The system is isolated from my primary work. I built these workflows on problems where, if something went awry, the cost would be inconvenience, not harm.

The automation standard

Not everything should be automated this way. The test I apply is simple: can this workflow be reduced to a set of rules clear enough for an LLM to follow reliably? If the task requires nuanced professional judgment, it does not belong here. If it requires consistent application of straightforward classification criteria, is this item research, teaching, or service?; then it is a candidate.

The workflows I have built with Chuck are not intellectually demanding. You do not need an advanced degree to classify a conference presentation as a research activity or to summarize a forwarded link about a new language model. What these tasks require is consistency and low friction across many small inputs over time. That is precisely what a persistent agent provides and precisely what I, as a human with a full schedule, was failing to provide for myself.

What this is and what it is not

I am not suggesting everyone needs an OpenClaw instance. If you do not have a real problem that demands this kind of solution, do not bother. This is not a tool to play with. It is infrastructure for people who have specific, recurring workflows that current tools handle poorly.

What I am suggesting is that the underlying pattern, a persistent, locally controlled AI agent that connects to your existing communication channels and takes autonomous action on structured tasks, is not hype. It is a genuinely new capability. And unlike my previous attempts with note-taking apps and spreadsheets, it is not brittle. The system has been running for weeks, and I continue to extend it because adding a new workflow only requires a natural-language conversation with Chuck about what I want.

The entire system was built without writing code. Every component, the database schemas, the classification logic, the digest formatting, the student adaptation, and the annual evaluation tracker, was defined through conversation with an AI agent. I can’t tell you how much time I have squandered over the years trying to code brittle, disconnected systems to create these workflows. For me, we have a fundamentally different shift in how we think about technology. That is the real shift. Not that the tools are impressive, but that the distance between having an idea for a workflow and having a working implementation of it has collapsed to nearly zero.

Chuck Norris counted to infinity. Twice. Chuck, the agent, just counts my forwarded links, classifies them, and sends me a newsletter. Less dramatic, but considerably better than swimming in a sea of information and trying to remember stuff.

This approach may not get you the information you need to move forward.

A stakeholder demo is not a performance. Rather, consider the demo a structured conversation with a working prototype as the focal point. You are not there to impress anyone. You are there to learn things about your tool that you cannot learn on your own, things that only someone embedded in the actual workflow can tell you. Everything in this guide serves that purpose.

I have organized this around two questions that students in the AI Law and Policy Clinic should consider before presenting their work. The first is: what do you actually need to know before the meeting starts? The second is: how do you run the meeting so that you walk out with feedback you can act on?

What you need to know before the meeting

Know your audience’s relationship to the problem

The standard advice is “know your audience,” but that is too vague to be useful. What you need to understand is how your stakeholder currently experiences the problem your tool is meant to address. Three dimensions matter.

First, technical fluency. How much can you assume about their understanding of what your tool does? If the answer is very little, and it usually is, then your language during the demo has to change entirely. You do not explain the retrieval pipeline. You explain what happens when they upload a document and ask a question.

Second, the current workflow. What does the stakeholder currently do to accomplish this task without your tool? How long does it take? Where are the friction points? This is the baseline your tool is measured against. If you do not know it, you cannot interpret their feedback. A stakeholder who says “this is interesting” may mean “this solves a real problem I have every day” or “this is a neat trick that has nothing to do with how I actually work.” You need enough context to tell the difference.

Third, institutional constraints. What rules, norms, or resource limitations govern how work gets done in their setting? A tool that requires a login that their IT department will not approve is dead on arrival, regardless of how well it performs. A tool that assumes reliable internet access in a field office that lacks it is solving the wrong problem. What about data security, such that no data can ever leave the premises? These are things you should know before the meeting, not things you discover during it.

If you walk in without this knowledge, you will spend the meeting learning things you should have already understood, and you will lose the opportunity to get feedback on the tool itself.

Understand your own technology

This one is uncomfortable, but it matters. Using a no-code builder does not exempt you from understanding the technology you have built. If you assembled a RAG-based chatbot using a platform that handles the infrastructure for you, such as Josef Q, you still need to be able to explain, at a conceptual level, what happens between the moment a user submits a query and the moment an answer appears on screen.

I am not talking about mathematics. I am talking about the architecture. Documents are split into chunks. Those chunks are converted into numerical representations and stored. When a user asks a question, the system finds the chunks most semantically similar to the query and passes them to a language model, which generates a response grounded in that retrieved text. That is a paragraph. You should be able to deliver it clearly and confidently because the stakeholder will ask questions that require clarity and confidence.

Here is why this matters practically. Document preparation has consequences that are invisible unless you understand chunking. A poorly formatted 200-page PDF will produce worse answers than a set of clean, well-organized documents, and if you do not understand why, you will blame the model when the problem is the corpus. Stakeholder implementation questions require real answers. When a legal aid organization asks what happens when a statute changes and the documents need updating, you need to explain the update process. When they ask where their data is stored, you need to know whether the platform sends data to a third-party API or keeps it local. These are not hypothetical questions. They come up in every stakeholder meeting I have observed.

There is a deeper issue as well. A RAG system that retrieves the wrong chunks will still produce a fluent, confident-sounding answer. If you do not understand retrieval, you will not know how to test for this. You will demo a tool that looks like it works, and miss that it is pulling from the wrong section of the wrong document. The stakeholder will not catch this either, because the output reads well. This is exactly the kind of failure that erodes trust once someone discovers it in practice.

A chatbot built on a RAG system is not “an AI that reads your documents.” It is a system that requires document curation, ongoing maintenance, monitoring, and periodic evaluation. If you cannot articulate this, the stakeholder will underestimate what adoption actually requires, and the tool will fail after you hand it off.

You should be able to explain, in plain language, what your tool does at each stage of its process, what can go wrong at each stage, and what decisions you made that affect its performance. If you cannot do that, you are not ready to demo.

Be honest about what you are showing

This is the principle students are most likely to skip, and it is the one that matters most for the stakeholder relationship. You need to walk into the meeting knowing whether you are showing a proof of concept or a production-ready tool, and you need to say so explicitly.

A proof of concept demonstrates that a workflow can be supported by the technology. It does not demonstrate that the tool should be deployed. If that is what you have, say so: “This shows that the approach is viable. Moving to production would require formal evaluation, corpus maintenance planning, integration work, and resources that go beyond what I can provide in this clinic.” You should be able to provide a rough sense of what lies between the prototype and what a client or litigant would actually interact with. Not a budget, but an honest description of the remaining work.

If you are claiming something closer to production readiness, the standard is substantially higher. In a legal context, the eyeball test is not sufficient. You cannot watch the tool produce a plausible-looking output and call it validated. Every critical task the AI performs needs systematic evaluation: test cases that cover clear inputs, boundary cases, and out-of-scope queries; defined acceptable error rates grounded in the consequences of getting it wrong; and verification at each step of the pipeline, not just the final output. I have written about this evaluation framework in detail in a previous post. A student claiming production readiness should be able to describe the evaluation they conducted, not just the features they built and observed.

The reason this matters for the stakeholder meeting is straightforward. Stakeholders will often assume that a working demo means the tool is ready. It is your responsibility to set expectations accurately. Overselling a proof of concept as production-ready creates a trust problem that is very difficult to recover from. And underselling production-ready work by failing to describe the evaluation behind it leaves the stakeholder without the confidence they need to advocate for adoption within their organization.

Anticipate concerns and be prepared to speak to them

Stakeholders in legal settings will have questions that go beyond the tool’s features. The most important one is some version of: can we trust this? You need to be ready for it, and the answer cannot be “it seems to work well.”

This means you have to be able to explain clearly where you are in the evaluation process. What have you tested? How thoroughly? What do you know about the tool’s accuracy, and what do you not yet know? If your honest answer is that you have tested it informally but have not conducted a systematic evaluation, say that. The stakeholder can work with honesty. What they cannot work with is vague assurance.

A common mistake is investing heavily in an impressive-looking user interface while neglecting the harder work of testing accuracy and consistency. A polished interface creates the impression that the tool is further along than it actually is. If the stakeholder asks a question the tool has not been evaluated against, and the tool produces a confident but wrong answer, the visual polish makes the failure worse, not better. It signals that you prioritized appearance over reliability.

This is particularly important to understand when it comes to tools built on large language models. These are probabilistic models. They do not return the same answer every time in the same way. A tool that produces the correct response to a query today may produce a slightly different, and potentially incorrect, response to the same query tomorrow. This is not a bug in your implementation. It is a fundamental characteristic of the technology. It means that testing a tool once and seeing a correct answer is not sufficient. You need to verify that the tool produces correct and consistent answers across repeated queries, across varied phrasings of the same question, and across the full range of inputs it is likely to encounter in practice.

If you cannot speak to this, the stakeholder’s trust concerns will go unanswered, and unanswered trust concerns are where adoption goes to die. You do not need to have all the answers. You do need to demonstrate that you understand the questions and have a plan for addressing them.

How to run the meeting

Present the MVP and nothing else

The technologies available for building AI tools make it trivially easy to add features. A sidebar here, a summary function there, a second input modality that seemed interesting. Each addition feels like progress. It is not. Each addition is a new thing that can break, a new thing you have to explain, and a new thing the stakeholder has to form an opinion about. You are splitting their attention across features rather than concentrating it on the core function the tool exists to perform.

Present the smallest version of the tool that performs the core task end-to-end. One form, one workflow, one use case. Not a vision of what the tool could become, but a working instance of what it does right now. The stakeholder needs to react to something concrete, not to a roadmap.

If you built extra features, do not hide them. Just do not demo them yet. The stakeholder meeting is about validating the core workflow. Once that is validated, you earn the right to expand.

Demo the workflow, not the tool

Students naturally demo tools in isolation. They open the interface, show its features, and walk through its capabilities. What they should do instead is demo the tool inside the workflow it is meant to support.

If the tool helps a legal aid attorney draft a section of a filing, do not demo the tool in a vacuum. Walk through the attorney’s actual process: they receive the case, they review the relevant documents, they reach the point where your tool enters, they use the tool, they review the output, and they continue with the rest of their work. The stakeholder should see where the tool enters and exits their existing process.

This is how you surface the real friction points. A tool that works perfectly in isolation may create problems at the integration points: the output format does not match what the attorney needs for the next step, the tool assumes information the attorney does not have at that stage of the workflow, or the review process takes longer than doing the task manually. These are things a feature demo will never reveal. A workflow demo exposes them immediately.

Structure the feedback you are asking for

Do not present your tool and then ask, “so what do you think?” The stakeholder will give you polite generalities. They are experts in their domain, not in giving product feedback. You have to structure the conversation so that their expertise gets channeled into observations you can act on.

Come with specific questions prepared in advance. Does this output match what you would produce yourself, and where does it diverge? At what point in your current workflow would you actually use this? What would stop you from using this tomorrow? Is there anything in the output that would be unacceptable in your professional context, whether that means filing it with a court, submitting it to an agency, or sharing it with a client? Based on what you have seen, what would you need to see before you would be comfortable using this with real cases?

That last question ties the feedback conversation back to the proof-of-concept question. It tells you what the stakeholder’s deployment threshold actually looks like, in their own terms, and gives you a concrete target for the next iteration.

Listen like a researcher, not a developer

When stakeholders give feedback, students have two common failure modes. The first is defensiveness: explaining why the tool did what it did, justifying design choices, narrating the technical constraints. The stakeholder does not need this. They are telling you about their experience. Your job is to record it.

The second is real-time problem-solving. “I can fix that right now,” or “that would be an easy add.” This derails the conversation from feedback collection into a live debugging session. The impulse is understandable, but it is counterproductive. The meeting is for gathering information. The building happens after.

The discipline is straightforward. Take notes. Ask follow-up questions that deepen your understanding of their reaction. Resist the urge to respond with solutions. You are conducting a structured interview, not a co-design session. The synthesis and prioritization happen after the meeting, not during it.

Translate feedback into the next concrete step

After the meeting, convert what you heard into a specific, bounded next action. Not a feature list. A single iteration. What is the one thing you now know needs to change, and what does the next version of the MVP look like with that change made?

Scope creep happens when students try to incorporate everything they heard at once instead of prioritizing. The stakeholder may have given you ten observations. Three of them are about the core workflow. Two are about nice-to-have features. Five are about things that matter for production but not for the current stage of development. Sort them. Act on the ones that address the core function. File the rest for later.

The cycle then repeats: a constrained MVP, demoed inside a real workflow, with structured feedback that produces a single next iteration. This is how tools get built. Not in one inspired sprint, but through repeated contact with the people who will actually use them.

These are reasonable questions, but they are the wrong first questions to ask. The first question is: how will you know if this thing works?

This is not a philosophical point. It is a project management discipline that, in my experience, separates AI tools that actually get deployed from those that stall out in perpetual prototyping. If you cannot articulate what correct performance looks like before you start building, you will not be able to identify incorrect performance after you finish. You will have a system that produces outputs, and no principled basis for determining whether those outputs are acceptable.

An important distinction before I go further. When you are learning to build AI tools, the natural and pedagogically sound sequence is to build first and then evaluate what you built. This is how you develop intuition for what the technology can and cannot do. You need to see a language model retrieve documents, populate form fields, or draft summaries before you can design meaningful criteria for judging whether it did those things well. Courses and clinics that structure the work this way are doing something right; you cannot evaluate a capability you have never observed.

But the sequence changes when the context changes. When a system moves from a learning exercise to a tool that someone will rely on, a self-represented litigant navigating a court process, a legal aid attorney making triage decisions, a caseworker acting on extracted data, evaluation design moves to the front of the process. Not because the learning sequence was wrong, but because the stakes are different. In a classroom, a system that produces a plausible but incorrect output is a teaching moment. In a legal aid office, it is a missed deadline, an incorrect filing, or a client who does not return. The discipline I am describing in this post is what the work looks like when you are building for deployment, and it is the standard that anyone learning to build AI tools is working toward.

Evaluation is a design activity

Most people think of evaluation as something that happens at the end. You build the system, you run some tests, and you see how it did. This is backwards. Evaluation criteria should be the first deliverable of any AI development project, not the last.

Here is what I mean concretely. Before you write a line of code, before you select a model, before you design an interface, you should produce two documents. The first is a functional specification that defines what the system does and does not do. The second is an evaluation benchmark that defines, for every capability in the specification, how you will test whether the system performs that capability correctly.

These two documents are developed together because they are logically dependent. You cannot write a meaningful test case without knowing what the system is supposed to do. And you cannot claim the specification is complete until you can articulate how each requirement would be verified. If you find yourself writing a specification requirement for which you cannot imagine a concrete test, that is a signal that the requirement is too vague to build against.

The reason this matters for people new to AI development is that language models produce fluent, confident output regardless of whether that output is correct. Unlike traditional software, where a bug typically produces an obvious error or crash, a language model that fails does so by generating plausible-sounding text that is wrong. If you do not have predefined criteria for what “right” looks like, you will not catch it.

The two-person test

I use a heuristic I return to constantly, both in my own work and in consultations: for any task you want the AI to perform, ask whether two trained humans could perform it independently and arrive at the same answer.

This is not a metaphor. It is a practical test you should actually apply. Take the task you want to automate. Write down the inputs. Give those inputs to two qualified people, separately, without letting them communicate. Compare their outputs. If they produce the same result, the task is a strong candidate for AI automation. If they frequently disagree, you have a problem that needs to be solved before any technology enters the picture.

The logic is straightforward. We want AI systems to be fungible with competent human decision-making. That is the standard. An AI that extracts the correct filing deadline from a court order is performing a task that a competent paralegal would also perform correctly, given the same document. An AI that identifies which SCAO form is required for a particular motion type is performing a task that two trained legal aid staff would agree on. These are tasks with verifiable answers, and verifiable answers are what make reliable systems possible.

When the two humans disagree, it means the task involves judgment that is not fully captured by the available inputs, the existing rules, or both. That does not mean the task cannot be automated; it means it requires substantially more work before it can be. You need to define the decision rules more precisely. You need to identify the sources of disagreement and resolve them through clearer criteria, better training, or a narrower scope of tasks. Skipping this step and handing the task to an AI produces a system that is confident and inconsistent; exactly the kind of system that erodes trust.

Why rubric-based evaluation is harder than it looks

This principle has a direct implication for how you evaluate AI systems, and it is the place where I see people get into trouble most quickly. When an AI performs a judgment task, drafting a legal memo, summarizing a client’s situation, or identifying the strongest arguments in a case file, the natural instinct is to evaluate the output with a scoring rubric. Rate the quality of the summary on a 1-to-5 scale. Score the memo’s accuracy, completeness, and organization. This feels rigorous. It produces numbers. Numbers feel like measurement.

The problem is that a rubric creates the appearance of objectivity by attaching numbers to judgments, but if the underlying judgments are subjective, the numbers inherit that subjectivity. And for most AI judgment tasks worth evaluating, the underlying judgments are subjective.

Consider an AI system that drafts a section of a country conditions report for an asylum case. You ask two experienced immigration attorneys to evaluate the output using a rubric. One dimension is “comprehensiveness”: does the report address the relevant country conditions for this particular claim? The rubric defines “5” as “thoroughly addresses all material conditions” and “1” as “substantially incomplete.” Attorney A scores it a 4; Attorney B scores it a 3. Neither is wrong. They weigh the same evidence differently. They have different thresholds for what constitutes “thorough.” The rubric channels their judgment; it does not eliminate their disagreement.

Now consider a different task: the AI extracts the respondent’s date of entry from an immigration filing and populates a form field. You ask two paralegals to check whether the extracted date is correct. They will agree every time, because the task has a verifiable answer. The date is either right or it is not.

The difference between these two evaluation scenarios lies in whether tasks are ready for reliable evaluation or require substantially more groundwork. When humans cannot agree on how to score an AI’s output, the evaluation itself becomes unreliable. You end up with scores that reflect evaluator variability as much as system performance. You cannot distinguish a system that is performing well from one that is performing poorly, because your measuring instrument is imprecise.

This does not mean judgment tasks are off-limits for AI, or that rubrics are useless. It means you need to understand the reliability of your evaluation method before you trust its results. The practical step is to have multiple evaluators independently score the same set of AI outputs, then measure their agreement. If two attorneys applying your rubric to the same outputs agree at a high rate, the rubric is producing a reliable signal. If they frequently disagree, you do not have an AI performance problem; you have an evaluation design problem. The rubric needs sharper criteria, the scoring anchors need concrete examples, or the task needs to be decomposed into smaller components that humans can assess more consistently. Fix the evaluation instrument before drawing conclusions about the system it is supposed to measure.

What counts as “high enough” agreement depends entirely on the task and its consequences, which brings us to a question that every AI project must answer explicitly.

What is an acceptable error?

No system is perfect. This is true of AI systems, and it is equally true of the human performance against which those systems are compared. Humans make mistakes. Some of those mistakes are systematic; a consistent bias in how a particular attorney interprets a particular type of claim. Some are random; an error introduced by fatigue, distraction, or the simple reality that attention is finite. Human error is a baseline fact of every professional workflow, not an aberration.

AI systems also make mistakes, and for different reasons. A language model may misinterpret a query, retrieve the wrong document, or generate a confident response based on a subtle misreading of context. These errors look different from human errors, but they are errors nonetheless. The question is not whether the AI will make mistakes. It will. The question is what error rate is acceptable for the specific task it is performing, and how that rate compares to the alternative.

This is where I see people adopt one of two unrealistic positions. The first is the expectation that the AI must achieve perfect performance before it can be trusted. This standard is not applied to the humans currently performing the same work, and applying it exclusively to AI creates a threshold that no system, human or machine, can meet. A legal aid organization that processes hundreds of intake forms per month does not expect its staff to make zero errors. It expects a manageable error rate, with review processes in place to catch mistakes before they cause harm. Holding an AI tool to a standard of perfection that the existing workflow does not meet is not rigor; it is a reason to never deploy anything.

The second unrealistic position is the assumption that because AI is “just a tool,” any error rate is acceptable as long as a human reviews the output. Human review is essential, but it is not magic. If the AI produces errors at a high rate, the review process becomes the bottleneck, and the efficiency gains that justified the tool disappear. Worse, reviewers develop automation complacency; when most outputs look correct, attention drifts, and errors that a fresh reading would catch slip through. A tool that requires the reviewer to redo the work is not a tool.

The realistic position sits between these extremes, and it requires you to think carefully about two things: the consequences of error and the current baseline.

Some tasks have very low tolerance for error because the consequences are severe and irreversible. An AI system that populates court filing dates has to get them right, because an incorrect date can result in a missed deadline, a default judgment, or a waived right. For tasks like these, the acceptable error rate is effectively zero for the specific fields that carry legal consequences. That does not mean the system is useless if it occasionally makes mistakes elsewhere; it means the high-stakes components need to be evaluated to a higher standard and may require mandatory human verification, regardless of the overall system's accuracy.

Other tasks tolerate more error because the consequences are recoverable or because the task itself involves variable human performance. An AI system that drafts an initial summary of a client’s situation for attorney review does not need to produce a perfect summary every time. It needs to produce a summary that is sufficiently accurate to serve as a useful starting point, and the attorney’s review is a built-in correction step, not an afterthought. For tasks like these, the relevant question is whether the AI’s output saves the attorney enough time to justify the occasional need for correction.

The practical framework is this: for each task your system performs, define the acceptable error rate before you begin evaluation. Ground that definition in two reference points. First, what is the consequence of an error in this task? A wrong filing date has different consequences than a slightly incomplete case summary. Second, what is the current human error rate for this task, if it is known or estimable? If human intake staff currently misroute 8% of cases to the wrong legal category, an AI system that misroutes 5% is an improvement, even though it is imperfect. If human staff currently achieve near-zero error on date extraction, the AI needs to match that standard.

Defining acceptable error forces a conversation that many project teams avoid: what are we actually trying to achieve with this system, and how good does it need to be to achieve it? That conversation is far more productive when it happens during evaluation design than when it happens after someone notices the system got something wrong in practice.

The core is easy; the boundaries are where systems fail

Here is something that consistently surprises people when they start writing evaluation benchmarks: the obvious cases are not the problem. If you build a legal document retrieval system and test it with a query that matches a single statute perfectly, the system will get it right. If you build a form classifier and test it with a filing that is unambiguously one form type, the classifier will get it right. The core of most tasks is straightforward, and AI systems handle the core well.

The work is at the edges.

Edge cases are the inputs that sit at the boundary of your system’s defined capabilities. A query that partially matches three legal provisions but perfectly matches none. A client situation that could require any of several different court forms. A request that is almost but not quite outside the system’s scope. These are the cases where performance degrades, where confident wrong answers appear, and where users lose trust.

What makes edge cases critical for evaluation design is that you cannot identify them after the fact. If you wait until the system is built and then start testing, you will naturally gravitate toward the clean cases; the ones you already have in mind, the ones that confirm the system works. The messy cases, the ones that probe boundaries and expose ambiguity, require deliberate effort to construct. They require you to sit with the specification and ask: where does this get hard? Where would a reasonable query fall between two categories? Where would two domain experts disagree about the correct response?

This connects directly to the two-person test. For the core cases, two trained humans will agree easily. A query asking for the form required to file a motion to set aside a default judgment returns that form; there is nothing to argue about. But consider a self-represented litigant who describes a situation involving both a landlord-tenant dispute and potential housing discrimination. Does the system retrieve resources for a standard eviction defense, for a fair housing complaint, or for both? Should it flag that the situation may require coordinated filings in different forums? Two legal aid attorneys might handle this intake differently, and that disagreement is precisely the information you need before you build the system, not after a self-represented litigant encounters it without any attorney present.

The practical discipline is this: for every capability in your specification, write test cases at three levels. First, the clear cases where the correct answer is obvious. Second, the boundary cases where reasonable people might disagree about the correct response. Third, the cases that should be declined or redirected because they fall outside the system’s scope. The boundary cases are the hardest to write and the most valuable. They force you to make decisions about your system’s behavior in the gray areas; decisions that will otherwise be made implicitly by whatever the model happens to generate on a given day.

If you skip this step, you will end up with a system that performs impressively in demonstrations and unpredictably in practice. Demos use clean inputs. Real users do not.

Every task in the pipeline needs verification

A further practical point that people new to AI development frequently overlook: a multi-step system is only as reliable as its least reliable step, and errors early in the pipeline compound.

Consider a legal document automation system that first identifies the type of court filing required, then extracts relevant facts from the client’s intake narrative, then populates the appropriate form fields. If the first step misidentifies the filing type, the extraction step will look for the wrong facts, and the completed form will be internally consistent but wrong for the client’s situation. Each subsequent step performs correctly on its own terms; given its inputs, it does what it was designed to do. But the inputs were corrupted upstream, and the downstream steps have no way to detect that. A self-represented litigant receives a professionally formatted document that files the wrong motion.

This is why evaluation cannot be applied only to the final output. Each step in the pipeline needs its own verification. You need test cases for the filing type classifier. You need test cases for the fact extractor. You need test cases for the form population step. And you need integration test cases that verify the pipeline end-to-end because some failure modes only emerge from interactions between steps.

The practical takeaway: decompose your system into its component tasks and evaluate each one independently before evaluating the whole. If you discover that step two has a 15% error rate, that is something you need to address at step two, not something you will notice by staring at final outputs and trying to figure out why some of them feel wrong.

Getting started: using AI to design your evaluation

All of this can sound like a lot of work, and it is. But here is something worth knowing: the same AI tools you are learning to build can help you design the evaluation criteria for those tools. This is not circular. The AI is not evaluating itself. It is helping you, the human designer, think through the dimensions of evaluation that your system requires. You still make every decision. The AI accelerates structured thinking.

I have found this particularly useful in the early stages of a project, when a team knows what they want the system to do but has not yet articulated what correct performance looks like across the full range of inputs. A well-constructed prompt can walk you through the evaluation design process in a single session, producing a draft framework that the team can then review, challenge, and refine. It does not replace the domain expertise needed to determine whether a test case is realistic or whether a threshold is appropriate. It provides the scaffolding so that the domain experts’ time is spent on substance rather than on figuring out what questions to ask.

The following prompt is designed for this purpose. It embodies the principles from this post: evaluation as a front-end design activity, the two-person reliability test, edge case identification, acceptable error calibration, and pipeline decomposition. Copy it, replace the bracketed sections with your specifics, and use the output as a starting point for your evaluation design; not as a finished product.

You are an AI evaluation design consultant helping a legal services team develop evaluation criteria for an AI-assisted tool before development begins.

The tool we are building: [Describe your tool in 2-3 sentences. What does it do? Who uses it? Example: “A retrieval system that helps legal aid attorneys identify relevant court forms and filing instructions based on a client’s described legal situation. Users are staff attorneys and paralegals at a legal aid organization serving self-represented litigants.”]

The specific tasks the tool performs: [List each discrete task. Example: “1. Accepts a natural language description of a client’s legal situation. 2. Identifies the relevant court form(s). 3. Returns the form(s) with filing instructions and deadlines.”]

For each task listed above, work through the following evaluation design steps:

Step 1: The two-person reliability test. For this task, could two trained professionals (e.g., two experienced paralegals, two legal aid attorneys) independently perform it and arrive at the same answer? Identify which tasks would produce high agreement and which would produce frequent disagreement. For tasks with likely disagreement, explain what makes the judgment ambiguous and suggest how the task could be scoped more narrowly or defined more precisely to improve agreement.

Step 2: Test case design at three levels. For each task, draft example test cases at three levels: (a) clear cases where the correct answer is unambiguous, (b) boundary cases where the correct response is genuinely uncertain or where the input partially matches multiple categories, and (c) out-of-scope cases that the system should decline or redirect. For the boundary cases, explain specifically what makes them boundary cases and what decision the team needs to make about how the system should handle them.

Step 3: Acceptable error analysis. For each task, assess the consequence of error. What happens if the system gets this task wrong? Distinguish between tasks where errors have severe, potentially irreversible consequences (e.g., wrong filing deadlines, incorrect form selection that leads to a rejected filing) and tasks where errors are recoverable (e.g., an imperfect initial summary that an attorney will review). Recommend an appropriate performance threshold for each task, grounded in the severity of consequences, and explain your reasoning.

Step 4: Pipeline dependencies. If the system performs multiple tasks in sequence, identify where an error in an earlier task would corrupt downstream tasks. Specify which tasks need independent evaluation checkpoints and which failure modes would only be visible through end-to-end testing.

Step 5: Evaluation method. For each task, recommend how the evaluation should be conducted. Distinguish between tasks that can be evaluated with an answer key (correct/incorrect) and tasks that require human judgment with a rubric. For rubric-based evaluation, flag the risk of evaluator disagreement and recommend how to measure and manage it.

Present the output as a structured evaluation design document that the team can review and refine. Flag any areas where you need additional information from the team to complete the framework.

The output of this prompt is a draft, not a deliverable. It will surface evaluation dimensions you had not considered, propose test cases that expose ambiguity in your specification, and identify pipeline risks you may have overlooked. It will also make incorrect assumptions, propose thresholds that are too high or too low for your context, and generate test cases that do not reflect how your users actually behave. That is fine. The value is in the structured starting point, not in uncritical adoption. Review it with your team, revise it against your domain knowledge, and use it to produce the evaluation framework that the team owns and stands behind.

The technology changes. The models improve. The principle does not: you cannot evaluate what you never defined, and you cannot trust what you never evaluated.

One exchange with a reviewer stands out. We were asked to clarify that AI models are not connected to the internet and do not inherently know about APIs or current documentation. At the time, that was a reasonable and accurate correction. The major LLMs operated within the boundaries of their training data and could not browse the web or access external services on their own.

That was roughly a year ago. The landscape has shifted considerably.

I still use APIs every day. They remain foundational infrastructure for anyone working with AI tools in research or practice. What has changed is that I almost never write API code by hand anymore. The AI tools themselves have become capable of handling that work. When I need to connect to a service, I describe what I need, and the model generates the code, tests it, and troubleshoots errors. The manual coding workflow we documented so carefully in that paper has been largely automated by the very technology we were writing about.

This is not a story about that article becoming obsolete. It is a story about the pace of change, and about why understanding these concepts matters even more now than it did then.

What an API actually does

An API is a structured way for one piece of software to talk to another. When you use a weather app on your phone, the app does not generate forecasts. It sends a request through an API to a weather service, receives data back, and displays it on your screen. The app and the weather service are separate systems; the API is the connection between them.

In research, APIs allow you to send data to a service and receive processed results. I use an API via OpenRouter to access various language models for tasks such as classification, translation, summarization, and extraction. The data leaves my computer, gets processed on a remote server, and comes back with the results. That basic request-and-response pattern is the same whether you are geocoding addresses through the U.S. Census Bureau, retrieving article metadata from PubMed, or sending text to an LLM for analysis.

The key concepts are straightforward. An endpoint is the specific web address where you send your request. An API key is the credential that authenticates you with the service. The request contains your data and instructions. The response contains the results. Every API interaction follows this pattern, regardless of the service.

What has changed: from hand-coded connections to MCPs

When we wrote the JSSWR article, connecting an LLM to an external service meant writing Python code to handle authentication, construct the request, send it to the endpoint, parse the response, and manage errors. It was not impossibly difficult, but it required a working knowledge of programming and a willingness to debug.

The emergence of the Model Context Protocol (MCP) has substantially changed this dynamic. MCP is a standardized way for AI models to connect to external tools, data sources, and applications. Rather than writing custom code for each connection, MCP provides a common framework that any compatible tool can use to plug into an LLM.

Think of it this way. APIs are like having a phone number for every business you want to contact; you need to know each number, the right way to ask for what you want, and how to interpret the response. MCP is more like a universal switchboard operator. You tell the operator what you need, and it handles the connection, the protocol, and the translation for you.

In practical terms, this means I can connect Claude to my Google Calendar, Google Drive, Gmail, Notion, and a growing number of other services through MCPs. The LLM can then read my documents, check my schedule, search my email, or pull data from a database to complete a task. I did not write code to make any of these connections. I enabled them through a configuration interface, and the model handles the rest.

This is a significant development. The barrier to integrating AI with other tools has dropped from “you need to know Python” to “you need to know what you want to accomplish.”

Why both concepts still matter

MCPs have not replaced APIs. They sit on top of them. When Claude connects to Google Drive through an MCP, an underlying API still handles the actual data exchange. MCP standardizes and simplifies the connection layer, but the API is still doing the work.

This distinction matters for several reasons. First, not every service has an MCP integration yet. If you need to connect to a specialized database, a government data portal, or a niche research tool, you may still need to work with the API directly. Second, understanding what happens at the API level helps you make informed decisions about data security and privacy. When you enable an MCP connection, your data is still traveling to external servers. Knowing that an API request sends your data to a specific endpoint, where it is processed under specific terms of service, is essential for responsible use, particularly when working with sensitive client data, protected health information, or privileged legal communications.

Third, and most practically, understanding these layers helps you evaluate what AI tools can and cannot do. When someone tells you their AI system “integrates with everything,” you now have the vocabulary to ask the right questions. What APIs does it connect to? What data gets sent? Where is it processed? What happens to it afterward?

What this looks like in practice

Here is a concrete example. I frequently use OpenRouter, a service that provides API access to dozens of language models through a single endpoint. Instead of maintaining separate API keys and code for OpenAI, Anthropic, Google, Mistral, and others, I route requests through OpenRouter and switch between models as needed. This is an API-level integration that gives me the flexibility to choose the right model for each task.

At the same time, I use MCP connections in Claude to interact with my institutional tools. If I need to find a document in Google Drive, draft a response based on its contents, and schedule a meeting to discuss it, Claude can handle all of that through MCP connections without me switching between applications or writing any code.

These two patterns, direct API access for specialized research tasks and MCP connections for everyday productivity, coexist in my daily workflow. Neither has eliminated the other.

What you need to know (and what you do not)

If you are a social work or law professional, you do not need to know how to build APIs or develop MCP servers. That is engineering work, and there are people who specialize in it.

What you do need is a working understanding of these concepts. You need to know that when an AI tool connects to an external service, an API is involved, and that connection has implications for data privacy and security. You need to know that MCP is making these connections more accessible, but that the underlying mechanics have not changed. You need to know that the ability to connect AI models to your existing tools and data sources is what transforms a chatbot from a novelty into a functional component of your workflow.

When you are evaluating an AI tool for your organization, these concepts will help you ask better questions. When you are designing a research workflow, they will help you understand what is possible. When you are teaching students, they will help you explain not just what AI does, but how it does it.

The acronyms are accumulating. APIs, MCPs, LLMs, RAG, NLP. The terminology can feel overwhelming. But the core ideas are not complicated. Software needs structured ways to talk to other software. Language models need structured ways to connect to tools and data. Everything else is an implementation detail.

The field is moving fast enough that the specific tools will keep changing. The concepts will not. Understanding what an API is and what an MCP does gives you a stable foundation for evaluating whatever comes next.