Open the Hood: How AI Models Actually Work, From Parameters to RAG

Weights, tokens, thinking budgets, tool calls, and the vector databases behind RAG — the five ideas underneath almost everything you'll build as an AI agent developer, explained with real file formats and real numbers instead of hand-waving.

Ask someone building with AI to explain what's actually happening between typing a prompt and getting an answer back, and most people wave their hands at "the AI" like it's one mysterious thing. It isn't. It's five specific, documented mechanisms — a file full of numbers, a way of chopping text into pieces, an option to think longer before answering, a protocol for calling outside functions, and a system for looking facts up before answering. Learn to name each one and "AI" stops being magic — it starts being infrastructure you can actually debug.

What's actually in the file

Parameters and the run file

When people say a model has "8 billion parameters" or "405 billion parameters," they mean exactly that: a count of the numbers — weights — the model learned during training, each one a small float that gets multiplied and added as your input flows through the network. Meta's Llama 3.1 ships in 8B, 70B, and 405B variants, and because it's an open-weights model, Meta actually publishes the internals: the 405B version has 126 layers and a 16,384-dimension hidden size. That's real, checkable architecture, not a marketing number.

Nobody discloses the big proprietary numbers. OpenAI has never confirmed GPT-4's parameter count, and Anthropic has never confirmed Claude's — the "1.8 trillion" and "5 trillion" figures you'll see quoted around the web are community estimates and unverified claims, not official numbers. Treat them as rumor, not spec.

Two ways those weights get packaged

Weights on their own are a wall of bytes — nothing in a raw file of floats tells you the architecture, the vocabulary, or how to prompt the model. So they ship in one of two common formats.

• SafeTensors, from Hugging Face, stores raw tensors in a memory-mapped layout with a plain JSON header — no executable code, so it loads faster than the older pickle format and can't run arbitrary code when you open it. It's weights only, so a SafeTensors model repo also ships a separate `config.json` (architecture) and `tokenizer.json` (vocabulary and merge rules) alongside it.
• GGUF, built by the llama.cpp project, bundles the weights, the tokenizer, and all the architecture and quantization metadata into one portable file — nothing else to download or keep in sync, which is why most local-inference tools (Ollama, LM Studio) standardize on it.

The run file — the part that isn't the weights at all

This is where "run file" comes in, and it's easy to conflate with the weights themselves — it isn't the same thing. Take Ollama: to actually run a GGUF file, you register it with a Modelfile, a small text file whose only required line is `FROM ./model.gguf`. From there you can add `PARAMETER` lines (temperature, context window size, stop sequences), a `TEMPLATE` for how the chat prompt gets assembled, a `SYSTEM` prompt, and `MESSAGE` lines for few-shot examples. None of that touches a single weight — it's entirely about how the runtime loads and talks to a model that's already been trained, which is exactly the separation you want: the same GGUF file can be run under a dozen different Modelfiles, each tuning it for a different job, without ever re-training anything.

The unit everything else is measured in

Tokens, and how models actually read text

A model doesn't read letters or words. It reads tokens — integer IDs from a fixed vocabulary, each one standing in for a chunk of text. Most modern tokenizers build that vocabulary with byte-pair encoding (BPE): start from the 256 possible byte values, count which adjacent pairs show up most often across a huge training corpus, merge the single most frequent pair into a new symbol, and repeat until the vocabulary hits a target size. Because it starts from raw bytes, byte-level BPE can represent any Unicode text — emoji and non-Latin scripts included — without ever falling back to an "unknown token."

Why one word can become two tokens

A common, frequently-seen word usually earns its own single token. A rarer or more complex one doesn't — it gets split into smaller pieces the tokenizer already knows. "Tokenization" itself is the textbook example: it's uncommon enough on its own that a subword tokenizer typically splits it into "Token" + "ization" rather than storing the whole word as one entry — the same mechanism that lets "unbelievably" split into pieces like "un" + "believ" + "ably." It's the trick that lets a vocabulary of a few hundred thousand entries cover a language with millions of possible words.

As a working rule of thumb for English text, OpenAI documents roughly 4 characters per token, or about three tokens for every four words — good for a back-of-envelope estimate, never exact. And the ratio isn't fixed across models: Anthropic's own documentation notes that Claude Opus 4.7 and later models use a newer tokenizer that produces roughly 30% more tokens for identical input text than earlier Claude models did. Tokens aren't a universal unit — they belong to whichever tokenizer shipped with the specific model you're calling, which is exactly why every provider ships a token-counting endpoint instead of asking you to estimate.

Spending compute at the moment you ask, not just at training time

Test-time compute and extended thinking

Every model above has a fixed set of weights, decided once training finishes. Test-time compute is a separate lever entirely: instead of changing the model, you let it spend more computation per answer, at the moment you ask the question. OpenAI's o1, released in September 2024, was the first widely-used "reasoning model" built around this idea — it generates an extended chain of reasoning before committing to a final answer, and its accuracy on hard problems in math, coding, and science scales with how much of that reasoning it's allowed to do.

Anthropic's version is extended thinking. You set a `thinking.budget_tokens` value on the request — a minimum of 1,024 tokens, and it has to leave room under `max_tokens` for the actual reply — and Claude spends up to that budget reasoning before it writes the final answer. It's a target, not a hard floor: Claude can stop reasoning early if it doesn't need the full budget. Anthropic's own testing found accuracy tends to improve logarithmically as the thinking budget grows — real gains, but with diminishing returns — and in one reported benchmark, Claude 3.7 Sonnet using a 64k-token thinking budget across 256 parallel samples reached 84.8% on the GPQA science benchmark.

It's a dial, not a switch. More thinking tokens means more latency and more cost — thinking tokens are billed as output tokens. Start near the 1,024-token floor for simple lookups, and reach for 16,000 or more on something you'd actually want to double-check by hand.

Letting the model act, not just answer

Function calling (tool use)

On its own, a model can only produce text. Function calling — Anthropic calls it tool use — is the mechanism that lets it trigger real code: a weather lookup, a database query, a calendar write. You describe each function as a tool: a `name`, a plain-language `description`, and an `input_schema` written in JSON Schema that spells out the arguments it takes. Claude reads that schema the same way it reads your prompt — as documentation — and decides for itself, per turn, whether calling a tool is the right move.

The full loop is two round trips, not one. First, Claude answers with `stop_reason: "tool_use"` and a `tool_use` content block naming the tool and its arguments — it never executes anything itself, it just names the call. Your application runs the real function, then sends the result back as a `tool_result` block tied to that same call's ID. Only then does Claude see the outcome and write the actual reply. That split matters: Claude decides when to call a tool; your code always decides what the tool does — exactly the boundary that keeps a model from, say, actually deleting a file just because it decided to.

Two flavors worth telling apart

• Client tools — anything you define yourself, plus a few Anthropic-defined schemas like the bash and text-editor tools — run inside your application. You get the `tool_use` block and you write the code that executes it.
• Server tools — web search, web fetch, code execution — run on Anthropic's own infrastructure instead. You get the finished result directly, with no execution code of your own to write.

Giving a model facts it wasn't trained on

Embeddings, vector databases, and RAG

A model's knowledge is frozen at its training cutoff, and it has no idea what's sitting in your company's internal docs. Retrieval-augmented generation (RAG) fixes that without retraining anything — it fetches the relevant text and hands it to the model as context right before it answers. That takes two pieces working together: embeddings, to turn text into something searchable by meaning, and a vector database, to search it fast.

What an embedding actually is

An embedding model turns a chunk of text into a vector — a list of numbers, typically somewhere between 256 and 3,072 numbers long depending on the model — positioned so that texts with similar meaning land close together in that space, regardless of whether they share any of the same words. "The cat sat on the mat" and "a feline rested on the rug" end up near each other; "the stock market fell" lands far away. Distance between two vectors is usually measured with cosine similarity — the angle between them, ignoring length — and a training technique called Matryoshka representation learning lets one embedding model produce a vector you can truncate to a smaller size (say, 1024 down to 256 dimensions) and still keep most of its accuracy, trading precision for speed and storage.

Why you need a vector database at all

Comparing a query's vector against every stored vector one at a time works fine at a few thousand documents and falls over at millions. A vector database solves that with an index built for approximate nearest-neighbor search — HNSW (Hierarchical Navigable Small World) is the most common structure, a multi-layer graph that lets a search hop toward the right neighborhood in a few hundred comparisons instead of scanning everything. Postgres users get this without standing up a separate system through pgvector, an extension that adds a vector column type plus cosine-distance, Euclidean-distance, and inner-product operators, with both HNSW and IVF indexing, right inside the database you're probably already running.

The pipeline end to end: documents get split into chunks small enough to stay specific but large enough to keep their surrounding context — too small and you lose the meaning around a sentence, too large and the one relevant line gets diluted by everything next to it. Each chunk gets embedded and stored ahead of time. At query time, the question gets embedded with that same model, matched against the store, and the top few chunks come back to be stitched into the prompt alongside the original question. The model never sees your whole knowledge base — only whatever the search step decided was relevant, which is exactly why retrieval quality, not model quality, is usually the real bottleneck in a RAG system.

Naive chunking breaks more RAG systems than bad models do. A chunk stripped of its context — "revenue grew by 3% over the previous quarter" with no company or date attached — is nearly useless to a retriever, even though it reads fine to a human. Anthropic's own tests on exactly this failure mode found that prepending a short, generated context note to each chunk before embedding it cut retrieval failures by 35% on its own, by 49% combined with keyword search, and by 67% once a reranking step was added on top.

Where this leaves you

Five mechanisms, one mental model

None of this requires believing in magic. A model is a file of numbers loaded by a runtime you configure separately. Text in and out is tokens from a fixed vocabulary. More thinking time is a budget you set explicitly. Acting on the world is a schema the model fills in and your code executes. And remembering things it wasn't trained on is a search problem, solved with embeddings and an index. Once you can point at the actual mechanism behind each one, "prompt engineering" starts looking a lot more like the ordinary systems engineering it already is.

Build the web research agent — Course 13 puts retrieval and tool-calling concepts to work for real: live search, page scraping, and a citation guard that checks every source the AI cites instead of trusting it blindly.

Frequently asked questions

What's the difference between a model's parameters and its context window?

Parameters are fixed at training time — the weights baked into the file, unchanged no matter what you send the model. The context window is per-request: how many tokens of conversation, system prompt, and tool definitions the model can hold in view for a single reply. A bigger context window doesn't mean more parameters, and more parameters don't automatically buy a bigger context window — they're independent numbers.

Is a GGUF file the same thing as a Modelfile?

No. GGUF is the weights format — the trained numbers, tokenizer, and architecture metadata bundled into one file. A Modelfile is a separate, small text file that points at a GGUF file with a FROM line and layers runtime settings like temperature and the system prompt on top of it. You can point several different Modelfiles at the exact same GGUF file.

Does extended thinking make a model "smarter"?

Not in the sense of changing its weights — it's the same model either way. It gives the model more room to reason step by step before committing to an answer, which measurably improves accuracy on tasks that benefit from working through steps, like math, multi-step logic, or debugging, with real but diminishing returns as the budget grows, and no free lunch on cost or latency.

Can a model call a function without me writing any code?

For your own tools, no — you always write the code that actually runs when the model requests it; the model only ever names the call and its arguments. Anthropic's server tools, like web search and code execution, are the exception, since those run on Anthropic's own infrastructure and hand you the finished result directly.

Do I need a dedicated vector database, or can I just use Postgres?

For most projects, Postgres with the pgvector extension is enough — vector search alongside your regular relational data, with both HNSW and IVF indexing, and one fewer system to run. Dedicated vector databases earn their keep at very large scale or when you need features they specialize in, but they're not a requirement to build RAG.

Why does the same sentence sometimes count as a different number of tokens on different models?

Because tokens aren't universal — each model ships with its own tokenizer and vocabulary, trained on its own merge rules. Anthropic's own documentation flags exactly this: Claude's newer tokenizer, on Opus 4.7 and later models, produces roughly 30% more tokens for identical text than earlier Claude models did. Always count tokens against the specific model you're planning to call, never reuse a count measured on a different one.