From the outside, ChatGPT looks like a single thing: a text box, a response, a magic happening somewhere in between. From the inside, it is at least a dozen pieces of software talking to each other — and once you can see those pieces, you can reason about every AI product you'll encounter. This lesson hands you the X-ray glasses.
What "system design" means
Architects don't design buildings by placing every brick. They sketch how rooms relate, where load-bearing walls go, how plumbing runs through the floors, and what happens when the elevator breaks. The bricks are details. The shape — the relationship between the parts — is the design.
Software system design works the same way. It's the set of high-level decisions about how the major components of a product connect, communicate, and divide work. What runs in the user's browser? What runs on your servers? What runs on someone else's servers? Where does data live? How do the pieces talk? When something breaks, which piece notices?
For most of software's history, system design was an engineer's concern and a manager's afterthought. AI products have changed that. The pieces of a modern AI feature have such different cost, latency, and failure characteristics that a designer or PM who doesn't understand the system will routinely propose features that are impossible, ruinously expensive, or unsafe. The good news: you don't need to know how each piece is built, just how they fit together. That's what we'll teach you.
The four parts every AI product has
Almost every AI product, no matter how fancy, is made of four kinds of pieces. Different traditions call them different things; we'll use plain words.
Interface is what the user touches. A chat window, a button labeled "Summarize", a voice assistant, a slash command inside an existing app. The interface's only job is to take the user's intent and hand it off, then take the answer and present it. It is rarely where the magic happens, but it is overwhelmingly where users decide whether a product feels good.
Brain is the AI model itself — the large language model, the image generator, the recommendation engine. The brain is almost never code you wrote; it's a service you call. For most AI products in 2026, the brain belongs to OpenAI, Anthropic, Google, or one of the open-source families like Llama or Mistral. You rent it by the token.
Memory is everything the product remembers. A database storing your past conversations, a vector store of company documents, a cache of yesterday's answers, the user's profile and preferences. Without memory, an AI product is a stranger every time you open it.
Plumbing is the unglamorous middle: the server that connects interface to brain to memory, handles authentication, enforces rate limits, logs everything, processes payments, calls external APIs, retries on failure, and prevents one user from seeing another user's data. It is almost always the largest piece of code in the product and the one most likely to cause your incident.
Memorize these four: interface, brain, memory, plumbing. When something goes wrong in an AI product, the cause almost always lives in one of them, and being able to name them is half the diagnosis.
Input → Processing → Output
Zoom out one more level and every interaction with an AI product follows the same three-act structure: input, processing, output. The user types or clicks or speaks. The system thinks. The system responds.
This sounds obvious, but it's a useful frame because the bottlenecks live in different acts. If a feature feels slow, ask: is the input slow to collect, the processing slow to run, or the output slow to render? If a feature feels dumb, ask: is the input incomplete, the processing wrong, or the output poorly presented? The same three questions cover almost every product problem you'll see.
In AI products specifically, "processing" tends to dominate. The user's input is small (a sentence or two), the output is moderately sized (a paragraph or two), but in between sits a call to an AI model that may take several seconds and cost real money. Most of the engineering you'll see — caching, streaming, batching, picking smaller models — is about making that middle act faster and cheaper.
How a single user message becomes a response
Let's trace a real journey. You type "What's the weather in Paris?" into ChatGPT and hit enter. Here's roughly what happens behind the glass.
Your text leaves your browser and travels across the internet to OpenAI's servers. The plumbing receives it, checks that you're logged in, checks that you haven't exceeded your rate limit, looks up your conversation history, and bundles all of that into a single block of text formatted the way the model expects. That block — your message, your history, and a hidden "system prompt" telling the model how to behave — is sent to the brain.
The brain starts predicting a response, one word fragment (a token) at a time. As each token comes out, ChatGPT streams it back across the internet to your browser, which paints it on the screen. That's why you see ChatGPT "type" — it isn't pretending; it's literally producing the answer as you watch. Meanwhile, the system stores the new turn in your conversation history so it can be sent back in the next message's context.
For the Paris question, the brain might also decide to call a tool — a separate weather API — to look up real-time data. If so, it pauses generation, the plumbing makes the API call, the result is handed back to the brain, and generation resumes. None of this is visible to you. From your seat, you typed a question and got an answer.
Once you understand this loop, you can read about almost any AI feature and place each part in it. The chat surface is the interface. The model call is the brain. The conversation history and any retrieved documents are memory. Everything else is plumbing.
Latency and cost are design decisions, not afterthoughts
Traditional software has a comfortable set of assumptions: most user actions complete in milliseconds, and the marginal cost of one extra user action is effectively zero. AI breaks both.
An LLM response takes anywhere from half a second to thirty seconds depending on the model, the length of the answer, and how busy the provider is. That's two to four orders of magnitude slower than a database query. If you design an AI feature assuming it'll feel like a database query, your users will think the product is broken.
The cost story is even more uncomfortable. Calling a frontier model on a long conversation can cost several cents per turn. Cents are nothing once — until you multiply by a million users sending fifty messages a day. Suddenly your AI feature has a six-figure monthly bill that didn't exist for the non-AI version. Companies have shipped beautiful AI features and then quietly pulled them after the invoice arrived.
The healthy mindset is to treat latency and cost as first-class design constraints, the same way you'd treat accessibility or security. Every architectural choice — which model to use, how long to make prompts, whether to stream, whether to cache, whether to summarize old conversations — moves the dial on one or both of them. We'll come back to this in every chapter.
ChatGPT's architecture from your message to its reply. Each turn travels through at least six pieces of infrastructure: the web client, a load balancer, an authentication service, a rate-limit checker, the model-serving fleet (running on specialized GPU clusters), and a logging pipeline that records the exchange for safety review and future training. Behind those sit conversation storage, billing systems, abuse detection, and content moderation — all invisible to you, all essential.
The lesson for product people: when you read "ChatGPT launched X feature this week," you're seeing the tip of a system that took thousands of engineers years to build. The interface change is the part you notice; the plumbing changes that made it safe and affordable are the work.
- Thinking the interface is the product. The chat box is the thinnest layer; the plumbing and memory around it are where most of the engineering — and most of the failures — actually live.
- Assuming the "brain" is code your team wrote. In nearly every AI product the model is a rented service you call by the token, which means its speed, cost, and outages are largely outside your control.
- Designing an AI feature as if a model call behaves like a database query. Model calls are seconds, not milliseconds, and cost real money per turn, so latency and cost have to shape the design from day one.
- Forgetting that conversation history is sent back to the model every turn. Skipping the memory piece leads to features that "forget" context or to prompts that balloon in size and cost as a chat grows.
- Blaming "the AI" when something breaks. Most incidents trace to one of the four parts — usually the plumbing — so naming the part is the first step to diagnosing it.
Draw the journey of "what's the weather in Paris?" through ChatGPT on paper (about 10–20 minutes, no code required). Label every step you can guess. Don't worry about being correct — when you compare your sketch to the one in this lesson, you'll discover which parts of the system you intuited and which you missed. The gap is your learning.
Key Takeaways
- System design is the shape of a product — how the pieces connect — not the bricks themselves.
- Every AI product has four pieces: interface, brain, memory, plumbing. Name them in any product you encounter.
- Every interaction is input → processing → output. The slow, expensive part of an AI product is almost always the processing.
- A single chat message touches authentication, rate limits, history, the model, tool calls, streaming, and logging — most of which the user never sees.
- Latency and cost are not afterthoughts. They are baked in at design time, and they constrain what features are even possible.
Every AI product, however polished, is four parts — interface, brain, memory, and plumbing — wired together so that each user interaction flows from input through processing to output. Tracing a single message through that loop reveals how much invisible work sits behind a simple reply, and why the processing step dominates. Carry the four-part vocabulary and the input → processing → output frame into every product you study, and treat latency and cost as design constraints from the very first sketch.