Every AI Product Must Answer These Six Questions
When I started building WebonAI, I thought the prompt would be the hard part.
Find the right words.
Give the model better instructions.
Ask more clearly.
That helped.
But it wasn't the real problem.
The harder work was everything around the prompt.
What should the model know before it answers?
What should it remember from the conversation?
Which tool should it call?
What happens when the output is invalid?
How do I stop an agent from looping?
Which model should handle which task?
The prompt mattered.
The system mattered more.
That changed how I think about AI products.
The model may be the most visible part, but it is only one part. Every production AI system has to answer the same six questions.
The answers determine whether the product feels magical or frustrating.
1. What should the model know?
More context isn't better.
Relevant context is.
Every request starts with the same problem:
What information should the model see right now?
That's context engineering.
Sometimes the answer is the conversation history.
Sometimes it's documentation.
Sometimes it's data from a database.
Sometimes it's nothing at all.
This is also where Retrieval-Augmented Generation (RAG) comes in.
Good RAG isn't just attaching a vector database to an LLM.
You have to decide:
- How documents are chunked
- Which embedding model to use
- Whether to combine vector search with keyword search
- How retrieved documents are reranked
- How fresh the data should be
If retrieval is poor, the smartest model in the world will still produce poor answers.
Garbage in.
Garbage out.
2. What should the model remember?
Memory is expensive.
Which means you have to decide what deserves remembering.
There are several kinds of memory.
Prompt caching remembers identical requests.
Semantic caching remembers similar requests.
Both save money by avoiding repeated work.
Then there's the KV cache.
Large language models don't reread every previous token while generating text.
Instead, they keep an internal memory called the Key-Value cache.
The faster you can reuse that memory—and the smarter you are about evicting old conversations when GPU memory fills up—the more users your infrastructure can support.
Memory isn't just a feature.
It's an infrastructure problem.
3. How do we make it fast?
Users don't measure FLOPS.
They measure waiting.
Making models feel fast turns out to be a collection of different optimizations.
Inference begins with prefill, where the model reads the prompt.
Then comes decoding, where it generates one token at a time.
These two phases behave differently.
Prefill benefits from parallelism.
Decoding doesn't.
Then there are infrastructure optimizations.
Continuous batching keeps GPUs busy by serving multiple users simultaneously.
Paged attention organizes memory more efficiently.
Speculative decoding lets a smaller model predict what a larger model will say next.
Quantization shrinks the numbers used to represent model weights.
Distillation trains smaller models to imitate larger ones.
Even quantization has layers.
INT8.
INT4.
FP8.
AWQ.
GPTQ.
Each makes a different tradeoff between speed, memory, and quality.
Optimization isn't about finding the fastest technique.
It's about finding the cheapest technique your users can't notice.
4. How do we make it reliable?
Language models are probabilistic.
Production software can't be.
That's why reliability matters more than prompting.
Suppose your model returns JSON.
Will it always be valid?
No.
So you validate it.
Repair it.
Retry it.
Fall back to another strategy if needed.
The same idea applies to function calling.
A model may choose the wrong tool.
Pass invalid arguments.
Or call the same API twice.
Good systems define clear tool contracts.
They validate every argument.
They make important operations idempotent so retries don't create duplicate side effects.
Never trust model output.
Verify it.
5. How do we keep it under control?
The dream is an autonomous agent.
The nightmare is an autonomous agent.
Left alone, an agent can loop forever.
Spend money forever.
Call tools forever.
Good engineering sets boundaries.
Maximum reasoning steps.
Tool budgets.
Time budgets.
Termination conditions.
Constraints aren't limitations.
They're part of the design.
The smartest systems know when to stop.
6. Which model should answer?
Not every question deserves your biggest model.
Grammar correction doesn't need frontier reasoning.
Summarization may not either.
Complex planning probably does.
That's why production systems route requests.
Cheap models handle simple work.
Powerful models handle difficult work.
And if one model becomes unavailable?
The system falls back gracefully.
Users don't care which model answered.
They care that it worked.
The Pattern
Building WebonAI taught me something simple.
Prompts matter.
But prompts don't make a product reliable.
Systems do.
None of these six questions are really about finding the perfect words for a model.
They're about designing what happens before the model answers, while it works, and after it fails.
Prompt engineering gets people interested in AI.
System engineering is what turns AI into a product.
As models continue getting cheaper and smarter, the advantage won't belong to the company with the cleverest prompts.
It will belong to the company that best answers these six questions.