Prompt
ConceptThe user’s message. In an enterprise, prompts are rarely typed by hand — they are generated by software, other agents, and workflows.
Think of it as: the work order.
L1 deep dive / Prompt lifecycle
The life of a prompt
What actually happens between pressing “send” in a chat window and watching the answer type itself out — told through the handful of components that every AI product, from a demo to an enterprise platform, is built from.
01 / The cast
Nine actors, plain language
Every prompt meets the same characters on its way through the system. Learn these nine and you can read any AI architecture diagram, vendor pitch, or technical diligence memo.
System prompt
ConceptStanding instructions written by the product team: who the assistant is, what it may and may not do, tone, rules. It is silently prepended to every single request.
Think of it as: the employee handbook the model re-reads
before answering anything.
Skills & tools
ConceptPackaged capabilities the model can ask for: search the web, query a database, run code, open a ticket. A skill ships the instructions; a tool is the actual connection that executes.
Think of it as: the apps on the agent’s desk, with
instructions on when to use each.
The harness
Component / containerThe software container around the model. It assembles the full request (system prompt + tools + memory + history + your message), sends it, and executes whatever the model asks for in reply. Claude Code, coding copilots, and most “agents” are harnesses.
Think of it as: the chief of staff — prepares the
brief, carries the messages, executes the decisions.
The LLM
Component / modelThe reasoning engine itself. Two ways to source it: rent a frontier model by API (OpenAI, Anthropic, Google) or run open weights (Llama, Qwen, Mistral) on your own hardware.
Think of it as: the brain — either rented by the
hour or hired in-house.
API gateway
Component / control pointThe front desk every request passes through. Checks identity and permissions, enforces budgets and rate limits, chooses which model or provider handles the request, and logs everything.
Think of it as: airport security plus air-traffic
control, in one building.
Inference server
Component / engine roomThe software that actually runs the model on GPUs (vLLM, SGLang, TensorRT-LLM). It batches hundreds of requests together and manages the scarce GPU memory that determines speed and cost.
Think of it as: the factory floor manager — keeps
every machine busy, nothing idle.
KV-cache
Component / memory trickThe model’s working memory of text it has already read. Without it, the model re-reads the entire conversation for every new word it writes. With it, only the new part is read.
Think of it as: the bookmark that saves re-reading the
whole file. Section 04 explains why it matters.
GPU cluster
Infrastructure / physical layerThe hardware: racks of GPUs, the networks between them, and the power they consume. Whether owned by a cloud, a model provider, or the enterprise itself, everything above is a tenant here.
Think of it as: the power plant — everyone else is
just a customer.
02 / The canonical stack
The industry’s standard layer list, translated
AI architecture diagrams almost always show some version of these fourteen layers. Here is the canonical list, mapped to the zones and named products of the L2 machinery page.
Layer 02
API Gateway
Single entry point for requests: admission, throttling, hand-off to identity checks.
Control room
Layer 04
Prompt Management
Versioned home of the system prompt and templates — the handbook, under change control.
Workbench
Layer 05
Model Gateway
One interface over many models: routing, fallbacks, budgets per team or product.
Control room
Layer 07
RAG
Retrieval-augmented generation: fetching the right documents into the dossier before the model reads it.
Workbench
Layer 08
Vector Database
Where documents live as searchable embeddings, so retrieval is fast.
Workbench
Layer 09
Tools & APIs
The actions the agent can take in other systems — search, query, ticket, execute.
Workbench
Layer 10
Memory
User and conversation state carried across sessions, so the dossier starts pre-filled.
Workbench
Layer 11
Guardrails
What the model may not say or do — enforced in-line, not just promised in a policy.
Control room
Layer 13
Evaluation
Quality scoring and regression tests, before release and continuously after.
Control room
Layer 14
Deployment & Scaling
GPUs, clusters, and schedulers that keep the whole machine alive under load.
Power plantWhat the classic list misses: the parts that actually decide cost and speed — the inference server, the KV-cache, and the GPU substrate beneath them. Vendors rarely diagram what they rent rather than sell. Sections 04 and 05 go deeper, and the L2 page names who builds each piece.
Layer names adapted from a widely shared community diagram by Greg Coquillo (Product Leader), re-drawn in this report’s visual language and mapped to its zones.
03 / The journey
Seven steps from keystroke to answer
One message, end to end. The same seven steps run inside a five-line demo script and inside a billion-dollar enterprise platform — only the sophistication of each step changes.
Follow the flow — scroll right →
1
Originate
Type and send
A user writes a message in the chat window — or, more often in enterprise settings, an agent or workflow generates it automatically. Either way, a prompt is born.
On stage
Prompt
Chat UI
2
Assemble
The harness builds the brief
The harness never sends your sentence alone. It assembles the full dossier: the system prompt, the available tools and skills, relevant memory and documents, the conversation so far, and finally your message. All of it is converted to tokens — the billing unit of AI.
This is why “one question” can secretly cost 50,000 tokens: the model receives the whole file, not the one line you typed.
On stage
Harness
System prompt
Skills & tools
Memory
3
Admit
The gateway checks and routes
The request reaches the API gateway. It authenticates who is asking, checks permissions and remaining budget, applies policy (some data may never leave the building), and picks a route: which provider, which model, which region. Cheap model for easy questions, flagship model for hard ones.
On stage
API gateway
Router
Identity & policy
4
Read
Prefill: the model reads the dossier
The inference server hands the request to the model. First the model reads the entire assembled context — the “prefill” phase. This is where the KV-cache does its magic: whatever the model has already read earlier in the conversation is remembered, not re-read. Details in section 04.
On stage
Inference server
LLM
KV-cache
GPU
5
Write
Decode: one token at a time
Now the model generates the reply — literally one token (a fraction of a word) per step, each one informed by everything in the cache. This sequential loop is why answers “type out” and why generation speed, not reading speed, defines how fast AI feels.
On stage
LLM
KV-cache
GPU
6
Deliver
Streaming back to the window
Tokens are not batched and returned at the end — they are streamed back through the gateway the moment they exist, and the chat window renders them live. Streaming is a product feature disguised as a transport detail: it makes a 20-second answer feel instant.
On stage
API gateway
Streaming connection
Chat UI
7
Act & record
The loop closes — and all is logged
If the model asked to use a tool (search, query, run code), the harness executes it and sends the result back to the model. An agentic task may loop dozens of times before the user sees a word.
Meanwhile, every step is recorded: who asked, what was sent,
what it cost, what came back. These traces are the raw material
for auditing and improving the system.
On stage
Harness
Skills & tools
Observability
04 / The magic trick
KV-cache: the bookmark that runs the economics
The single most consequential technical detail for understanding AI cost and speed — and the reason “long conversations” and “long documents” are expensive.
The model has no memory. The cache is the workaround.
A model does not remember your conversation. Every new word it writes is, in principle, computed by re-reading the entire dossier: system prompt, documents, history, everything. Reading is compute; compute is money and time.
The KV-cache stores the model’s already-computed reading of that dossier in GPU memory. When the conversation continues, only the genuinely new part is read; the rest is recalled. Same answer, a fraction of the work.
The catch: the cache lives in the scarcest resource in the building — GPU memory. Managing it well is why inference servers exist as a category, and why API providers now charge less for repeated context.
Follow the cache and you follow the money: latency, unit cost,
and half the inference-software market are all arguments about
this one data structure.
Without cache — follow-up question
System prompt
2,000 tokens + Documents
40,000 tokens + History
8,000 tokens + New question
50 tokens = Read 50,050
2,000 tokens + Documents
40,000 tokens + History
8,000 tokens + New question
50 tokens = Read 50,050
Every follow-up re-reads the whole file. Slow, and billed at full price each time.
With KV-cache — same follow-up
Recalled from cache
50,000 tokens + New question
50 tokens = Read 50
50,000 tokens + New question
50 tokens = Read 50
Only the new words are read. Faster first token, and providers increasingly price cached tokens at a steep discount.
Prefill (“reading”)
Ingesting the context. Parallel, fast per token — but
proportional to dossier size unless cached.
Decode (“writing”)
Generating the answer, one token at a time. Sequential
— this is the speed the user feels.
Time to first token
The pause before typing starts. Mostly prefill; the
metric product teams watch.
Prefix caching
Reusing cache across different requests that share
the same beginning — e.g. one system prompt serving a
whole company.
05 / Two engines
Frontier API or self-hosted: sourcing the brain
Step 4–5 of the journey can run in two very different places. This is a sourcing decision — rent versus own — and most enterprises end up with both.
Frontier model, rented
A- What it is. The strongest models, consumed by API: OpenAI, Anthropic, Google. Their GPUs, their inference stack, their problem.
- Why choose it. Best quality per effort; zero infrastructure; new capabilities arrive automatically.
- The trade-offs. Data leaves your walls (mitigated by enterprise agreements), unit cost scales with success, and you build on someone else’s roadmap.
Typical buyer posture: start here for everything, stay here for the hardest reasoning tasks.
Open weights, self-hosted
B- What it is. Published model weights (Llama, Qwen, Mistral, DeepSeek) run on your own GPUs with an inference server like vLLM — in your cloud or your data center.
- Why choose it. Data never leaves; cost is predictable at high volume; no vendor can deprecate your model; it can be fine-tuned on private data.
- The trade-offs. You now own a GPU fleet, an inference stack, and an operations team — and you trail the frontier on the hardest tasks.
Typical buyer posture: high-volume, well-defined, or regulated workloads — the “factory work,” not the “strategy work.”
06 / The mapping
Concept → component → infrastructure → why it matters
The whole page on one grid: each plain-language idea, the technical component that implements it, the infrastructure it runs on, and the investment lens on each row.
Instructions & behavior
System prompt, skills, tool definitions
Agent harness
Orchestration frameworks, agent runtimes
Containers
Kubernetes, sandboxed execution environments
Where product differentiation lives today; historically hard to
monetize as infrastructure, but the control loop is sticky once
adopted.
Front desk & traffic control
Who is asking, what is allowed, where it goes
AI gateway / router
LiteLLM, Kong, Portkey, cloud-native gateways
API infrastructure
Load balancers, identity providers, edge networks
A governance control point every request crosses — high
conviction in the L0 landscape. Durable, model-agnostic.
The brain
Reasoning over the assembled dossier
LLM, frontier or open weights
Model APIs or self-hosted weights
GPU clusters
Cloud regions, neoclouds, on-prem accelerators
Enormous value, brutal competition, capex-heavy. The landscape
treats models as tenants, not as the investment.
The bookmark
Not re-reading what was already read
Inference server + KV-cache
vLLM, SGLang, TensorRT-LLM; prefix caching
GPU memory & fast interconnect
HBM, high-speed networking, emerging cache tiers
The cost-and-latency lever of the whole stack. Efficiency gains
here directly reset the unit economics of every AI product.
The typing effect
Answer appears as it is generated
Streaming transport
SSE / WebSockets, token-by-token delivery
Edge & network
CDNs, regional endpoints, last-mile latency
Mostly solved plumbing — but it sets user expectations that
force spending everywhere else in the chain.
The paper trail
Every request logged, metered, auditable
Observability & evaluation
Tracing, cost metering, quality scoring
Data pipelines
Event streams, log storage, analytics
The evidence layer of the L0 landscape: accumulates operating
history that compounds into a moat.
07 / Pocket glossary
Ten terms for the next board meeting
The vocabulary that recurs across every vendor pitch and technical diligence session, in one breath each.
Token. The unit of text AI reads, writes, and bills —
roughly three-quarters of a word. Prices are quoted per million.
Context window. The maximum size of the dossier the model can
consider at once. Bigger windows raise both capability and cost.
Inference. Using a trained model to produce answers —
as opposed to training it. Where the ongoing spend is.
Prefill / decode. The reading and writing halves of
inference. Reading is parallel and cheap-ish; writing is sequential
and sets perceived speed.
KV-cache. GPU-resident memory of text already read. The
reason conversations get faster after the first exchange.
Latency / TTFT. Time to first token: the pause before the
answer starts appearing. The headline responsiveness metric.
Throughput. Tokens produced per second across all users.
Determines how many customers one GPU can serve.
Batching. Serving many requests through the model at once.
The main trick inference servers use to lower unit cost.
Harness / scaffolding. The software around the model that
turns a chat model into an agent that can act.
Tool call. The model’s formal request for the harness
to do something — search, query, execute — mid-answer.
PTV Fund II / L1 takeaway
The prompt’s journey is the industry’s map.
Assemble, admit, read, write, stream, record — every company in the AI stack monetizes one of these verbs. When a pitch gets technical, ask which step of the journey it makes cheaper, faster, or safer. That question is the whole diligence.