The Three Layers of the AI Stack
Most software systems are easier to understand when you split them into layers. Web applications have user interfaces, backend services, databases, and infrastructure. Mobile applications have apps, operating systems, runtimes, and hardware. The boundaries are not perfect, but they help teams decide who owns what, where problems can happen, and which trade-offs matter.
AI systems have a similar shape. In this course, we will use three layers:
- Infrastructure: compute, storage, networking, data pipelines, and deployment platforms
- Model: foundation models, embedding models, fine-tuned models, inference runtimes, and model APIs
- Application: product logic, prompts, retrieval, tools, workflows, evaluation, observability, and user experience
AI engineers spend most of their time in the application layer. Still, they need enough understanding of the model and infrastructure layers to make good decisions about latency, cost, privacy, reliability, and system design.
The Three AI Layers
Let's walk through the layers from the bottom up.
Layer 1: Infrastructure
The infrastructure layer is where compute, data, and deployment constraints show up. It affects which models you can run, how fast they respond, how much they cost, where data is stored, and what happens when something fails.
What lives here:
- Accelerators: GPUs, TPUs, NPUs, and other chips used for training and inference. Training the largest models requires large clusters. Serving models in production is a different problem: you care about low latency, high utilization, predictable cost, and enough capacity during traffic spikes.
- Inference runtime: The serving layer that loads model weights, schedules requests, batches work, streams outputs, manages memory, and exposes an API. For self-hosted models, this may involve runtimes such as vLLM, TensorRT-LLM, or a cloud-managed serving system.
- Cloud AI platforms: Services such as AWS Bedrock, Google Vertex AI, Microsoft Foundry, and similar managed platforms bundle model access, deployment, storage, security, evaluation, and observability features. They often blur the line between the infrastructure, model, and application layers.
- Data systems: Object stores, warehouses, databases, vector indexes, document stores, event streams, and labeling pipelines. AI systems depend heavily on data movement and data quality, not just model quality.
- Networking and security: High-bandwidth cluster networking matters for training. For application teams, the more common concerns are private networking, regional availability, tenant isolation, access control, audit logs, and data retention.
Who works here:
Infrastructure engineers, platform engineers, hardware engineers, cloud teams, and security engineers.
Why AI engineers should care:
You do not need to design a training cluster to build useful AI applications. But you should understand the infrastructure constraints that affect your product:
- A larger model may be more accurate but too slow for an interactive workflow.
- A self-hosted model may improve data control but add operational burden.
- A long-context prompt may work in a demo but cost too much at scale.
- A provider outage or regional capacity issue can become a user-facing incident.
- A data residency requirement may eliminate otherwise attractive model options.
Infrastructure is not just the hardware under the system. It sets practical limits around cost, reliability, privacy, and scale.
Layer 2: Models
The model layer contains the learned systems your application depends on. Some models are accessed through provider APIs. Others are open-weight models you can run yourself. Some are general-purpose foundation models. Others are smaller models built for embeddings, reranking, moderation, transcription, classification, or extraction.
What lives here:
- Foundation models: General-purpose models from providers such as OpenAI, Anthropic, Google, Meta, Mistral, DeepSeek, Qwen, and others. They can handle broad tasks such as conversation, code, summarization, reasoning, multimodal understanding, and tool use.
- Specialized models: Embedding models for semantic search, rerankers for retrieval quality, speech models for transcription, vision models for document and image understanding, moderation models, and small classifiers used for routing or guardrails.
- Fine-tuning and adaptation: Further training or adapting a model for a narrower task. This can improve format adherence, domain language, classification behavior, or tool-use patterns. It is not a shortcut around good retrieval, evaluation, or product design.
- Inference APIs and serving: The interface through which applications call models. This includes hosted APIs, managed cloud endpoints, and self-hosted inference servers. Serving choices affect latency, throughput, context length, streaming behavior, batching, and cost.
- Open-weight vs closed models: A practical divide in the model layer:
Who works here:
ML researchers, model providers, ML engineers, applied scientists, and inference engineers.
Why AI engineers should care:
You need to choose models with judgment and build around their limits. That means understanding context windows, tokenization, latency, temperature, structured output modes, tool calling, multimodal input, rate limits, model versioning, privacy terms, and common failure modes.
You do not need to derive transformer attention from first principles to build an AI application. But you do need to know enough about model behavior to avoid weak architecture. For example:
- A model with a large context window can still miss or misread important details if the prompt is poorly structured.
- A better benchmark score does not guarantee better performance on your task.
- A reasoning model may improve difficult tasks but add latency and cost.
- A small model may be the right choice for classification, routing, or extraction.
- A fine-tuned model can still make unsupported claims if the application gives it weak evidence.
Layer 3: Application
The application layer is where model capability turns into product behavior. This is where AI engineers spend most of their time.
The application decides what the model sees, which tools it can use, what output format is acceptable, when to ask a human, how to recover from failure, and how to measure whether the feature is working.
What lives here:
- Prompts and context engineering: System instructions, user input handling, examples, retrieved context, conversation memory, tool results, and output constraints. The prompt is part of the application interface, not just text pasted into an API call.
- Retrieval and grounding: Search, chunking, embeddings, vector databases, keyword search, metadata filters, reranking, citations, freshness, permissions, and evidence handling. RAG is a retrieval system with a generation step at the end, not a pile of documents in a vector database.
- Tools and agents: Tool schemas, API integrations, function calling, MCP (Model Context Protocol) servers, workflow engines, sandboxes, permissions, and human approval steps. Tool use should be scoped and observable, especially when tools can read data, write data, execute code, or spend money.
- Structured outputs and validation: JSON schemas, parsers, retries, business-rule checks, policy checks, and downstream contracts. A fluent answer is not enough if the next system expects valid structured data.
- Evaluation and observability: Test sets, offline evaluations, human review, LLM-as-judge with calibration, trace logging, quality dashboards, cost tracking, and regression testing for prompt, retrieval, and model changes.
- Product experience: Streaming, citations, confidence signals, correction flows, escalation paths, undo, and user controls. A good AI feature makes uncertainty visible and gives users a way to recover.
- Deployment and operations: Rate limiting, caching, fallbacks, rollout controls, incident response, provider abstraction, secrets management, and data retention.
Who works here:
AI engineers, software engineers, product engineers, data engineers, designers, security engineers, and domain experts.
How Data Flows Through the Stack
Consider what happens when a user asks a question in an AI-powered customer support chatbot:
- Step 1: Receive and authorize the request. The user asks, "How do I reset my password?" The application checks identity, tenant, permissions, and any relevant policy constraints.
- Step 2: Prepare retrieval. The application cleans up the query, may classify intent, and sends text to an embedding model to create a search vector.
- Step 3: Retrieve candidate documents. The application searches documentation, policy pages, and possibly support history. It applies metadata filters so the user only receives information they are allowed to see.
- Step 4: Rerank and filter. A reranker or scoring function orders the candidate chunks. The application removes stale, low-confidence, or unauthorized results.
- Step 5: Build the prompt. The application constructs the model input: task instructions, user question, retrieved evidence, citation rules, output format, and limits on what the model may infer.
- Step 6: Call the language model. The model generates an answer using the supplied context. It may return plain text, structured JSON, tool calls, or a mix depending on the application design.
- Step 7: Validate and respond. The application checks the output for format, citations, policy violations, and unsupported claims. It streams or returns the answer, records the trace, and may route uncertain cases to a human.
In this flow, the application makes several calls to the model layer, but it owns the control flow: authorization, retrieval, filtering, prompt construction, validation, logging, and response formatting.
The infrastructure layer may be mostly invisible when you use managed APIs. But it still shapes the system through latency, rate limits, data location, reliability, and cost.
This is the daily work of AI engineering. Building a feature like this means designing the path that data, decisions, and errors take through the system. It is not just calling an LLM.
Where the Boundaries Blur
These layers are useful, but real systems do not follow them perfectly.
Fine-tuning crosses the Model-Application boundary
When you fine-tune a model for a specific use case, application data becomes part of model behavior. The work sits between application design and model development. The hard part is usually not launching the fine-tuning job. It is preparing the dataset, preventing data leakage, evaluating regressions, and deciding whether fine-tuning is actually better than retrieval, prompting, or a smaller task-specific model.
Self-hosting crosses the Model-Infrastructure boundary
If you deploy an open-weight model on your own infrastructure, you take on serving, scaling, monitoring, upgrades, security patches, and capacity planning. This can be the right choice for cost, privacy, latency, or control. It is not automatically simpler than using a provider API.
Managed platforms package multiple layers together
Modern AI platforms increasingly bundle model access, retrieval, agents, evaluation, guardrails, logging, and deployment. This is useful, but it can hide important design decisions. You still need to understand which layer owns data access, tool permissions, quality measurement, and failure handling.
Edge deployment compresses the stack
Running smaller models on phones, laptops, browsers, vehicles, or embedded devices compresses the stack into a local runtime. This can reduce latency and improve privacy, but it introduces constraints around model size, memory, battery, updates, and offline behavior.
For this course, we will focus mainly on the application layer using hosted model APIs. We will still cover enough retrieval, model choice, self-hosting, and operations to help you make informed decisions. That is where most AI engineering work starts, and it is where much of the production complexity lives.
Quiz
The Three Layers of the AI Stack Quiz
10 quizzes