The Three Layers of the AI Stack
Last Updated: March 15, 2026
Every technology has layers. Web development has frontend, backend, and infrastructure. Mobile has the app, the OS, and the hardware. Understanding these layers matters because it tells you where to focus your energy and what you can safely treat as a black box.
The AI stack has three layers: Infrastructure, Model, and Application. AI engineers live primarily in the Application layer, but they need enough understanding of the layers below to make good decisions.
The Three AI Stacks
Let's walk through each layer, starting from the bottom.
Layer 1: Infrastructure
The infrastructure layer is where raw compute meets raw data. This is the foundation that makes everything above it possible.
What lives here:
- GPU clusters: Training large language models requires thousands of GPUs running in parallel for weeks or months. NVIDIA's A100 and H100 GPUs dominate this space, with each card costing tens of thousands of dollars. Companies like OpenAI, Google, and Meta operate clusters with tens of thousands of these GPUs.
- Cloud AI platforms: AWS (SageMaker, Bedrock), Google Cloud (Vertex AI), and Azure (Azure AI) provide managed infrastructure for training, fine-tuning, and serving models. These abstract away much of the hardware complexity.
- Data storage and processing: Training data for foundation models is measured in terabytes. Storing, cleaning, and processing this data at scale requires distributed storage systems, data pipelines, and specialized tooling.
- Networking: Distributed training requires moving massive amounts of data between GPUs. High-bandwidth interconnects (NVLink, InfiniBand) are critical for keeping thousands of GPUs working together efficiently.
Who works here:
Infrastructure engineers, platform engineers, hardware engineers. These are the people who keep the lights on for AI.
Why AI engineers should care (but not too much):
You don't need to know how to configure a GPU cluster. But you should understand that training a model like GPT-4 costs over $100 million in compute, which is why you're paying per token through an API instead of training your own. You should know that GPU availability and cost directly affect model pricing, inference latency, and what's economically viable for your application.
Layer 2: Models
The model layer is where intelligence is created. This is where research teams design architectures, curate training data, and run the massive training jobs that produce foundation models.
What lives here:
- Foundation models: These are the large, general-purpose models trained on broad datasets. They're called "foundation" models because they serve as the base for many different applications. Examples:
- GPT-5, GPT-4o (OpenAI)
- Claude 4.6 Opus, Claude 4.6 Sonnet (Anthropic)
- Gemini 3.0, Gemini 2.5 (Google)
- Llama 3, Llama 4 (Meta, open source)
- Mistral Large, Mixtral (Mistral, open source)
- Fine-tuning: Taking a foundation model and training it further on domain-specific data. This produces a specialized model that performs better on specific tasks while retaining general capabilities. Fine-tuning is cheaper than training from scratch but still requires ML expertise and compute resources.
- Open vs closed source. A fundamental divide in the model layer:
- Model serving: Once a model is trained, it needs to be deployed behind an inference server that can handle requests efficiently. This involves optimization techniques like quantization (reducing model precision to save memory), batching (processing multiple requests together), and caching (storing common responses).
Who works here:
ML engineers, ML researchers, model providers. These are the people who create and optimize the models.
Why AI engineers should care:
You need to understand model capabilities and limitations to choose the right model for your use case. You should know what a context window is, how tokenization works, what temperature does, and why some models are better at certain tasks than others. You don't need to understand the math behind transformer attention mechanisms (though we cover this at a practical level), but you need to understand how model behavior affects your application design.
Layer 3: Application
The application layer is where models become useful. This is where AI engineers spend most of their time, building systems that connect models to real-world data and user needs.
What lives here:
- Prompting and prompt management. Designing the instructions that tell models what to do. This includes system prompts, few-shot examples, chain-of-thought patterns, and prompt templates. It also includes versioning and testing prompts systematically.
- Retrieval-Augmented Generation (RAG). Models can't access your private data or information that's newer than their training cutoff. RAG solves this by retrieving relevant documents from your data sources and injecting them into the prompt. This involves embeddings, vector databases, chunking strategies, and ranking.
- Agents and orchestration. Complex tasks require multiple model calls, tool access, and decision-making. Agents are systems where the model decides what actions to take, which tools to call, and how to combine results. Orchestration frameworks help manage these multi-step workflows.
- Evaluation and guardrails. Measuring whether your AI system is working correctly. This includes automated evaluation metrics, human evaluation protocols, safety guardrails (preventing harmful outputs), and regression testing for prompt changes.
- Deployment and operations. Getting your AI application into production with proper monitoring, cost tracking, caching, rate limiting, and failure handling.
Who works here:
AI engineers, software engineers adding AI to their products.
How Data Flows Through the Stack
To make this concrete, let's trace what happens when a user asks a question in an AI-powered customer support chatbot:
- Step 1: User sends a question: "How do I reset my password?" The application receives this as a text string.
- Step 2: Generate embedding: The application sends the question to an embedding model (Model Layer) to convert it into a numerical vector that captures its meaning.
- Step 3: Vector search: The application uses this embedding to search a vector database for similar documentation chunks. It finds three relevant help articles about password reset.
- Step 4: Build prompt: The application constructs a prompt that includes a system message ("You are a helpful support agent"), the retrieved documentation, and the user's question.
- Step 5: Call language model: The application sends this prompt to an LLM (Model Layer). The model processes it and generates a response.
- Step 6: Post-process: The application validates the response (checking for hallucinations, enforcing format requirements), adds source citations, and returns it to the user.
Notice what happened. The application made two calls to the Model Layer (one for embedding, one for generation) but handled everything else: the retrieval logic, prompt construction, validation, and response formatting.
The Model Layer just processed the requests it received. The Infrastructure Layer was invisible, running somewhere behind the model provider's API.
This is the daily reality of AI engineering. You're orchestrating the flow of data through models, not building the models themselves.
Where the Boundaries Blur
These layers aren't always cleanly separated:
Fine-tuning crosses the Model-Application boundary
When you fine-tune a model for your specific use case, you're doing work that sits between the model and application layers. Some providers (OpenAI, Anthropic) offer fine-tuning through their APIs, which keeps AI engineers in the application layer even when customizing models.
Self-hosting crosses the Model-Infrastructure boundary
If you deploy an open-source model like Llama on your own infrastructure, you're taking on responsibilities from both the model and infrastructure layers. This is uncommon for AI engineers at most companies but becomes relevant at scale or when data privacy requirements are strict.
Edge deployment compresses all layers
Running smaller models directly on devices (phones, laptops, embedded systems) collapses the entire stack into a single deployment. This is an emerging area that doesn't yet follow the standard layer separation.
For this course, we'll assume you're working primarily in the Application Layer, using hosted model APIs. This is where the vast majority of AI engineering work happens today.