§1 — How an LLM actually works

One sentence: an LLM is a next-token prediction engine — it doesn't "know" things, it completes patterns based on everything it saw in training plus everything you put in front of it right now.

Under the hood, modern LLMs are built on the transformer architecture, which uses a mechanism called self-attention to weigh how much each token in the input matters to every other token. This is what lets a model "understand" that "it" in "The bank raised its rates because it was losing money" refers to the bank, not some other entity — the attention mechanism learned those statistical relationships from trillions of tokens during training.

1.1 How models are trained — the two phases

Every LLM goes through two distinct training phases, and understanding them explains why models behave the way they do:

1.2 Tokens — the unit of everything

Models don't see words or characters; they see tokens (~¾ of an English word, less for code). Tokenization is done by a separate model (usually BPE — Byte Pair Encoding) that splits text into subword pieces. Common words like "the" are one token; rare words like "hallucination" might be split into hall + uc + ination — three tokens.

Input:   "def calculate_total(items):"
Tokens:  ["def", " calculate", "_total", "(items", "):"]     → 5 tokens

Input:   "hallucination"
Tokens:  ["hall", "uc", "ination"]                             → 3 tokens

Input:   "the"
Tokens:  ["the"]                                               → 1 token

Rule of thumb:
  1 token   ≈ 4 characters ≈ 0.75 words
  1K tokens ≈ 750 words    ≈ 1.5 pages
  Code is token-heavier than prose (symbols, indentation)
  Non-English text is often token-heavier (more tokens per word)

Why you care: cost, latency, and context space are all billed per token. A 3,000-line file you pasted "just in case" costs money, slows the response, and crowds out the content that mattered. Different providers tokenize differently — the same text might be 100 tokens with OpenAI's tokenizer but 120 with Anthropic's.

1.3 Prediction is probabilistic — and tunable

The model outputs a probability distribution over the next token, then samples from it. Two key parameters control this sampling:

Prompt: "The database connection failed because the"

Next-token probabilities:
  "connection"  31%   ██████████
  "server"      24%   ████████
  "credentials" 18%   ██████
  "pool"        11%   ████
  "network"      9%   ███
  ...

Temperature controls how adventurous the sampling is. Top-p (nucleus sampling) limits the token pool to the smallest set of tokens whose cumulative probability exceeds p (e.g., top-p=0.9 means only the top 90% probability mass is considered). Top-k simply limits sampling to the k most likely tokens. In practice, you usually adjust temperature alone — top-p and top-k are fine-tuning knobs:

1.4 Model sizes and the capability-cost trade-off

Models come in different sizes, measured in parameters (the learned weights). Bigger isn't always better — the right model depends on your use case:

Rule of thumb: start with the smallest model that works. If it handles 80% of your cases well, route the remaining 20% to a larger model. This "model routing" pattern saves significant cost at scale.

1.5 System prompts — the hidden lever

Most APIs let you set a system prompt (also called a system message) that sits before the conversation. The model treats it as high-priority instructions from the "system" — not something the user said. This is where you define:

• Role & persona: "You are a senior Python code reviewer"
• Behavioral rules: "Never edit production database migrations directly"
• Output format: "Always respond in JSON with keys: severity, file, line, fix"
• Guardrails: "If you are unsure, say 'I don't know' rather than guessing"

System prompts are not magic — the model can still "forget" them in very long conversations (they sit at the start of context, so they benefit from the "primacy effect" but can be diluted by 100K tokens of conversation history). Keep them concise and prioritized.

1.6 Streaming — token by token, not all at once

LLMs generate tokens one at a time, left to right. You don't wait for the full response — you receive tokens as they're produced. This is streaming, and it's why ChatGPT "types" its answers. For engineers, streaming matters because:

• Perceived latency drops — the user sees the first token in 200–500ms even if the full response takes 10 seconds
• You can cancel early — if the model starts going off track, stop generation before it wastes tokens
• Each token costs the same — streaming doesn't change the bill, it changes the UX

1.7 Cost breakdown — input vs. output tokens

API pricing is split into input tokens (your prompt + context) and output tokens (the model's response). Output tokens are typically 3–5x more expensive than input tokens because they require a full forward pass through the model for each token generated:

Example pricing (Claude Sonnet 4, as of mid-2025):
  Input:  $3.00  per 1M tokens
  Output: $15.00 per 1M tokens

A typical coding session:
  You paste 5K tokens of code + context  →  $0.015  (input)
  Model generates 2K tokens of response  →  $0.030  (output)
  Total:                                  $0.045

Key insight: keeping prompts concise saves money on INPUT.
             keeping responses focused saves money on OUTPUT (3-5x more).

Many providers also offer prompt caching — if you repeat the same prefix (system prompt + large context), the provider caches it and charges a fraction of the cost on subsequent requests. Anthropic offers 90% discount on cached input tokens; Google offers similar. This makes the "project memory file" pattern from §2 even more cost-effective.

1.8 The four failure modes to internalize

🎤"Every 'the AI gave me garbage' story I've debugged was really 'the AI was missing context'. That's the theme of this talk."

1.9 Beyond text — multimodal models

Modern LLMs aren't text-only anymore. Multimodal models accept images, audio, and sometimes video as input alongside text. GPT-4o, Claude Sonnet, and Gemini can all "see" screenshots, read diagrams, analyze charts, and process photos. This means you can:

• Paste a screenshot of an error instead of copying the stack trace
• Share a architecture diagram and ask the model to generate code from it
• Feed UI mockups and get HTML/CSS implementations
• Process PDFs and documents with complex layouts (tables, charts, images)

Note: images are converted to tokens internally — a screenshot might cost 1,000+ tokens, so use them intentionally.

§2 — The context window: the model's entire working memory

The context window is the only thing the model can "see". Everything else — including your last conversation — does not exist.

Scale intuition: 200K tokens ≈ a 500-page book ≈ roughly 15–20K lines of code. Sounds huge — fills up fast in real sessions (every tool result, every file read, every retry accumulates).

2.1 Context window sizes across models (2025)

Not all models offer the same context window. Bigger isn't always better — larger windows cost more and can degrade quality at the extremes:

Practical advice: for most engineering tasks (code review, debugging, feature implementation), 128K is more than enough. You only need 200K+ for whole-repo analysis, very long document processing, or large log ingestion. The real bottleneck is rarely the window size — it's how well you fill it (signal vs. noise).

2.2 Context is a budget — three engineering consequences

① Curate, don't dump.

❌  [pastes entire 12-file module]  "why is this slow?"

✅  "Profile output shows 80% time in db/queries.py:get_orders (below).
     The orders table has 40M rows; indexes listed below.
     Why is this slow and how do I fix it without changing the API?"
     + get_orders() source + EXPLAIN ANALYZE output + index list

Relevant 2K tokens beat noisy 100K tokens — signal-to-noise matters more than volume.

② Front-load the stable stuff — the project memory file.

Persist conventions in CLAUDE.md / AGENTS.md / .cursorrules so every session starts pre-loaded:

# CLAUDE.md
## Stack
Python 3.12 · FastAPI · Postgres 16 · pytest · Docker

## Commands
make test    # full suite (use -k for one test)
make lint    # ruff + mypy — must pass before commit

## Conventions
- Type hints everywhere; no bare except
- Never edit db/migrations by hand — use alembic
- One feature per branch; conventional commits

## Gotchas
- tests/fixtures/ is auto-generated — don't hand-edit
- payments/ is PCI-scoped: propose changes, never auto-edit

This is the cheapest productivity win available: 30 minutes to write, benefits every AI session in the repo afterwards. When the AI makes the same mistake twice, the fix is usually one new line in this file.

③ Long chats drift — restart deliberately.

A polluted 150K-token history (failed attempts, dead ends, stale file versions) actively degrades output. When a session gets messy: ask for a summary of decisions + current state, start a fresh session with that summary. A clean 2K-token restart beats a noisy 150K continuation.

🎤"Context is a budget. Spend it on what matters for THIS task."

2.3 Context caching — the cost shortcut

When your system prompt + project rules are the same every request, you're paying to send the same tokens over and over. Prompt caching (supported by Anthropic, Google, and OpenAI) lets providers store the prefix of your request and charge a fraction on repeat calls:

This is why the CLAUDE.md / AGENTS.md pattern from §2.2-② is cost-effective: you send the same 2K-token project rules every request, but only pay full price for the first call in each cache window.

2.4 Conversation management — strategies that work

Real-world conversations are messy. Here are practical patterns for keeping context clean:

§3 — Prompting: from 'fix this' to engineering-grade instructions

3.1 The 5-block anatomy

Every strong prompt has: Role → Context → Task → Constraints → Output format. Weak prompts are usually missing three of them.

❌ WEAK                                  ✅ STRONG
─────────────────────                    ─────────────────────────────────────────────────
"Fix this code."                         You are a senior Python reviewer.          ← ROLE

                                         Context: FastAPI service; cache.py fails   ← CONTEXT
No context, no goal,                     under concurrent load (trace attached).
no definition of done                    Python 3.12, Redis 7.
→ generic guesses.

                                         Task: find and fix the race condition.     ← TASK

                                         Constraints: no new dependencies;          ← CONSTRAINTS
                                         keep the public API; ruff must pass.

                                         Output: (1) unified diff, (2) two-line     ← FORMAT
                                         root cause, (3) a pytest regression test.

🎤Rule of thumb: if a new teammate couldn't act on your prompt, neither can the model.

3.2 Five techniques, in order of power

⓪ Zero-shot — just ask. No examples, no context beyond the task itself. This is the baseline — try it first because it's the cheapest and fastest. Modern models are surprisingly good at zero-shot for straightforward tasks:

Classify this commit message as feat, fix, or chore:
"add retry to S3 client"
→ feat

Zero-shot works when the task is unambiguous and the model has seen similar patterns in training. It fails when you need specific formatting, domain-specific logic, or consistent edge case handling — that's when you move to few-shot.

① Few-shot — show, don't tell. The model copies patterns far better than it follows abstract rules. Providing 3–5 examples of input→output pairs dramatically improves consistency and accuracy:

Classify commit messages as feat | fix | chore:
"add retry to S3 client"  → feat
"bump lodash to 4.17.21"  → chore
"null check in parser"    → fix
Now classify these 40: ...

② Chain-of-thought — plan before code. Forcing reasoning first measurably cuts logic errors on non-trivial tasks. The model shows its work — like a math student who gets better grades when they write out each step instead of jumping to the answer:

Before writing any code:
1. Outline the algorithm
2. List the edge cases (empty input, concurrency, unicode, limits)
3. Then implement + tests
Do not skip steps.

Why it works: when the model commits to a plan in writing, it's less likely to contradict itself mid-implementation. You also get to review the plan before any code is written — catching architectural mistakes early. Research shows CoT improves accuracy by 10–40% on complex reasoning tasks depending on the model.

③ Structured output — machine-readable answers you can pipe into scripts, dashboards, CI. Define the exact format you want and the model will follow it:

Review this PR. Return JSON only, no prose:
[{ "severity": "high|med|low", "file": str, "line": int, "issue": str, "fix": str }]

④ Self-critique — one extra turn, outsized defect catch rate.

Now review your own diff against this checklist:
concurrency, error handling, security, naming, edge cases.
List problems found, then fix them.

⑤ Grounding / RAG — kill hallucinated APIs. Paste (or retrieve) the authoritative source and constrain the model to it:

Answer ONLY from the attached API docs.
If the answer is not in the docs, say "not in docs" — do not guess.

3.3 Common prompting mistakes (and the fix)

3.4 Think in pipelines, not single prompts

Each stage gets a small, focused prompt that validates the previous one. Small steps beat one giant ask — and this pipeline is exactly what agentic tools automate. →

3.5 The prompt iteration workflow

Prompting is iterative, not one-shot. Even experienced engineers write 3–5 versions before landing on a reliable prompt. Here's the workflow:

Key principle: treat prompts like code — version them, test them, review them. A prompt that works for 90% of cases but fails silently on the other 10% is a bug, not a feature. Store important prompts in files (not just in chat history) so you can diff, roll back, and share them.

3.6 Meta-prompting — using AI to improve prompts

One of the most powerful techniques: ask the model to critique and improve your prompt. The model understands its own failure modes better than you'd expect:

Here is my current prompt for code review:
"""
Review this PR diff. Focus on bugs and security issues.
"""

Problems with this prompt:
1. What's missing? (role, context, constraints, output format)
2. What failure modes will I hit?
3. Rewrite it using the 5-block anatomy. Show before/after.

You can also use meta-prompting to generate few-shot examples — ask the model to create 10 diverse test cases for a classification task, then use those as examples in your actual prompt.

3.7 Evaluating prompts — measure, don't guess

How do you know if a prompt is "good"? You measure it:

Build a small eval dataset — 20–50 input/output pairs that represent your real use cases. Run every prompt change against this dataset before deploying. This is the same discipline as unit testing, applied to prompts.

§4 — Agentic coding: the AI does the task, not just the line

Agent = LLM + tools + a loop. It doesn't just suggest code — it edits files, runs commands, reads the results, and iterates until done (or stuck). The key enabler is tool use (also called function calling): the model generates a structured JSON call describing which tool to invoke and with what arguments, your code executes it, and the result is fed back into the conversation.

4.1 How tool use actually works

When you give an agent tools, the model doesn't directly execute them — it generates a tool call in a structured format, and your runtime executes it. Here's what happens under the hood:

1. You define tools:
   tools = [
     { name: "read_file",  parameters: { path: string } },
     { name: "write_file", parameters: { path: string, content: string } },
     { name: "run_tests",  parameters: { command: string } },
   ]

2. Model generates a tool call (not visible to user):
   { "tool": "read_file", "args": { "path": "src/auth.py" } }

3. Your code executes it, returns the result:
   "def authenticate(user, token): ..."

4. Result goes back into context, model continues:
   "I see the auth module uses JWT. The refresh token rotation..."

5. Model generates next tool call:
   { "tool": "write_file", "args": { "path": "src/auth.py", "content": "..." } }

6. Loop continues until the model decides it's done.

The model chooses which tools to call based on the tool descriptions (their docstrings/JSON schemas). This is why good tool descriptions are critical — if the model doesn't understand what a tool does, it won't use it correctly (or at all).

4.2 The workflow to copy tomorrow: Explore → Plan → Code → Commit

# 1 — EXPLORE (read-only)
> Read the repo and explain how authentication works end-to-end.
> Don't write any code yet.

# 2 — PLAN
> We need refresh-token rotation. Write a step-by-step plan to PLAN.md.
> Flag risky steps. Wait for my approval.

# 3 — CODE (small steps, test-gated)
> Approved. Implement steps 1–3 of PLAN.md.
> Run the test suite after every step.

# 4 — COMMIT
> All green? Commit with a conventional message, push a branch,
> and open a draft PR with a summary.

Why it works:

• Plan first, approve, then build — skipping the plan is the #1 cause of runaway, wrong-direction diffs.
• "Run tests after every step" gives the agent a feedback signal to self-correct (tests become the objective function).
• Explore is free — agents are excellent at "how does X work here?"; use them to learn unfamiliar code.
• You own the merge — the agent opens a draft PR; a human reviews and merges. Always.

🎤"The single highest-leverage habit: make the AI produce a plan and approve it BEFORE it touches code."

4.3 Error recovery — when things go wrong

Agents fail. Tests break, commands error out, the model goes off track. The difference between a useful agent and a frustrating one is how it recovers. Here are the patterns:

4.4 Multi-agent systems — divide and conquer

For complex tasks, a single agent can struggle with context overload. Multi-agent systems decompose the work across specialized agents, each with a focused role:

When to use multi-agent: when a task requires different expertise areas (design + implementation + testing), when the context would be too large for a single agent, or when you need adversarial review (one agent builds, another critiques). Frameworks like CrewAI, AutoGen, and LangGraph make this pattern easier to implement.

4.5 Memory — short-term and long-term

Agents need to remember things beyond the current conversation:

4.6 Agent failure modes — what goes wrong and how to prevent it

§5 — MCP: wiring AI into your real systems

The problem: an LLM in a chat box can't see your GitHub, your database, Jira, or your deploy pipeline. Historically, every AI app needed a custom integration for every tool:

Without a standard:   N apps × M tools  =  N×M custom integrations   😱
With MCP:             N apps + M servers =  N+M                       ✅  build once, use everywhere

MCP (Model Context Protocol) is an open standard (open-sourced by Anthropic, late 2024; now adopted across the industry — think "USB-C for AI").

5.1 Architecture

Each server exposes three primitives:

5.2 How MCP servers communicate — transport mechanisms

MCP supports two transport mechanisms, and choosing the right one depends on your deployment:

Rule of thumb: start with stdio for local experimentation and prototyping. Move to streamable HTTP when you need to share a server across team members, deploy to production, or run the server on a dedicated machine with better resources than the developer's laptop.

5.3 What a tool call actually looks like

The model decides which tools to call, chains them, and synthesizes — you just asked a question in English.

5.3 Plugging in an existing server — a few lines of config

{
  "mcpServers": {
    "github":   { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"] },
    "postgres": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-postgres",
                                             "postgresql://localhost/appdb"] }
  }
}

Ready-made servers already exist for: GitHub · GitLab · Postgres · Slack · Jira · Filesystem · Sentry · Google Drive · browsers (Puppeteer) — plus a fast-growing public registry.

Then you just ask — across systems:

• "List my open PRs and summarize why their checks are failing."
• "Which tables grew the most this week? Chart it."
• "Take this Sentry stack trace, find the bug, open a fix PR, and post the summary in #eng."

5.4 Building your own server — wrap an internal API in ~15 lines

# pip install "mcp[cli]"
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("deploy-tools")

@mcp.tool()
def deploy_status(service: str) -> str:
    """Latest deploy status for a service."""      # ← docstrings ARE the UX:
    return ci.get_status(service)                  #   the model picks tools by reading them

@mcp.tool()
def rollback(service: str, version: str) -> str:
    """Roll back a service (requires human approval flag)."""
    return ci.rollback(service, version)

if __name__ == "__main__":
    mcp.run()   # stdio for local; streamable HTTP for remote

High-leverage targets to wrap first: deploy status, feature flags, ticketing, internal CRM, observability — anything people currently look up by hand.

5.6 Authentication and credential management

MCP servers need real credentials to access real systems. How you handle these determines whether your setup is secure or a liability:

Never hardcode tokens in source code. Never commit .env files to git. Use per-tool, per-repo tokens with minimum required scopes. If a GitHub MCP server only needs read access to one repo, don't give it a personal access token with admin rights to every repo.

5.5 Related tech in the same family

5.6 ⚠️ Security — the part you must not skip

MCP servers run with real credentials and have real side effects:

AI drafts. Humans own the merge. Branch protection, mandatory review, and CI gates stay exactly as strict as before — AI code gets zero exemptions.

5.8 MCP vs. alternatives — why the standard matters

MCP isn't the only way to give an LLM access to tools. Here's how it compares:

5.9 Scaling MCP — from prototype to production

Moving from "it works on my machine" to a production MCP setup requires thinking about:

• Server lifecycle: who manages the server process? For stdio, the host manages it. For HTTP, you need a process manager (systemd, Docker, Kubernetes).
• Concurrency: multiple agents or users hitting the same MCP server simultaneously — ensure your server handles concurrent tool calls without race conditions.
• Rate limiting: tools that call external APIs (GitHub, Jira) have rate limits. Your MCP server should handle rate limit errors gracefully and queue requests.
• Monitoring: log every tool call with timestamps, inputs, outputs, and latency. You need to answer "what did the agent do?" after the fact.
• Cost tracking: each tool call consumes tokens (the call itself + the result fed back into context). Track per-tool costs to identify expensive operations.

§6 — Takeaways

• ✅LLMs predict, they don't know — verify, run, review. Always.
• ✅Context is a budget — curate it, persist project rules in CLAUDE.md, restart drifted sessions.
• ✅Prompt like a tech lead — Role, Context, Task, Constraints, Format… and demand a plan first.
• ✅Climb the technique ladder — few-shot → chain-of-thought → structured output → self-critique → grounding → pipelines.
• ✅Agents do tasks, not lines — Explore → Plan → Code → Commit, with tests as the feedback loop.
• ✅MCP turns AI into a teammate wired into your real systems — plug in servers today, wrap one internal API next; guardrails are non-negotiable.

🎤Closing ask: "This week, everyone tries ONE thing: rewrite a real prompt with the 5 blocks, run the Explore→Plan→Code→Commit loop on one ticket, or plug one MCP server into your agent. Post the result — good or bad — in #ai-engineering."

Go deeper: docs.claude.com (prompt engineering & Claude Code) · modelcontextprotocol.io (MCP spec, SDKs, server registry) · mermaid.live (edit these diagrams)