Lesson 4

LLM Observability

Monitoring & debugging

The Problem: AI in production is a black box. How do you know when it's making mistakes, slowing down, or costing too much? How do you debug issues?

The Solution: Health Monitoring for Your AI

LLM Observability means tracking, logging, and monitoring all aspects of your AI system in production. It's like a patient monitoring system in a hospital — vital signs, alerts, and detailed records of everything that happens. Key metrics include latency and cost per request, and it feeds into your guardrails strategy.

Think of it like patient monitoring in a hospital:

1. Instrument all LLM calls: Log every request with: prompt text, response text, token counts (input/output), latency (ms), model used, and cost
2. Set up dashboards: Track cost/day, p95 latency, error rate, and tokens/request — visualize trends over time, not just current values
3. Create alerts: Alert on: cost spike > 2x daily average, p95 latency > 5s, error rate > 5%, quality score drop > 10%
4. Review traces for bottlenecks: Drill into slow requests — is it oversized context? Missing cache? Wrong model? Each trace tells a story
5. Iterate prompts based on data: Use observability data to find and fix the worst-performing prompts first — the top 10% of costly prompts usually account for 50%+ of spend

What to Monitor

Performance: Track p50, p95, p99 latency per endpoint — p95 > 3s means 5% of users are waiting too long
Debug This Trace: Trace example: request (5ms) -> prompt build (120ms) -> LLM API (4,200ms) -> guardrail (50ms) -> response. LLM step is 10x slower than expected — possible causes: oversized context, model congestion, or missing cache hit
Cost: Track cost-per-conversation (not just per-request) — a multi-turn chat can cost 10-50x more than a single exchange
Quality: Track hallucination rate, user thumbs-up/down ratio, and relevance scores — quality regression is silent without metrics

Fun Fact: Teams that add observability from day 1 typically find 30-40% cost savings opportunities within the first month just by seeing their actual usage patterns — most discover that their longest prompts are also the least effective ones.

Try It Yourself!

See what LLM monitoring looks like and understand the key metrics to track.

Try it yourself

Interactive demo of this technique

Technique Comparison

Demo Mode

Pre-recorded responses

TaskBeginnerAnalysis

Debug LLM response quality issue in production

Without technique

Prompt

Users complain that the bot answers incorrectly. No logs. How to find the problem?

Response

Try rewriting the prompt and see if it gets better. Maybe the model isn't suitable for the task.

Tokens:25/30

Time:350ms

Quality:

With Observability

With technique

Prompt

You are an SRE engineer specializing in LLM systems. Problem: users complain about incorrect bot responses. No logs. Design a minimal observability system for the LLM: 1. What to log per request? 2. What metrics to collect? 3. What alerts to set up? Format: specific fields/metrics with example values.

Response

1. Per-request logging:

{
  "request_id": "req_abc123",
  "timestamp": "2024-01-15T10:30:00Z",
  "user_id": "u_456",
  "prompt": "...",
  "response": "...",
  "model": "gpt-4",
  "tokens_in": 150,
  "tokens_out": 200,
  "latency_ms": 1200,
  "temperature": 0.7,
  "user_rating": null
}

2. Metrics (Prometheus/Grafana):

llm_latency_p95: < 3s (alert > 5s)
llm_error_rate: < 1% (alert > 5%)
llm_token_cost_hourly: track trend
llm_user_satisfaction: thumbs up/down ratio

3. Alerts:

Latency P95 > 5s → PagerDuty
Error rate > 5% for 5 min → Slack
Satisfaction < 70% for 1 hour → email team

👁️Without logs, impossible to tell: is it a bad prompt, wrong temperature, or hallucinations?

🧠Minimum set: request_id + prompt + response + metrics → reproducibility of any issue

✅Alerts on latency + error rate + satisfaction catch issues BEFORE mass complaints

Tokens:95/250

Time:1500ms

Quality:

Why this works

LLM observability minimum: log every request (prompt + response + metrics) + dashboard (latency, error rate, satisfaction) + alerts. Without this, you're debugging blind.

1 / 2

Practice Challenges

Create a free account to solve challenges

4 AI-verified challenges for this lesson

Related lessons:Cost Optimization Guardrails

This lesson is part of a structured LLM course.

My Learning Path

Lesson 4

LLM Observability

Monitoring & debugging

The Problem: AI in production is a black box. How do you know when it's making mistakes, slowing down, or costing too much? How do you debug issues?

The Solution: Health Monitoring for Your AI

Think of it like patient monitoring in a hospital:

1. Instrument all LLM calls: Log every request with: prompt text, response text, token counts (input/output), latency (ms), model used, and cost
2. Set up dashboards: Track cost/day, p95 latency, error rate, and tokens/request — visualize trends over time, not just current values
3. Create alerts: Alert on: cost spike > 2x daily average, p95 latency > 5s, error rate > 5%, quality score drop > 10%
4. Review traces for bottlenecks: Drill into slow requests — is it oversized context? Missing cache? Wrong model? Each trace tells a story
5. Iterate prompts based on data: Use observability data to find and fix the worst-performing prompts first — the top 10% of costly prompts usually account for 50%+ of spend

What to Monitor

Performance: Track p50, p95, p99 latency per endpoint — p95 > 3s means 5% of users are waiting too long
Debug This Trace: Trace example: request (5ms) -> prompt build (120ms) -> LLM API (4,200ms) -> guardrail (50ms) -> response. LLM step is 10x slower than expected — possible causes: oversized context, model congestion, or missing cache hit
Cost: Track cost-per-conversation (not just per-request) — a multi-turn chat can cost 10-50x more than a single exchange
Quality: Track hallucination rate, user thumbs-up/down ratio, and relevance scores — quality regression is silent without metrics

Try It Yourself!

See what LLM monitoring looks like and understand the key metrics to track.

Try it yourself

Interactive demo of this technique

Technique Comparison

Demo Mode

Pre-recorded responses

TaskBeginnerAnalysis

Debug LLM response quality issue in production

Without technique

Prompt

Users complain that the bot answers incorrectly. No logs. How to find the problem?

Response

Try rewriting the prompt and see if it gets better. Maybe the model isn't suitable for the task.

Tokens:25/30

Time:350ms

Quality:

With Observability

With technique

Prompt

Response

1. Per-request logging:

{
  "request_id": "req_abc123",
  "timestamp": "2024-01-15T10:30:00Z",
  "user_id": "u_456",
  "prompt": "...",
  "response": "...",
  "model": "gpt-4",
  "tokens_in": 150,
  "tokens_out": 200,
  "latency_ms": 1200,
  "temperature": 0.7,
  "user_rating": null
}

2. Metrics (Prometheus/Grafana):

llm_latency_p95: < 3s (alert > 5s)
llm_error_rate: < 1% (alert > 5%)
llm_token_cost_hourly: track trend
llm_user_satisfaction: thumbs up/down ratio

3. Alerts:

Latency P95 > 5s → PagerDuty
Error rate > 5% for 5 min → Slack
Satisfaction < 70% for 1 hour → email team

👁️Without logs, impossible to tell: is it a bad prompt, wrong temperature, or hallucinations?

🧠Minimum set: request_id + prompt + response + metrics → reproducibility of any issue

✅Alerts on latency + error rate + satisfaction catch issues BEFORE mass complaints

Tokens:95/250

Time:1500ms

Quality:

Why this works

LLM observability minimum: log every request (prompt + response + metrics) + dashboard (latency, error rate, satisfaction) + alerts. Without this, you're debugging blind.

1 / 2

Practice Challenges

Create a free account to solve challenges

4 AI-verified challenges for this lesson

Related lessons:Cost Optimization Guardrails

This lesson is part of a structured LLM course.

My Learning Path