Key Metrics
Essential metrics for Claude-based applications include: response latency (time to first token, total response time), token usage (input, output, cached), error rates (by error type — 429, 500, context overflow), response quality (task completion rate, user feedback), and cost (per request, per user, per task).
Track these metrics at multiple granularities: per-request (for debugging), per-hour (for trend analysis), and per-day (for cost management). Set alerts on anomalies — sudden latency spikes, error rate increases, or cost jumps.
Tracing
Tracing records the full execution path of a request through your system. For agentic applications, this means tracking: the initial prompt, each model call and response, each tool call and result, each guardrail check, and the final output.
Traces are essential for debugging. When a user reports a bad result, the trace shows exactly what happened — which tools were called, what the model was thinking at each step, and where things went wrong. Without tracing, debugging agents is guesswork.
Logging Best Practices
Log at appropriate levels: errors (always), warnings (rate limit approaches, quality degradation), info (request summaries, model selection), and debug (full prompts and responses, tool call details).
Be careful about logging sensitive information. Full prompts and responses may contain user data. Implement log redaction for PII and configure log retention policies. In production, log at the info level by default and enable debug logging when investigating issues.
Key Concept
Monitor the Model, Not Just the Application
Traditional application monitoring (uptime, error rates, response time) is necessary but insufficient for AI applications. You must also monitor model-specific concerns: response quality degradation, prompt injection attempts, unexpected tool usage patterns, and cost anomalies. Model behavior can change without any code change — a new model version, changed API behavior, or shifting user patterns can all affect quality.
Exam Traps
Only monitoring errors and latency
Missing quality monitoring means you won't detect gradual degradation in model output. Track task completion rates and user feedback as quality signals.
Logging full prompts and responses in production
Full logging can capture sensitive user data. Implement redaction and consider privacy regulations before logging detailed model interactions.
Not implementing tracing for agentic systems
Without tracing, debugging multi-step agent behavior is extremely difficult. The exam expects you to know that tracing is a critical capability for production agents.
Check Your Understanding
Your Claude-powered application has been running in production for a month. Users report that response quality has decreased, but error rates and latency are normal. What monitoring would help diagnose this?
Build Exercise
Build a Monitoring Dashboard
What you'll learn
- Track key AI application metrics
- Implement request tracing
- Set up alerts for anomalies
- Build a cost tracking system
Create a metrics collector that captures latency, token usage, and error type for each API call. Store metrics in a structured format.
WHY: Metrics collection is the foundation of observability.
YOU SHOULD SEE: A growing log of metrics with timestamp, latency, tokens, and status for each request.
Implement request tracing for an agentic loop: record each step (model call, tool call, guardrail check) with timing and outcome.
WHY: Tracing provides the detail needed to debug complex agent behavior.
YOU SHOULD SEE: A trace object with nested spans showing the full execution path.
Build a simple dashboard that shows: requests per minute, average latency, error rate, and cumulative cost. Display rolling 1-hour windows.
WHY: A dashboard makes metrics actionable — trends and anomalies become visible.
YOU SHOULD SEE: A dashboard with key metrics updating in real time.
Add alert rules: trigger alerts when error rate exceeds 5%, latency exceeds 2x baseline, or daily cost exceeds the budget threshold.
WHY: Alerts ensure you are notified of issues before they impact users.
YOU SHOULD SEE: Alerts fire when simulated anomalies are introduced.
Sources
- Error Handling and Monitoring— Anthropic Documentation
- Claude Agent SDK Tracing— Anthropic Documentation