Deployment & Monitoring

Deploy pipeline, canary releases, monitoring, and traces

Deployment & Monitoring

This guide covers deploying agents from development to production and monitoring their health, performance, and cost.

Deployment Pipeline

Overview

Agents follow a structured deployment pipeline:

Draft → Build → Test → Stage → Canary → Production

Each stage has validation gates:

• Build: TypeScript compilation, schema validation
• Test: Eval suite passes, guardrail compliance
• Stage: Smoke tests in staging environment
• Canary: Small % of production traffic
• Production: Full traffic, monitored

Creating a Deployment

1. Navigate to Studio > Deployments
2. Click Create Deployment
3. Select the agent and version
4. Choose deployment strategy:

Deployment Configuration

agent: agent-hr-assistant
version: 2.1.0
strategy: canary
canary:
  initialPercent: 10
  incrementPercent: 25
  intervalMinutes: 15
  successThreshold: 95  # % success rate required to promote
rollback:
  automatic: true
  triggerOn:
    - successRate < 90
    - latencyP95 > 5000
    - errorRate > 5

Rollback

Rollback options:

• Automatic — triggered when metrics drop below thresholds
• Manual — click Rollback in the Deployments page
• Version pin — revert to a specific previous version

Monitoring Dashboard

Studio > Monitoring

The monitoring dashboard shows real-time agent health across four views:

1. Agent Health Grid

At-a-glance status of all deployed agents:

• Green: healthy (success rate > 95%)
• Yellow: degraded (success rate 80-95%)
• Red: unhealthy (success rate < 80%)
• Gray: no traffic

2. Performance Metrics

Per-agent metrics updated in real-time:

3. Cost Attribution

Per-agent cost tracking:

• Token usage — input + output tokens per model
• Cost per conversation — average LLM cost
• Cost per day/week/month — trend over time
• Cost by model — which models drive spend

4. Tool Usage

Which tools are being called and how often:

• Tool invocation count
• Tool success/failure rate
• Tool latency
• Most-used tool combinations

Trace Explorer

Studio > Traces provides detailed execution traces:

1. Timeline view — see each step of a conversation
2. Tool calls — input/output for every tool invocation
3. LLM calls — prompt, completion, token count, latency
4. Memory operations — what was stored/retrieved
5. Guardrail checks — which guardrails fired and why

Use traces to:

• Debug conversation failures
• Optimize prompt engineering
• Identify slow tool calls
• Verify guardrail behavior

Behavioral Baselines

The platform automatically establishes behavioral baselines for each agent:

• Normal conversation length — average turns per conversation
• Typical tool usage — which tools, how often
• Response patterns — average response length, sentiment
• Error patterns — common error types

When behavior deviates significantly from the baseline, an alert is generated.

Setting Up Alerts

In Admin Console

1. Navigate to Admin > Dashboard
2. Configure alert thresholds per agent or globally:

Prometheus & Grafana

The platform exports metrics to Prometheus:

• agentic_agent_request_duration_seconds — response time histogram
• agentic_agent_requests_total — request counter (by status, agent)
• agentic_tool_invocations_total — tool call counter
• agentic_llm_tokens_total — token usage counter (by model, direction)
• agentic_memory_operations_total — memory operation counter

Pre-built Grafana dashboards are available in infra/grafana/.

Best Practices

Deployment

1. Always use canary for production deployments
2. Run eval suite before every deployment
3. Set automatic rollback with conservative thresholds
4. Deploy during low traffic when possible
5. Monitor for 30 minutes after promotion before considering stable

Monitoring

1. Set alerts before deploying — don't deploy without monitoring
2. Review traces weekly — look for patterns, not just failures
3. Track cost trends — catch runaway costs early
4. Compare versions — use A/B metrics to validate improvements
5. Baseline new agents — let the platform establish baselines for 1-2 weeks before trusting anomaly detection

Cost Optimization

1. Use appropriate models — don't use Claude Sonnet for simple classification
2. Set token limits — cap max_tokens to prevent runaway generation
3. Cache common queries — use memory to avoid redundant LLM calls
4. Monitor tool usage — expensive tools (external APIs) add up
5. Review cost attribution — identify which agents drive the most spend

Next Steps

• Agent Design Patterns — patterns for reliable agents
• Security & Compliance — production security checklist
• FinOps & Cost Management — detailed cost management
• Troubleshooting — common deployment issues