Eval Suite & Playground

Test agents with golden test sets and the prompt playground

Builder Guide: Eval Suite & Prompt Playground

This guide covers two closely related Studio features for testing and iterating on agent behavior: the Eval Suite for systematic quality assessment, and the Prompt Playground for rapid interactive experimentation.

Where to use: Agent Studio > Evals (http://localhost:3001/evals) and Agent Studio > Playground (http://localhost:3001/playground).

Table of Contents

• Eval Suite Overview
  • Core Concepts
  • Creating Test Cases
  • Running Evaluations
  • LLM-as-Judge Grading
  • Regression Tracking
  • API Reference
• Prompt Playground Overview
  • Features
  • Model Selection
  • Parameter Controls
  • Version Management
  • Live Metrics
• Workflow: From Playground to Eval
• Model Browser

Eval Suite Overview

The Eval Suite provides systematic, repeatable quality testing for agents. Instead of manually chatting with an agent to check if it works, you define golden test cases and run them automatically.

Core Concepts

Creating Test Cases

Test cases are managed in Agent Studio at /evals. Each test case includes:

• Input prompt -- The message sent to the agent
• Expected output -- The reference answer (used by graders)
• Tags -- Categorization labels (e.g., safety, accuracy, edge-case)
• Grading method -- How to evaluate the response (llm-judge, exact-match, contains, regex)

Example test case:

{
  "input": "What is the capital of France?",
  "expectedOutput": "Paris",
  "tags": ["factual", "geography"],
  "assertion": "contains_expected"
}

Assertion types & the required criterion (as built)

The editor picks an assertion type; whichever criterion that method grades against is required (the Save button stays disabled until it's filled, and the backend rejects an empty criterion — see "no free passes" below).

For "expect output semantically": use llm_judge_reference (put the ideal answer in Expected Output) when a correct answer exists; use llm_judge_pointwise (write a rubric of explicit criteria) when the answer is open-ended. The judge passes at a default threshold of 3/5 and runs at temperature 0.1.

No free passes: an empty Expected Output (contains_expected/llm_judge_reference) or an empty rubric (llm_judge_pointwise) now fails the case instead of silently passing — empty criteria were the source of a previous "always green" bug.

Keep a negative control. Add at least one case a correct agent should fail (e.g. ask for "apple", expect "banana" with contains_expected). If it ever passes, the harness is broken. The seeded "Starter checks (example)" suite on the tester agent demonstrates this (5 real cases + 1 negative control).

Running Evaluations

1. Navigate to Evals in Studio.
2. Select the agent and test cases to include.
3. Click Run Evaluation.
4. The system executes each test case against the agent and records:
   • The actual response
   • The grade (pass/fail/partial)
   • Latency (ms)
   • Token usage (input/output)
   • Cost estimate

Results are displayed in a table with pass/fail indicators and can be filtered by tag or status.

In the Studio Test drawer Eval tab, the latest run shows an X/Y passed summary plus a per-case breakdown (each case ✓/✗ with its reason and judge score) — so a failing case (a true negative) is visible with why, e.g. ✗ contains_expected — Output does not contain expected: "banana". Runs are ordered newest-first, so "Latest run" always reflects your most recent run (an earlier bug showed the oldest run, masking later failures).

LLM-as-Judge Grading

For test cases that require nuanced evaluation (not just string matching), the eval suite uses an LLM as a judge:

1. The judge LLM receives: the input, the expected output, and the actual response.
2. It evaluates on criteria such as: factual accuracy, completeness, tone, safety.
3. It returns a structured grade with a score and explanation.

The judge model is configurable per eval run (defaults to the same model used by the agent).

Regression Tracking

The eval suite tracks results over time. When you modify an agent's prompt, model, or tools:

1. Re-run the eval suite.
2. The system compares results to the previous run.
3. Regressions are highlighted -- test cases that previously passed but now fail.
4. This prevents accidental quality degradation when iterating on agent configuration.

Eval API Reference

# List eval test cases
GET /api/evals/cases

# Create a test case
POST /api/evals/cases
{
  "input": "...",
  "expectedOutput": "...",
  "tags": ["..."],
  "gradingMethod": "llm-judge"
}

# Run an evaluation
POST /api/evals/runs
{
  "agentId": "agent-1",
  "caseIds": ["case-1", "case-2"],
  "model": "claude-sonnet-4"
}

# Get eval run results
GET /api/evals/runs/:runId

Prompt Playground Overview

The Prompt Playground is an interactive testing environment for experimenting with prompts, models, and parameters in real time. It is the fastest way to iterate on agent behavior before committing changes.

Playground Features

Model Selection

The playground's model selector shows all models available to the current tenant, including:

• Model name and provider
• Hosting type (EU self-hosted vs. external)
• Cost tier (budget / standard / premium / reasoning)
• D4 policy indicators -- whether the model is allowed by the tenant's model access policies

Models that are blocked by policy are shown but greyed out with an explanation.

Parameter Controls

Version Management

The playground supports saving and comparing prompt versions:

1. Write or modify a system prompt in the Monaco editor.
2. Click Save Version to snapshot the current prompt + parameters.
3. Switch between versions to compare behavior.
4. When satisfied, copy the prompt to the Agent Designer.

Live Metrics

Each playground interaction shows:

• Latency -- Time to first token and total response time
• Token usage -- Input tokens (prompt) and output tokens (response)
• Cost -- Estimated cost based on the selected model's pricing
• Model -- Which model generated the response

Workflow: From Playground to Eval

The recommended workflow for building high-quality agents:

1. Explore in Playground -- Experiment with different prompts, models, and parameters.
2. Save promising versions -- When a prompt produces good results, save the version.
3. Create eval test cases -- Based on your exploration, define golden test cases that capture expected behavior.
4. Apply to Agent Designer -- Copy the best prompt/model/params to the agent's configuration.
5. Run eval suite -- Verify the agent passes all test cases.
6. Iterate -- When you need to change the agent, re-run evals to catch regressions.

Model Browser

The Model Browser (Studio > Models, http://localhost:3001/models) provides a catalog of all LLM models available on the platform:

The model browser is read-only. Model access policies are managed by admins in the Admin Console.

Related Documentation

• Registering Agents Guide -- Full agent lifecycle and the 9-tab designer
• Admin Model Policies Guide -- Configuring model access policies
• LLM Provider Reference -- T-Systems AIFS model catalog
• External Agents Guide -- Integrating external agent frameworks