Skip to main content

AI Evaluation

Systematically test, score, and improve your AI workflow outputs.

Why Evaluate?​

AI outputs can vary. Evaluation helps you:

  • Measure quality consistently
  • Compare prompt versions
  • Identify failure cases
  • Optimize performance

Evaluation Concepts​

Traces​

A trace captures one AI interaction:

  • Input (user query)
  • Output (AI response)
  • Context (KB results, variables)
  • Metadata (latency, tokens)

Trace collection captures AI node inputs/outputs, stores them for analysis, and lets you build evaluation datasets.

Evaluators​

Functions that score traces. Use:

  • LLM Judges: AI-based evaluation
  • Regex Match: Pattern matching
  • Keyword Check: Required terms
  • Custom Scripts: Your own logic

Common evaluator dimensions:

EvaluatorMeasures
RelevanceAnswer addresses question
AccuracyFactually correct
CompletenessAll points covered
ToneAppropriate style
SafetyNo harmful content

Score Configs​

Define what "good" looks like:

  • Scoring criteria
  • Pass/fail thresholds
  • Weighted metrics

Golden Sets​

Curated reference datasets of ideal outputs:

  • Create from best executions
  • Manual curation
  • Used for comparison
{
  "input": "What are your pricing plans?",
  "expected_output": "We offer three plans...",
  "tags": ["pricing", "faq"]
}

A/B Testing​

Compare different approaches:

  • Prompt variations
  • Model comparisons
  • Parameter tuning

Creating Evaluations​

1. Capture Traces​

Enable tracing in your workflow:

  1. Open workflow
  2. Go to Settings > Tracing
  3. Enable Capture Traces
  4. Run workflow to collect data

2. Create Evaluator​

  1. Go to Automation > AI Evaluation
  2. Click + Create Evaluator
  3. Configure:
    • Name - What it measures
    • Type - LLM-based or rule-based
    • Prompt - Scoring criteria

Example LLM evaluator prompt:

Rate this response on relevance from 1-5:

Question: {{input}}
Response: {{output}}

Score 5: Directly answers the question
Score 3: Partially relevant
Score 1: Completely off-topic

Output only the number.

3. Build Golden Set​

  1. Click + Create Golden Set
  2. Add test cases:
    • Manual entry
    • Import from traces
    • Upload CSV
  3. Include expected outputs

4. Run Evaluation​

  1. Select traces or golden set
  2. Choose evaluators
  3. Click Run Evaluation
  4. View results

Evaluation Types​

Automated Evaluation​

Run evaluators on traces automatically:

  • Continuous quality monitoring
  • Alerts on degradation
  • Trend analysis

A/B Testing​

Compare prompt versions:

  1. Create two prompt variants
  2. Split traffic
  3. Collect traces from both
  4. Compare evaluation scores
  5. Select winner

Calibration​

Ensure consistent scoring:

  1. Have team members score samples
  2. Compare with evaluator scores
  3. Adjust evaluator if needed

Metrics Dashboard​

Track over time:

  • Average scores - By evaluator
  • Score distribution - Histogram
  • Failure rate - Below threshold
  • Trends - Improvement over time

Iterative Optimization​

Process​

  1. Baseline - Evaluate current state
  2. Identify issues - Low-scoring traces
  3. Hypothesize - What could improve
  4. Test - Try new prompt
  5. Measure - Compare scores
  6. Deploy - If improved

Common Improvements​

IssueSolution
Off-topic responsesBetter system prompt
Missing infoAdd examples
Wrong formatExplicit format instructions
HallucinationsAdd KB context

Best Practices​

  1. Start simple - Few evaluators first
  2. Real data - Use production traces
  3. Multiple metrics - Different quality aspects
  4. Regular reviews - Check edge cases
  5. Document decisions - Why changes were made

Next Steps​

Last updated on