Evaluation framework

The eval system helps you benchmark different LLMs for transaction categorization, merchant detection, and chat assistant functionality.

Quick start

Import a dataset

bin/rails 'evals:import_dataset[db/eval_data/categorization_golden_v1.yml]'

Run an evaluation

bin/rails 'evals:run[categorization_golden_v1,openai,gpt-4.1]'

Compare models

MODELS=gpt-4.1,gpt-4o-mini rake evals:compare[categorization_golden_v1]

Available commands

Dataset management

# List all datasets
rake evals:list_datasets

# Import dataset from YAML
rake evals:import_dataset[path/to/file.yml]

# Export manually categorized transactions
rake evals:export_manual_categories[family-uuid]

Running evaluations

# Run evaluation
rake evals:run[dataset_name,provider,model]

# Compare multiple models
MODELS=model1,model2 rake evals:compare[dataset_name]

# Quick smoke test
rake evals:smoke_test

# CI regression test
rake evals:ci_regression[dataset,provider,model,threshold]

Viewing results

# List recent runs
rake evals:list_runs

# Show detailed report
rake evals:show_run[run_id]

# Generate comparison report
rake evals:report[run_ids]

Cascade and analysis

# Benchmark the Bayes -> provider cascade
# (measures provider accuracy on the transactions Bayes declined to categorize)
rake evals:cascade[bayes_run_id,provider_run_id1,provider_run_id2]
# or: BAYES=<run_id> PROVIDERS=<id1>,<id2> rake evals:cascade

# Analyze a candidate provider against a baseline
# (shows calibration, disagreement, and a recommendation)
rake evals:analyze[baseline_run_id,candidate_run_id]
# or: BASELINE=<run_id> CANDIDATE=<run_id> rake evals:analyze

Langfuse integration

Track experiments in Langfuse for side-by-side comparison and analysis.

Setup

export LANGFUSE_PUBLIC_KEY="pk-..."
export LANGFUSE_SECRET_KEY="sk-..."
export LANGFUSE_REGION="eu"  # Optional, defaults to eu

Commands

# Check connection
bin/rails 'evals:langfuse:check'

# Upload dataset
bin/rails 'evals:langfuse:upload_dataset[categorization_golden_v1]'

# Run experiment
bin/rails 'evals:langfuse:run_experiment[categorization_golden_v1,gpt-4.1]'

# List datasets in Langfuse
bin/rails 'evals:langfuse:list_datasets'

What gets created

When you run a Langfuse experiment, the system creates:

Dataset - Named eval_<your_dataset_name> with all samples
Traces - One per sample showing input/output
Scores - Accuracy scores (0.0 or 1.0) for each trace
Dataset Runs - Links traces to dataset items for comparison

In the Langfuse UI you can:

Compare runs side-by-side
Filter by score, model, or metadata
Track accuracy over time
Analyze per-sample results

Evaluation types

Categorization

Tests transaction categorization accuracy across difficulty levels. Metrics:

Accuracy
Precision, recall, F1 score
Null accuracy (correctly returning null for ambiguous transactions)
Hierarchical accuracy (matching parent categories)
Per-difficulty breakdown

Datasets:

categorization_golden_v1 - 100 samples, US merchants
categorization_golden_v1_light - 50 samples, quick testing
categorization_golden_v2 - 200 samples, US and European merchants

Merchant detection

Tests business name and URL detection from transaction descriptions. Metrics:

Name accuracy (exact match)
Fuzzy name accuracy (similarity threshold)
URL accuracy
False positive/negative rates
Average fuzzy score

Datasets:

merchant_detection_golden_v1 - 90 samples

Chat assistant

Tests function calling and response quality for the AI assistant. Metrics:

Function selection accuracy
Parameter accuracy
Response relevance
Exact match rate
Error rate

Datasets:

chat_golden_v1 - 50 samples
chat_golden_v2 - routing and analytical tool scenarios; exercises CHAT classification (no tools), aggregate queries routed to get_income_statement / get_balance_sheet, and new analytical tool selection

Naive Bayes categorizer

The naive-bayes provider evaluates the family-trained Naive Bayes categorizer — the first stage of the auto-categorization cascade that runs before any paid provider sees a transaction. Pass PROVIDER=bayes to use the Bayes runner:

PROVIDER=bayes SPLIT_ROLE=test rake 'evals:run[categorization_golden_v2,naive-bayes]'

The Bayes runner trains the classifier on the dataset’s training split and scores it on the held-out test split. A SPLIT_ROLE of test restricts the run to the test split only; omit it to run on all samples. Important caveat: Golden datasets contain roughly one transaction per distinct merchant, chosen for coverage. Real families have the opposite shape — the same handful of merchants repeating hundreds of times — which is exactly the repetition Naive Bayes learns from. Numbers from this runner are a lower bound on production accuracy, not a measurement.

Cascade reporting

The cascade report shows how a paid provider performs on the transactions Bayes declined to categorize. This measures the value each provider adds on top of Bayes — the residual the LLM actually needs to handle. Produce the runs first, all on the same test split:

# Step 1: run Bayes on the test split
PROVIDER=bayes SPLIT_ROLE=test rake 'evals:run[categorization_golden_v2,naive-bayes]'

# Step 2: run your provider(s) on the same test split
PROVIDER=jev SPLIT_ROLE=test rake 'evals:run[categorization_golden_v2,~typesafe/jev-latest]'
PROVIDER=openai SPLIT_ROLE=test rake 'evals:run[categorization_golden_v2,gpt-4.1]'

# Step 3: view the cascade report
rake evals:cascade[<bayes_run_id>,<jev_run_id>,<openai_run_id>]

Candidate analysis

The analyze command compares a candidate provider against a baseline. It outputs a side-by-side accuracy comparison, a confidence-calibration table for the candidate, a transaction-level disagreement breakdown, and a recommendation verdict.

rake evals:analyze[baseline_run_id,candidate_run_id]

Both runs must cover the same dataset. The candidate is the provider under evaluation; the baseline is what it would replace.

Creating custom datasets

Export your manually categorized transactions as a golden dataset:

# Basic usage
rake evals:export_manual_categories[family-uuid]

# With options
FAMILY_ID=uuid OUTPUT=custom.yml LIMIT=1000 rake evals:export_manual_categories

This exports transactions where:

Category was manually set by the user
Category was NOT set by AI, rules, or data enrichment

The output matches the standard dataset format and can be imported with rake evals:import_dataset[path].

JSON mode configuration

Control how the LLM outputs structured data. Configure via environment variable or Settings UI. Modes:

auto - Tries strict first, falls back to none if >50% fail (recommended)
strict - Best for thinking models (qwen-thinking, deepseek-reasoner)
none - Best for standard models (llama, mistral, gpt-oss)
json_object - Middle ground, broader compatibility

# Set via environment
LLM_JSON_MODE=none bin/rails 'evals:run[...]'

# Or configure in Settings → Self-Hosting → AI Provider

Example output

================================================================================
Evaluation Complete
================================================================================
  Status: completed
  Duration: 150.1s
  Run ID: 66c70614-72f4-49cb-8183-46103fb554f2

Metrics:
  accuracy: 76.0
  precision: 78.75
  recall: 90.0
  f1_score: 84.0
  null_accuracy: 100.0
  hierarchical_accuracy: 68.0
  samples_processed: 100
  samples_correct: 76
  avg_latency_ms: 1494
  total_cost: 0.0
  cost_per_sample: 0.0

By Difficulty:
  easy: 80.0% accuracy (28/35)
  medium: 70.59% accuracy (24/34)
  hard: 63.16% accuracy (12/19)
  edge_case: 100.0% accuracy (12/12)

Getting started

Key concepts

App features

Development

Authentication

Third party providers

Knowledge Base

Quick start

Import a dataset

Run an evaluation

Compare models

Available commands

Dataset management

Running evaluations

Viewing results

Cascade and analysis

Langfuse integration

Setup

Commands

What gets created

Evaluation types

Categorization

Merchant detection

Chat assistant

Naive Bayes categorizer

Cascade reporting

Candidate analysis

Creating custom datasets

JSON mode configuration

Example output

​Quick start

​Import a dataset

​Run an evaluation

​Compare models

​Available commands

​Dataset management

​Running evaluations

​Viewing results

​Cascade and analysis

​Langfuse integration

​Setup

​Commands

​What gets created

​Evaluation types

​Categorization

​Merchant detection

​Chat assistant

​Naive Bayes categorizer

​Cascade reporting

​Candidate analysis

​Creating custom datasets

​JSON mode configuration

​Example output

Quick start

Import a dataset

Run an evaluation

Compare models

Available commands

Dataset management

Running evaluations

Viewing results

Cascade and analysis

Langfuse integration

Setup

Commands

What gets created

Evaluation types

Categorization

Merchant detection

Chat assistant

Naive Bayes categorizer

Cascade reporting

Candidate analysis

Creating custom datasets

JSON mode configuration

Example output