Evaluation framework

The eval system helps you benchmark different LLMs for transaction categorization, merchant detection, and chat assistant functionality.

Quick start

Import a dataset

bin/rails 'evals:import_dataset[db/eval_data/categorization_golden_v1.yml]'

Run an evaluation

bin/rails 'evals:run[categorization_golden_v1,openai,gpt-4.1]'

Compare models

MODELS=gpt-4.1,gpt-4o-mini rake evals:compare[categorization_golden_v1]

Available commands

Dataset management

# List all datasets
rake evals:list_datasets

# Import dataset from YAML
rake evals:import_dataset[path/to/file.yml]

# Export manually categorized transactions
rake evals:export_manual_categories[family-uuid]

Running evaluations

# Run evaluation
rake evals:run[dataset_name,provider,model]

# Compare multiple models
MODELS=model1,model2 rake evals:compare[dataset_name]

# Quick smoke test
rake evals:smoke_test

# CI regression test
rake evals:ci_regression[dataset,provider,model,threshold]

Viewing results

# List recent runs
rake evals:list_runs

# Show detailed report
rake evals:show_run[run_id]

# Generate comparison report
rake evals:report[run_ids]

Langfuse integration

Track experiments in Langfuse for side-by-side comparison and analysis.

Setup

export LANGFUSE_PUBLIC_KEY="pk-..."
export LANGFUSE_SECRET_KEY="sk-..."
export LANGFUSE_REGION="eu"  # Optional, defaults to eu

Commands

# Check connection
bin/rails 'evals:langfuse:check'

# Upload dataset
bin/rails 'evals:langfuse:upload_dataset[categorization_golden_v1]'

# Run experiment
bin/rails 'evals:langfuse:run_experiment[categorization_golden_v1,gpt-4.1]'

# List datasets in Langfuse
bin/rails 'evals:langfuse:list_datasets'

What gets created

When you run a Langfuse experiment, the system creates:

Dataset - Named eval_<your_dataset_name> with all samples
Traces - One per sample showing input/output
Scores - Accuracy scores (0.0 or 1.0) for each trace
Dataset Runs - Links traces to dataset items for comparison

In the Langfuse UI you can:

Compare runs side-by-side
Filter by score, model, or metadata
Track accuracy over time
Analyze per-sample results

Evaluation types

Categorization

Tests transaction categorization accuracy across difficulty levels. Metrics:

Accuracy
Precision, recall, F1 score
Null accuracy (correctly returning null for ambiguous transactions)
Hierarchical accuracy (matching parent categories)
Per-difficulty breakdown

Datasets:

categorization_golden_v1 - 100 samples, US merchants
categorization_golden_v1_light - 50 samples, quick testing
categorization_golden_v2 - 200 samples, US and European merchants

Merchant detection

Tests business name and URL detection from transaction descriptions. Metrics:

Name accuracy (exact match)
Fuzzy name accuracy (similarity threshold)
URL accuracy
False positive/negative rates
Average fuzzy score

Datasets:

merchant_detection_golden_v1 - 90 samples

Chat assistant

Tests function calling and response quality for the AI assistant. Metrics:

Function selection accuracy
Parameter accuracy
Response relevance
Exact match rate
Error rate

Datasets:

chat_golden_v1 - 50 samples

Creating custom datasets

Export your manually categorized transactions as a golden dataset:

# Basic usage
rake evals:export_manual_categories[family-uuid]

# With options
FAMILY_ID=uuid OUTPUT=custom.yml LIMIT=1000 rake evals:export_manual_categories

This exports transactions where:

Category was manually set by the user
Category was NOT set by AI, rules, or data enrichment

The output matches the standard dataset format and can be imported with rake evals:import_dataset[path].

JSON mode configuration

Control how the LLM outputs structured data. Configure via environment variable or Settings UI. Modes:

auto - Tries strict first, falls back to none if >50% fail (recommended)
strict - Best for thinking models (qwen-thinking, deepseek-reasoner)
none - Best for standard models (llama, mistral, gpt-oss)
json_object - Middle ground, broader compatibility

# Set via environment
LLM_JSON_MODE=none bin/rails 'evals:run[...]'

# Or configure in Settings → Self-Hosting → AI Provider

Example output

================================================================================
Evaluation Complete
================================================================================
  Status: completed
  Duration: 150.1s
  Run ID: 66c70614-72f4-49cb-8183-46103fb554f2

Metrics:
  accuracy: 76.0
  precision: 78.75
  recall: 90.0
  f1_score: 84.0
  null_accuracy: 100.0
  hierarchical_accuracy: 68.0
  samples_processed: 100
  samples_correct: 76
  avg_latency_ms: 1494
  total_cost: 0.0
  cost_per_sample: 0.0

By Difficulty:
  easy: 80.0% accuracy (28/35)
  medium: 70.59% accuracy (24/34)
  hard: 63.16% accuracy (12/19)
  edge_case: 100.0% accuracy (12/12)

Getting started

Key concepts

App features

Development

Authentication

Third party providers

Knowledge Base

Quick start

Import a dataset

Run an evaluation

Compare models

Available commands

Dataset management

Running evaluations

Viewing results

Langfuse integration

Setup

Commands

What gets created

Evaluation types

Categorization

Merchant detection

Chat assistant

Creating custom datasets

JSON mode configuration

Example output

​Quick start

​Import a dataset

​Run an evaluation

​Compare models

​Available commands

​Dataset management

​Running evaluations

​Viewing results

​Langfuse integration

​Setup

​Commands

​What gets created

​Evaluation types

​Categorization

​Merchant detection

​Chat assistant

​Creating custom datasets

​JSON mode configuration

​Example output

Quick start

Import a dataset

Run an evaluation

Compare models

Available commands

Dataset management

Running evaluations

Viewing results

Langfuse integration

Setup

Commands

What gets created

Evaluation types

Categorization

Merchant detection

Chat assistant

Creating custom datasets

JSON mode configuration

Example output