Comparing Models

Orion has two ways to run model comparisons: a head-to-head mode for testing a single prompt against multiple models side by side, and a batch comparison mode for running a full CSV across multiple models at once. This guide covers both — how to set them up, what to expect, and when to use each.

This guide assumes you're already comfortable writing and running prompts in Orion.

Head-to-head comparison

Head-to-head mode runs a single prompt against 2–6 models at the same time and shows all responses in a side-by-side grid. Use it for quick gut-checks and model selection decisions before you commit to a larger evaluation.

Setting up a comparison

Open a prompt in the editor.
Click the Compare button in the editor toolbar (next to the Run button). The "Head-to-Head Comparison" modal opens.
Each row in the modal is a model slot — a provider dropdown and, once you select a provider, a model dropdown. Two slots are shown by default.
Click Add Model to add more slots. You can compare up to 6 models per run.
To remove a model, click the X on its slot.
Click Compare to run. You need at least 2 models selected before the button activates.

What happens when you run

All models run simultaneously. The response area is replaced by a grid — one column per model — and responses stream in as they arrive.

Each column shows:

Element	What it shows
Header	Provider name and model ID
Response	The model's response, streaming in as markdown
Copy button	Copies that column's response to your clipboard
Footer	Latency (ms) and total token count, once complete

Every model receives the exact same prompt, system prompt, parameters, temperature, and attachments. If your prompt uses variables, they resolve the same way for every model.

If a model fails — bad API key, rate limit, network error — its column shows the error in red. The other columns continue running unaffected.

Re-running a comparison

Click Compare again at any time to open the modal and run a new comparison. You can keep the same models or swap them out. The previous comparison results are replaced when the new run starts.

Tip: The copy button on each column is useful for grabbing the best response to paste into a document or ticket without manually selecting text.

Batch comparison

Batch comparison combines the CSV batch runner with the comparison selector. Every row in your CSV runs against every model you've selected — the results table gains one response column per model.

If you haven't used batch testing before, read the Batch Testing guide first. Batch comparison builds directly on that workflow.

Setting up a batch comparison

Open a prompt in the editor and click the CSV Batch button in the toolbar to open the Batch panel.
Click Load CSV and select your file.
In the Batch panel toolbar, click the Compare button (the columns icon). The same "Head-to-Head Comparison" modal opens — select your models and click Compare.
Click Run Batch. Every row in the CSV runs against every model you've added.

Reading the results table

The results table works the same as a standard batch run, with one addition: there is a separate response, latency, and token column for each model. Columns are grouped by model, so you can scan down a single model's column or scan across a single row to compare model outputs for the same input.

Click any row to expand it and read the full responses side by side.

How concurrency works

Each model runs with its own independent concurrency pool, set per-provider in Settings → Providers. A cloud model at concurrency 5 and a local model at concurrency 1 run truly in parallel — neither waits on the other.

Tip: Set local models (Ollama, LM Studio, llama.cpp) to Batch Concurrency = 1 in Settings → Providers. Local servers process one request at a time, and sending more in parallel causes queuing and timeouts. Cloud models can safely run at higher concurrency. See Configuring Providers for details.

Assertions in batch comparison

If the prompt has assertions configured, they run for every model × row combination. The results table shows pass/fail per assertion, per model, per row — so you can see at a glance which models satisfy your eval criteria across the full input set.

When to use each mode

Goal	Use
"Which model sounds better for this prompt?"	Head-to-head
Picking a model before a production integration	Head-to-head
Checking a prompt change doesn't regress across providers	Head-to-head
Evaluating model quality across a representative input set	Batch comparison
Quantifying quality vs. cost tradeoffs at scale	Batch comparison
Running an eval with assertions across many inputs and models	Batch comparison
Building a labeled dataset for a specific task	Batch comparison

A few things to know

All models use the same parameters. Temperature, max tokens, top-p, and other settings apply equally to every model in the comparison. If you want to compare different parameter configurations, open the prompt in separate tabs and run them independently.
You can mix providers freely. GPT-4o, Claude 3.5 Sonnet, and a local Ollama model can all run in the same comparison.
Maximum 6 models per run. If you need to test more, export your first comparison and start a second run.
Variables resolve identically for all models. Values set in the Variable Test Panel — or mapped from a CSV column — apply the same way to every model. The comparison is apples-to-apples.

What's next

Batch Testing — learn how to set up batch runs, map CSV columns to variables, and export results before adding comparison mode.
Configuring Providers — set Batch Concurrency per provider and configure local model servers for reliable parallel runs.
Variables — use {{variable}} placeholders in your prompts to build reusable templates that work consistently across all models in a comparison.

Back to Getting Started