Batch Testing

Batch testing lets you run a prompt against every row in a CSV file automatically, with results collected in a live-updating table. This guide covers how to set up and run a batch, how variable mapping works, how to export results, and how to use comparison mode to test the same CSV across multiple models at once.

Batch testing lets you run a prompt against every row in a CSV file automatically, with results collected in a live-updating table. This guide covers how to set up and run a batch, how variable mapping works, how to export results, and how to use comparison mode to test the same CSV across multiple models at once.

This guide assumes you're already comfortable writing and running prompts in Orion.

What batch testing is for

A single prompt run tells you how a model responds to one input. Batch testing tells you how it responds to many — without clicking Run fifty times.

Common uses:

  • Evaluating prompt quality — test a prompt against a representative sample of real inputs to see where it holds up and where it breaks.
  • Regression testing — after changing a prompt, re-run the same CSV to compare results against a previous export.
  • Bulk processing — generate content, classify text, or extract data across a large input set and export the results.

Running your first batch

Step 1: Open the Batch panel

Open a prompt in the editor. In the toolbar below the user prompt, click the CSV Batch button (the table/spreadsheet icon). The Batch panel opens in place of the response area.

Step 2: Load a CSV file

Click Load CSV and select your file. The file must have a header row — the column names are how Orion knows which variable each column maps to.

Once loaded, the column names appear in the panel. Orion automatically maps any column whose name matches a {{variable}} in your prompt.

Step 3: Run the batch

Click Run Batch. Rows begin processing immediately. Each row shows its status as it progresses:

  • Pending — waiting to start
  • Running — request in flight
  • Done — response received
  • Error — request failed (shown in red with the error message)

You can stop a run at any time with the Stop button. Rows that have already completed stay in the table.

Tip: Start with a small CSV (5–10 rows) to validate your prompt and variable mapping before running hundreds of rows.

How variable mapping works

CSV column names map directly to {{variable}} placeholders in your prompt. A column named topic fills {{topic}}, a column named customer_name fills {{customer_name}}, and so on.

A few things to know:

  • Matching is case-sensitive. A column named Topic will not fill {{topic}}.
  • Extra columns are ignored. If a column has no matching variable in your prompt, it's still included in the results table — it just doesn't affect the request.
  • Prompt-level test variables act as fallbacks. If you've set a value for a variable in the Variable Test Panel, that value is used for any row that doesn't have a matching column. This lets you mix CSV-driven variables with fixed values — for example, a shared system persona while varying the {{user_input}} per row.

Example

Prompt:

Summarize the following article in {{length}} words or fewer, focusing on {{angle}}.

Article: {{article_text}}

CSV:

article_text angle length
"Scientists discovered..." key findings 100
"The city council voted..." policy impact 75
"A local startup raised..." business implications 150

All three columns match variables in the prompt, so each row produces a fully resolved request.

Attaching files per row

If your CSV includes a column whose values are absolute file paths, Orion loads each file at run time and attaches it to that row's request. The same extraction pipeline used for manual file attachments applies: PDFs get their text extracted, images are sent as base64, Word documents and spreadsheets are parsed automatically.

To use this, add a column to your CSV with the absolute paths to the files you want to attach, then reference the column name as a variable in your prompt (e.g., {{document}}). Orion detects supported file extensions and handles loading automatically.

Example CSV:

document question
/Users/alice/reports/q1.pdf What were the main findings?
/Users/alice/reports/q2.pdf How did revenue change?

Prompt:

{{document}}

Answer the following question based on the document above: {{question}}

Reading the results table

As rows complete, the results table fills in. Columns include:

Column What it shows
Row number The row's position in the CSV (1-indexed)
Input columns The values from the CSV for that row
Response The model's response (click the row to expand the full text)
Latency How long the request took, in seconds
Tokens Input and output token counts
Status Done, Running, Pending, or Error

Click any row to expand it and read the full response. Errors show in red with the error message inline — the rest of the batch continues running regardless.

Exporting results

When a run is complete (or after stopping), click Export CSV to save the results. The exported file includes all input columns plus response, latency, tokens, and status for every row.

This makes it straightforward to:

  • Open results in Excel or Google Sheets for further analysis
  • Version-control eval results alongside your prompts
  • Diff the output of two different prompt versions run against the same CSV

Comparison mode

Instead of running a CSV against one model, comparison mode lets you run it against multiple models simultaneously — side by side in the same table.

Setting up a comparison

  1. In the Batch panel, click the Compare button (or the columns icon) to switch to comparison mode.
  2. Add the models you want to compare. Each model gets its own response column in the results table.
  3. Click Run Batch. Every row runs against every model you've added.

How concurrency works in comparison mode

Each model runs with its own independent concurrency, set per-provider in Settings → Providers. A cloud model running at concurrency 5 and a local model at concurrency 1 run truly in parallel — the local model isn't throttled by the cloud model or vice versa.

When to use comparison mode

  • Picking the best model for a specific task before committing
  • Quantifying quality vs. cost tradeoffs across providers
  • Regression testing when a model version changes — compare old vs. new side by side

Concurrency and performance

The Batch Concurrency setting (in Settings → Providers) controls how many rows Orion sends to a provider in parallel.

Provider type Recommended concurrency
OpenAI, Anthropic, OpenRouter Start at 5; increase based on your rate limits
Ollama, LM Studio, llama.cpp Set to 1 — local servers handle one request at a time
Unknown or custom endpoints Start at 1; increase cautiously

Setting concurrency too high on a local model server causes queuing, timeouts, and errors. Set it to 1 and you'll get reliable (if sequential) results.

The prompt's model parameters — temperature, max tokens, top-p, and so on — apply to every row in the batch.

What's next

  • Configuring Providers — set Batch Concurrency per provider and configure local model servers like Ollama and LM Studio.
  • Variables guide — learn how {{variable}} substitution works across prompts, including the Variable Test Panel for setting fallback values.
  • Comparison mode — run the same prompt against multiple models or providers side by side without a CSV, using the standard comparison view in the editor.
Back to Getting Started