Assertions

Assertions are checks you define on a prompt that run automatically every time you send it. Instead of eyeballing each response to decide if it met your criteria, Orion evaluates your rules and shows pass/fail results inline. This guide covers all assertion types and how to use them effectively.

Assertions are checks you define on a prompt that run automatically every time you send it. Instead of eyeballing each response to decide if it met your criteria, Orion evaluates your rules and shows pass/fail results inline. This guide covers all assertion types and how to use them effectively.

Adding assertions

The Assertions section sits below the response area. It's collapsed by default — click the header to expand it, or click the Add button on the right side of the header row to add your first assertion and expand it in one step.

Each assertion is a row with:

  • A type selector — pick what kind of check to run
  • A value field — the string, number, or schema the check uses (not present for types that don't need one)
  • A pass/fail indicator — appears after each run
  • A delete button to remove the assertion

You can add as many assertions as you want. They all run on every response.

Assertion types

Contains

Checks that the response text includes a given string.

  • Value: the string to look for
  • Aa toggle: enable for case-sensitive matching (default is case-insensitive)

Use this to verify the model mentioned something specific — a product name, a required section heading, a conclusion phrase.

Not contains

The inverse of Contains — passes if the string is absent from the response.

  • Value: the string that must not appear
  • Aa toggle: enable for case-sensitive matching

Use this to catch unwanted content: disclaimers you've told the model to omit, forbidden phrases, or boilerplate the model sometimes adds despite instructions.

Regex match

Tests the response against a regular expression.

  • Value: the regex pattern (without delimiters — enter \d{3} not /\d{3}/)
  • Aa toggle: enable for case-sensitive matching

Use this for flexible pattern matching — verifying a response starts with a capital letter, contains a phone number format, or follows a specific structural pattern that's too flexible for a plain string check.

Valid JSON

Passes if the response is parseable as valid JSON.

No value field needed. Orion automatically strips markdown code fences (```json ... ```) before parsing, so this works whether the model wraps its output in a code block or not.

Use this whenever you've asked the model to return structured data.

JSON schema

Validates the response JSON against a JSON Schema you provide.

  • Value: a JSON Schema object entered in the multi-line textarea

Orion supports a draft-07 subset including: type, properties, required, items, enum, minimum, maximum, minLength, maxLength, additionalProperties, anyOf, oneOf, allOf.

The schema check implies valid JSON — if the response isn't parseable, the assertion fails before schema validation runs.

Example schema for a response that must be a JSON object with a required name string field:

{
  "type": "object",
  "required": ["name"],
  "properties": {
    "name": { "type": "string" }
  }
}

Min length

Passes if the response is at least N characters long.

  • Value: a number (character count)

Use this to catch truncated or unexpectedly short responses — for example, when you expect at least a paragraph but the model occasionally returns a one-liner.

Max length

Passes if the response is no more than N characters long.

  • Value: a number (character count)

Use this to enforce brevity constraints when you've asked the model to be concise.

LLM judge assertions

The two LLM judge types use a second model call to evaluate the response. Instead of a deterministic rule, you describe what a good response looks like in plain language and let a model decide.

Note: LLM judge assertions require a configured provider with a working API key. They run after the main response completes — you'll see a loading indicator while the judge evaluates, then a pass/fail result. They count against your API usage.

LLM rubric

Asks an LLM to score the response on a 1–5 scale against a rubric you write.

  • Rubric: describe what a good response looks like. Be specific and concrete.
  • Pass threshold: the minimum score to count as a pass (1–5). Default is 3.

After evaluation, click the Reasoning link to see the judge's explanation of the score it assigned.

Example rubric: "The response should be concise (under 3 sentences), professional in tone, and directly answer the question without preamble or unnecessary caveats."

Vague rubrics produce inconsistent scores. If your rubric just says "a good response", the results won't be meaningful. The more specific and observable your criteria, the more reliable the scoring.

LLM match

Asks an LLM to judge whether the response semantically matches a reference answer you provide.

  • Reference: the expected or ideal answer

This is more flexible than a Contains or regex check — paraphrasing and synonyms are treated as equivalent to the reference. Use it when you have a known-good answer and want to catch regressions as you iterate on a prompt.

Reading results

After each run, pass/fail indicators appear next to each assertion:

  • A green check — passed
  • A red X — failed
  • A spinner — pending (LLM judge is still evaluating)

The Assertions header shows a summary count — for example, 3 passed 1 failed — so you can see the overall result without expanding each row.

For failed assertions, a brief message appears below the row explaining what went wrong — for example, "Response does not contain 'Summary'" or "Response length 1,420 exceeds maximum 500".

For LLM judge assertions, click Reasoning to expand the judge's full explanation.

Assertions in batch runs

Assertions run on every row in a batch run. The results table shows per-row pass/fail status alongside the response, making it easy to spot which inputs cause failures. Export the batch results as CSV to analyze pass rates across your full dataset.

See Batch Testing for full details on running prompts at scale.

Tips

  • Start with one or two deterministic assertions (Contains, Valid JSON) before adding LLM judge checks. Deterministic assertions are instant and don't use API credits.
  • Use Valid JSON and JSON schema together — Valid JSON catches parse failures fast, and JSON schema tells you exactly which fields are missing or wrong.
  • You can stack as many assertions as you need. A prompt asking for structured data might have: Valid JSON + JSON schema + Min length — all three run on every response.
  • Assertions are saved as part of the prompt's .json file. They persist across sessions and are included when you export a collection.
  • If an LLM judge assertion is inconsistent run to run, tighten the rubric. Ambiguous criteria produce ambiguous scores.

What's next

  • Batch Testing — run assertions at scale by attaching a CSV file and evaluating every row at once.
  • Variables — use {{variable}} placeholders in your prompts to create templates you can drive from a CSV or fill in per run.
Back to Getting Started