promptfoo: unit tests for prompts and LLM evals in CI

A team edits one line of the system prompt to fix a customer-reported bug, deploys, and a week later discovers three other features have "drifted in tone" or return the wrong format. Ordinary software has unit tests to catch regressions; with LLM apps, many teams still test by… trying a few prompts by hand. promptfoo (repo promptfoo/promptfoo, MIT) is a CLI for writing tests for prompts, agents and RAG, comparing models, and running them in CI like unit tests. The README says the project was acquired by OpenAI but remains MIT open source.

Get started in 5 minutes

npx promptfoo@latest init --example getting-started
cd getting-started
export ANTHROPIC_API_KEY=...
npx promptfoo@latest eval      # run the test suite
npx promptfoo@latest view      # open the web UI to see results as a table

You can also install with npm install -g promptfoo, brew or pip. Evals run locally with caching for fast re-runs.

Config file structure

Everything lives in promptfooconfig.yaml: which prompts, run on which models, with which tests:

prompts:
  - file://prompts/triage.txt

providers:
  - anthropic:messages:claude-haiku-5-5
  - anthropic:messages:claude-sonnet-5-5

tests:
  - vars:
      ticket: "I was charged twice for the same order"
    assert:
      - type: is-json
      - type: javascript
        value: JSON.parse(output).priority === 'P1'
      - type: latency
        threshold: 5000

  - vars:
      ticket: "How do I export a report to Excel?"
    assert:
      - type: icontains
        value: how_to
      - type: llm-rubric
        value: Does not promise features that don't exist
        weight: 2

The result is a matrix: rows are test cases, columns are models. You see immediately whether the cheap model is good enough and whether the new prompt broke old cases.

Assertion types

Prefix with not- to negate (not-contains). weight sets a weight (default 1; 0 just records a metric); a test-level threshold lets a case pass when the weighted average score is high enough.

Putting it in CI

1. Commit promptfooconfig.yaml and the prompts folder to the repo, next to the code.
2. Add a CI job running promptfoo eval when a PR touches the prompts folder.
3. Fail the PR if the pass rate is below a threshold; attach the report link so reviewers can see per-case diffs.
4. Every production bug → one new test case. The suite grows into a regression suite.

Beyond evals, promptfoo has a red teaming mode to scan for vulnerabilities (prompt injection, data leakage…) and generate a report, very useful before exposing a chatbot publicly.

Risks and caveats

• LLMs grading LLMs isn't absolute: llm-rubric can grade wrongly. Prefer deterministic assertions (JSON, regex, code) where possible; use rubrics for the rest and spot-check by hand.
• Eval cost: 200 cases × 3 models × model grading = many calls. Use a small model as grader, enable caching, only run when prompts change.
• Variable results: LLM output isn't fixed. Run repeats and use rate thresholds instead of demanding 100%.
• Test data: cases taken from real tickets must be anonymised before committing.

How to apply it

• Dev: Start with 20 real cases for your most important prompt, mostly deterministic assertions. Run them before every prompt change.
• Tech Lead: Put evals in CI like unit tests; use the multi-model matrix to prove when you can move to a cheaper model.
• PM: Turn quality criteria into concrete rubrics ("doesn't promise features", "answers in Vietnamese"). Those are the acceptance criteria for an AI feature.