by TestPilot (LX AI)
TestPilot — Regression-test your LLM app before every ship. Use it as decision-support: demo mode works without a live key; live runs require configuration. No fabricated metrics, and no claims for SSO/CSV/Slack unless that surface is actually shipped.
Decision-support only — confirm before you publish or act.
I built TestPilot as a lightweight tool that helps you check whether a change to your LLM‑powered feature behaves the way you expect before you push it to users. The core idea is simple: you give it a prompt, you tell it what you consider correct output, and it runs a set of checks that compare the new output against a baseline you have approved. The tool does not decide for you; it surfaces differences and potential safety flags so you can review them.
When you start a session you see three main areas: a prompt editor, a specification box where you describe the expected behavior, and a results pane that shows a checklist. The checklist items are grouped into three categories: regression checks, safety checks, and format checks. Each item can be marked as pass, fail, or needs review.
The workflow is deliberately manual because I want you to stay in the loop. Automation is useful for catching obvious regressions, but nuanced judgments about tone, style, or domain‑specific correctness still need a human eye. TestPilot gives you a repeatable way to surface those judgments quickly.
Demo mode lets you explore TestPilot without configuring any external API keys. When you open the tool in demo mode it uses a built‑in mock model that returns deterministic responses based on a small lookup table. This means you can try the full UI, create a prompt, write an expectation, and see the checklist populate.
To start demo mode:
Because the mock model does not call any external service, there is no cost, no latency, and no data leaves your browser. This makes demo mode safe for sharing screenshots in documentation or for onboarding new team members.
When you are ready to test against your actual LLM provider you need to supply an API key and, if required, any endpoint details. TestPilot does not store your key; it holds it only in memory for the duration of the session.
Configuration steps:
Live runs consume your provider’s quota and incur the usual latency. I recommend starting with a small set of prompts to verify that the connection works before scaling up to a full regression suite.
Each time you run a check TestPilot builds a checklist that you can scan top‑to‑bottom. The checklist is broken into three sections, each with its own heading.
Below is an example of the table that appears when you expand the “Regression checks” section. The table shows each individual test case, its status, and a short note.
| Check ID | Description | Status | Notes |
|---|---|---|---|
| REG‑001 | Output matches baseline translation | Pass | Exact string match |
| REG‑002 | Output length within 20 chars of baseline | Fail | New output is 27 chars longer |
| SAF‑001 | Profanity scan | Pass | No hits |
| SAF‑002 | Personal data pattern (email) | Pass | No email‑like strings |
| FMT‑001 | Output is plain text | Pass | No JSON or markup detected |
You can click any row to see the diff view that highlights the exact differences between the baseline and the new output. For safety checks the tool shows the matched pattern and the surrounding text so you can judge whether it is a false positive.
If you have a collection of prompts that represent the typical usage of your feature you can save them as a suite and run them all with one click. This is useful before a release or after you change the model version.
To create a suite:
.data/ folder of the TestPilot origin.To run the suite:
Because the suite data stays local you retain full control over what leaves your machine. If you want to share the suite with a teammate you can export the JSON and they can import it into their own TestPilot instance.
TestPilot is designed to be called from a script so you can add it as a gate in your continuous‑integration workflow. The tool exposes a simple CLI that you can install via npm (npm i -g testpilot-cli) or download as a single binary from the releases page.
Typical CI steps:
TESTPILOT_API_KEY). The CLI reads this variable at runtime.testpilot run-suite --path ./testsuites/faq.json --mode live. The CLI loads the suite file, executes each prompt against the live model, and prints a JSON report to stdout.The CLI does not require a graphical browser; it runs headless and uses the same core logic as the UI. This ensures that the results you see locally match what CI sees.
If you prefer to keep the API key out of the CI environment you can use a secret‑management service (e.g., GitHub Secrets, GitLab CI variables) and inject it at runtime. The CLI will never write the key to disk.
I want to be clear about the limits of TestPilot so you do not rely on it for something it cannot do.
If any of the above points are unclear, feel free to open an issue on the repository and I will clarify the intended behavior.
Regression-test your LLM app before every ship.
Operators and builders who need a fast first draft or checklist from TestPilot.
Yes in explicit Demo mode. Live AI requires a configured key.
No. Outputs are decision-support; you still review before publishing or acting.
Only if those features are implemented in this product build — do not assume them from marketing copy.
Runs may be stored locally under the product's .data/ boundary; treat demos as ephemeral.