by TestPilot (LX AI)
Open TestPilot, provide the required inputs, run Demo to preview the workflow, then switch off Demo when a live key is configured.
Regression-test your LLM app before every ship.
Decision-support only — confirm before you publish or act.
I start by making a dedicated folder for the LLM feature I want to test. Inside the folder I add a file called prompt.txt that holds the exact prompt I plan to send to the model. I also add a file called expected.txt that describes the desired output in plain language or as a regex pattern. Keeping these two files side‑by‑side makes it easy to version‑control changes.
Next I open TestPilot from the command line or double‑click the desktop shortcut. The welcome screen shows three fields: Prompt, Expected, and Mode. I copy the contents of prompt.txt into the Prompt box and the contents of expected.txt into the Expected box. The Mode selector defaults to Demo, which is what I want for the first pass.
Before I run anything I click the Save button to store the current inputs as a session file (session.json) in the project folder. This lets me recover the exact setup later without re‑typing anything. I also enable the Auto‑save option so any tweak I make is written back to the session file automatically.
I treat the prompt as a contract. I include any system messages, role instructions, and few‑shot examples that the model will see at runtime. If the prompt uses variables I replace them with concrete values for the test case. For example, if the prompt contains {user_query} I put a real question like “What is the capital of Japan?” in the test file.
The Expected field accepts three formats: plain text, JSON schema, or a regular expression. For factual answers I use plain text and ask for an exact match. For structured output I paste a JSON schema that describes the required keys and value types. For free‑form text I write a regex that captures the allowed patterns, such as ^\\d{4}-\\d{2}-\\d{2}$ for a date stamp.
I add a short comment block at the bottom of expected.txt to remind myself of edge cases I want to check later: empty input, very long input, prompts that ask for disallowed content, and prompts that trigger the model’s safety filters. These notes do not affect the automated check but guide my manual review.
With the inputs loaded I press the Run Demo button. TestPilot calls a local stub that mimics the model’s response pattern without hitting an external API. The stub returns a canned answer that matches the Expected format if the prompt is simple, or a deliberately varied answer if the prompt includes ambiguity. The demo runs in under two seconds.
After the run TestPilot shows an eval checklist with four items: Prompt match, Expected match, Safety flag, and Version hash. Each item gets a green check if the stub output satisfies the condition, a red cross if it fails, and a yellow warning if the result is borderline. I look at the Safety flag first because it tells me whether the stub produced any text that matches patterns from the OWASP Top 10 for LLM applications (https://owasp.org/www-project-top-10-for-large-language-model-applications).
If any checklist item is red I edit the Prompt or Expected file, save the session, and run Demo again. I repeat this loop until all items are green or until I have a clear reason to accept a yellow warning (for example, the model may legitimately vary its wording while still meeting the functional goal). I keep a log of each iteration in a file called demo_log.md that records the timestamp, the changes made, and the checklist outcome.
When the demo passes I click Export Suite. TestPilot writes a folder named regression_suite containing:
test_cases.json – each test case with prompt, expected, and moderun.sh – a bash script that invokes the TestPilot CLI with the saved sessionREADME.md – a short description of how to run the suite locally or in CII add this folder to my version control so every teammate can run the same checks.
Before I switch off Demo I obtain a live API key from the model provider. I open the Settings pane in TestPilot, paste the key into the API Key field, and press Save. The UI immediately changes the Mode selector to Live and disables the Demo toggle.
I load the same session I used for demo and press Run Live. TestPilot now sends the prompt to the real model endpoint, waits for the response, and evaluates it against the Expected field. The call typically takes between one and three seconds depending on the model’s load and network latency.
The live checklist mirrors the demo checklist but adds two extra columns: Latency (ms) and Token usage. I verify that the latency stays within the budget I set for the feature and that the token count does not exceed the plan’s limits. If the Safety flag turns red I consult the NIST AI Risk Management Framework (https://www.nist.gov/itl/ai-risk-management-framework) to decide whether the output violates any of the recommended safety practices.
When the live checklist shows all green items I tag the current commit as a release candidate and push the regression suite to the CI pipeline. The pipeline runs the suite on every pull request, ensuring that any change to the prompt, model version, or dependencies is caught before it reaches production.
In my CI configuration file I add a step that clones the repository, installs the TestPilot CLI (if not already present), and runs ./regression_suite/run.sh. The step fails if any test case returns a non‑zero exit code, which happens when the checklist contains a red item.
To keep the pipeline fast I enable response caching. The CLI stores each live response in a local directory keyed by a hash of the prompt and model version. On subsequent runs the CLI returns the cached answer instead of hitting the API again, unless I explicitly bypass the cache with the --fresh flag.
After the step finishes I publish the checklist as a JUnit XML file so the CI UI can display a clear pass/fail matrix. I also upload the demo_log.md as an artifact for auditors who want to see the evolution of the prompt.
If a test case intermittently fails due to non‑deterministic model output I adjust the Expected field to use a regex that captures the acceptable variance. I document the reason for the change in the test case’s description field so future reviewers understand why the expectation is not a exact string match.
TestPilot does not guarantee that your LLM‑powered feature will be free of bugs. The outputs are decision‑support aids; you must still review the generated text before you ship or act on it. It does not provide 100 % coverage of all possible inputs, nor does it replace a full‑scale red‑team exercise. The tool does not include built‑in SSO, Slack notifications, or bulk CSV import unless those features are explicitly shipped in the current build. Do not assume any of those capabilities from marketing copy or demo videos.
I keep a written checklist next to every release that touches this topic. The list is boring on purpose: name the owner, name the evidence artifact, and name the rollback. If a step cannot produce a file, a ticket, or a log line, it is not a step — it is a wish.
| Signal | What to record | Stop-ship? |
|---|---|---|
| Unexpected write or send | Tool name, args, identity | Yes until gated |
| Missing confirmation on destructive action | Call path and bypass | Yes |
| Untrusted text treated as instruction | Source surface and resulting action | Yes for agent reach |
| Scanner clean, judgment open | Deferred items + owner | Only if core path safe |
testpilot-ai helps with the measurable first pass. Architecture choices, legal sign-off, and customer-facing claims still need a human owner. Treat every checklist row as decision support, not a certificate.
When a prospect asks whether the product "covers everything," I answer with the boundary: automated checks catch known patterns; novel chains and policy judgment still need review. That honesty keeps support load down and keeps the blog aligned with what the product actually does.
If you need a second opinion, pair this write-up with the references below and with your own staging reproduction. Do not skip the reproduction step just because a model wrote a confident summary.
Regression-test your LLM app before every ship.
Operators and builders who need a fast first draft or checklist from TestPilot.
Yes in explicit Demo mode. Live AI requires a configured key.
No. Outputs are decision-support; you still review before publishing or acting.
Only if those features are implemented in this product build — do not assume them from marketing copy.
Runs may be stored locally under the product's .data/ boundary; treat demos as ephemeral.