Turn Every Production Bug Into a Regression Test With AI

“Use AI to generate more tests” is the wrong goal.

More tests can mean more duplicated assertions, brittle mocks, and green checks that prove very little. A model can produce test-shaped code faster than a team can decide whether those tests protect behavior that matters.

I would set a narrower target:

Every confirmed production bug should become a verified regression test.

Give AI the bug report, logs, stack trace, environment, relevant code, and the test patterns already used by the repository. Ask it to find the smallest reproduction and prepare a candidate test. Then require one proof before that test can enter the suite:

1. The test fails against the broken revision for the expected reason.
2. The identical test passes against the fixed revision.

An engineer still decides whether the assertion represents the customer-visible failure, whether the test belongs at the right layer, and whether it will remain useful after the implementation changes.

This boundary turns AI from a test-volume machine into an organizational memory mechanism. A real failure becomes permanent coverage, and the evidence that justified the test travels with it.

Optimize for Failures Remembered, Not Tests Generated

Production bugs contain information that a generic test-generation prompt does not.

They reveal the combination of input, state, environment, timing, integration behavior, or mistaken assumption that escaped the existing controls. The bug is evidence that the current suite and review process did not cover a behavior that mattered.

Yet many teams close the immediate defect without preserving that evidence. The fix ships, the ticket closes, and the exact path that failed disappears into a collection of logs, incident notes, and pull-request comments. The team solved the symptom but did not make the system remember.

AI is useful here because the work is evidence-heavy and repetitive:

• Collect the relevant parts of a ticket, logs, stack trace, and change history.
• Compare the failure with nearby tests and repository conventions.
• Propose the smallest reproduction that explains the observed behavior.
• Draft candidate tests at the unit, integration, contract, or end-to-end layer.
• Run the same test across two known revisions and prepare the result for review.
• Link the accepted test back to the original defect.

The outcome is not “AI wrote a test.” The outcome is a confirmed production failure that the delivery system can now detect before release.

That difference should shape the workflow, permissions, and metrics from the beginning.

Start With the Current Bug-Fix Workflow

Before adding AI, map how a production defect moves through the organization today:

customer report or alert
  → triage
  → reproduction
  → code fix
  → review
  → deployment
  → ticket closure

Now find the places where evidence is lost.

• Does the engineering issue contain the exact observed and expected behavior?
• Can an engineer reproduce the failure without searching three other systems?
• Is the affected version, configuration, account state, or feature flag recorded?
• Does the fix include a test that would have caught the defect before release?
• Was that test proved against the broken revision?
• Can the team later connect the test to the production failure it protects?

Do not begin by asking a model to inspect every resolved ticket. Establish the baseline first.

For the previous 30 or 50 confirmed production bugs, record:

• Percentage with any new automated test
• Percentage with a regression test linked to the defect
• Percentage proved against the broken and fixed revisions
• Median time from confirmed bug to reliable reproduction
• Repeat incidents caused by the same behavior
• Flaky tests introduced during bug fixes
• Defects that could not be reproduced or lacked enough evidence

This baseline exposes the real bottleneck. If the team cannot reproduce defects because logs lack identifiers, test generation is not the first problem. If tests exist but nobody verifies them against the pre-fix code, the missing control is the two-revision proof. If every bug waits for one specialist who understands a fragile service, the workflow must also improve ownership and documentation.

AI should remove friction from the existing learning loop. It cannot compensate for evidence the company never captured.

The Boundary: What AI Does and What Engineers Own

The safest useful boundary is straightforward.

AI may prepare

• Extract structured evidence from approved tickets, logs, and incident records.
• Identify missing fields and ask for clarification.
• Locate relevant code paths, recent changes, and nearby tests.
• Suggest a minimal reproduction.
• Draft one or more candidate regression tests.
• Run approved test commands in an isolated environment.
• Compare results across the broken and fixed revisions.
• Prepare a draft pull request with evidence and limitations.

An engineer must decide

• Whether the reported behavior is a confirmed defect.
• What the expected product behavior should be.
• Whether sensitive production data may be used and how it must be redacted.
• Which code revision accurately represents the broken state.
• Whether the test asserts the external behavior rather than mirroring the fix.
• Whether the test belongs at the correct layer and is stable enough to maintain.
• Whether the implementation fix is safe.
• Whether the pull request is approved and merged.

AI should not change production, invent missing customer context, redefine expected behavior, merge its own work, or quietly replace a failed reproduction with a different assertion.

The model is preparing evidence and implementation. The accountable engineer owns the claim: this test detects the production defect we observed.

Build a Bug-to-Test Evidence Packet

The prompt is not the important artifact. The evidence packet is.

I would require seven sections before the model writes a test:

1. Observed failure — what the customer or system actually experienced
2. Expected behavior — the rule or outcome that was violated
3. Environment — version, configuration, account state, dependencies, and feature flags
4. Evidence — logs, stack trace, timestamps, correlation identifiers, screenshots, or traces
5. Reproduction — confirmed steps and explicitly unknown conditions
6. Code context — suspected path, relevant recent changes, owners, and nearby test patterns
7. Validation rule — how to prove the test fails before the fix and passes after it

A compact template can look like this:

bug:
  id: BUG-1842
  observed: >
    Submitting the same payment callback twice creates two ledger entries.
  expected: >
    A repeated callback with the same provider event ID is idempotent.

environment:
  affected_version: release-2026.07.24
  configuration:
    payment_provider: example-provider
    callback_retry: enabled
  feature_flags:
    new_ledger_writer: true

evidence:
  timestamps:
    - 2026-07-25T09:14:03Z
    - 2026-07-25T09:14:05Z
  correlation_ids:
    - redacted-correlation-id
  logs:
    - approved/redacted/log-excerpt.txt
  stack_trace: approved/redacted/stack-trace.txt

reproduction:
  confirmed_steps:
    - Create one valid provider callback payload.
    - Deliver the same payload twice with the same event ID.
    - Query ledger entries for that event ID.
  expected_failure:
    - Two entries exist instead of one.
  unknowns:
    - Whether the failure depends on concurrent delivery.

code_context:
  suspected_paths:
    - src/payments/callbacks/**
    - src/ledger/writer/**
  nearby_tests:
    - test/payments/callback-idempotency.test.ts
  broken_revision: 4e912ab
  fixed_revision: pending

validation:
  required_failure: >
    The candidate test fails on the broken revision because two ledger
    entries are created for one provider event ID.
  required_pass: >
    The identical test passes on the fixed revision without weakening
    any existing assertion.

This is not a demand for perfect information. Unknowns are allowed. Hidden unknowns are not.

If the packet is incomplete, the correct model behavior is to return questions such as:

• Which deployment first showed the failure?
• Is the repeated event ID visible in both requests?
• Was the second callback concurrent or sequential?
• Which account configuration was active?
• What outcome should be treated as correct when the first request is still processing?

The workflow should reward the model for exposing missing evidence and make fabricated context an immediate rejection.

The Workflow I Would Ship

1. Confirm the Defect and Preserve the Broken Revision

Not every support complaint, alert, or failed job is a product defect. Triage still comes first.

An accountable engineer or product owner confirms the expected behavior and identifies the last broken revision that can reproduce the problem. Preserve that revision in a branch, tag, worktree, container image, or reproducible build artifact.

Without a trustworthy broken state, the workflow cannot prove that a test detects the original defect. It can only prove that the test passes after the fix.

2. Assemble and Redact the Evidence

Bring the approved evidence into one bounded workspace. Prefer links and narrow excerpts over entire log archives or production database copies.

Apply the same security and privacy rules used for normal engineering work:

• Remove secrets, access tokens, credentials, and unnecessary personal data.
• Restrict the model to approved systems and repositories.
• Record which evidence sources were provided.
• Keep production access read-only where access is required at all.
• Use synthetic or minimized data for the actual regression fixture whenever possible.

The model should never need unrestricted production access to write a test.

3. Ask for the Smallest Reproduction First

Do not ask for a complete test suite. Ask the model to explain the suspected failure path and propose the smallest reproduction.

Require it to return:

{
  "observed_behavior": "What failed in production",
  "expected_behavior": "What should have happened",
  "missing_evidence": [],
  "suspected_path": ["Files or components involved"],
  "minimal_reproduction": ["Ordered steps"],
  "test_layer": "unit | integration | contract | end-to-end",
  "why_this_layer": "Why this is the narrowest reliable proof",
  "expected_failure": "The exact assertion or outcome that must fail"
}

An engineer reviews the reproduction before code generation during the pilot. If the model chooses a unit test for a transaction-boundary failure or an end-to-end test for a pure calculation error, correct the test layer first.

4. Generate a Candidate Test Without Reading the Fix

Where practical, generate the first candidate from the bug evidence and broken code, not from the final patch.

This reduces one common failure mode: the model reads the implementation, mirrors its internal choices in the assertion, and produces a test that confirms the patch rather than the required behavior.

The test should use the repository's existing fixtures, helpers, naming, and cleanup patterns. New test infrastructure requires a separate decision; it should not arrive unnoticed inside a bug fix.

5. Run the Test Against the Broken Revision

This is the mandatory gate.

The test must fail for the expected reason. “The command exited non-zero” is not enough.

Reject the candidate when:

• It fails to compile.
• Setup or fixture loading fails before the assertion.
• A network dependency is unavailable.
• A timeout occurs for an unrelated reason.
• A different assertion fails.
• The test passes.

The result should record the command, revision, exit status, failing assertion, and relevant output. That evidence belongs in the pull request.

If the test passes against the broken revision, it may still cover something useful. It is not a regression test for this bug.

6. Run the Identical Test Against the Fixed Revision

Apply the fix and run the same test without weakening or rewriting the assertion.

The candidate is valid only when it passes on the fixed revision and the existing relevant suite remains green. If the test must be changed after seeing the fix, run the updated version against both revisions again.

The useful proof is the pair:

broken revision + candidate test → expected failure
fixed revision  + identical test → pass

This is stronger than a screenshot of a green build because it demonstrates that the new test is sensitive to the behavior that changed.

7. Require Human Review Before Permanent Coverage

The engineer reviews more than syntax.

• Does the assertion describe the product or system contract?
• Is the fixture minimal and understandable?
• Is the test at the lowest reliable layer?
• Does it depend on unstable timing, external state, or excessive mocks?
• Does it duplicate existing coverage?
• Will it fail for a meaningful future regression?
• Can a future engineer understand which production failure it protects?

The final pull request should link the defect, evidence packet, broken-revision result, fixed-revision result, and any remaining limitation.

Once merged, the regression test becomes part of the system's memory—not a generated artifact that nobody owns.

Integrations and Permissions

A first implementation does not need an autonomous platform. It needs a narrow path through systems the team already uses.

System	Minimum access	Purpose
Issue tracker	Read approved issue; write a draft comment	Read expected behavior and attach the test result
Logs and traces	Read narrow, approved excerpts	Reconstruct the observed failure
Source repository	Read repository; write isolated branch	Find patterns and prepare the candidate test
CI	Run allowlisted checks; read results	Prove the test across the two revisions
Pull requests	Create draft; no merge permission	Present evidence for engineer review

Start with manual evidence collection if these integrations would delay the pilot. A person can prepare the packet and start an isolated coding-agent task. Automate only the handoffs that repeatedly consume time and have stable permission boundaries.

The AI identity should not have:

• Production write access
• Permission to change alerting, CI policy, or deployment configuration
• Generic secret access
• Permission to approve or merge
• Access to unrelated customer records
• The ability to suppress, quarantine, or delete failing tests

If a workflow needs those permissions to generate a regression test, its design is too broad.

Evaluate the Workflow Before Live Automation

Use historical confirmed bugs as the evaluation set. Select examples across different failure types:

• Incorrect business rule
• State or idempotency bug
• Integration contract mismatch
• Configuration or feature-flag interaction
• Concurrency or ordering problem
• Data migration defect
• Timeout or error-handling failure
• Bug with incomplete evidence

For each bug, record whether the workflow:

1. Identified missing evidence instead of inventing it.
2. Chose a sensible test layer.
3. Produced a minimal, maintainable candidate.
4. Failed on the broken revision for the expected reason.
5. Passed unchanged on the fixed revision.
6. Avoided sensitive data and unauthorized actions.
7. Required less engineering effort than writing the test manually.

I would make these non-negotiable launch gates:

• Every accepted test has the two-revision proof.
• No test enters the suite without engineer approval.
• No sensitive production data appears in prompts, fixtures, commits, or logs.
• The model exposes missing evidence rather than completing it with assumptions.
• A failed or inconclusive run is reported as failed or inconclusive—not summarized as success.

Useful-candidate rate, time saved, and test-layer accuracy can improve during the pilot. Evidence integrity and permission compliance are gates, not averages.

Roll Out in Four Stages

Stage 0: Measure the Existing Loop

Review the previous 30 or 50 confirmed production bugs. Establish the baseline and identify which services have reproducible builds and usable evidence.

Stage 1: Historical Shadow Mode

Run the workflow on resolved bugs with known fixes. Engineers compare the generated candidates with the tests that were actually merged. Nothing enters the repository.

This reveals whether the evidence packet, repository context, and two-revision runner work before a live deadline creates pressure.

Stage 2: One-Team Live Pilot

Choose one service with:

• Clear ownership
• A reliable test suite
• Reproducible builds
• Frequent but non-catastrophic defects
• Engineers willing to review the candidates

AI prepares draft tests and evidence. Engineers still control every repository change.

Stage 3: Draft-PR Automation

After the workflow proves reliable, allow it to create a draft pull request automatically when a confirmed defect has a complete evidence packet and a preserved broken revision.

Do not automatically merge. Do not broaden production access. Do not make every bug wait for AI if an engineer can resolve it faster.

Pause or roll back when:

• Tests repeatedly fail for unrelated reasons.
• Candidate tests pass against broken revisions.
• Flaky tests increase.
• Reviewers spend more time untangling generated tests than writing them.
• Sensitive data crosses the approved boundary.
• The workflow starts optimizing for closure rate instead of defect learning.

Adjust the Operating Model to Company Size

Small company

Keep the workflow simple. Add the evidence template to the bug issue, use a coding agent inside the repository, and require the engineer fixing the defect to attach the two-revision result.

Do not build a platform. A checklist, isolated branch, and CI job can be enough.

Medium-sized company

Create a shared evidence schema, reusable CI job, approved model access, and a small library of repository-specific test patterns. Let individual teams own the workflow outcome and review quality.

Track bugs without verified regression coverage across teams, but do not turn the metric into a quota that rewards low-value tests.

Large company

Centralize data controls, approved tools, audit logs, evaluation standards, and the two-revision execution service. Federate workflow ownership to the teams that understand each product and test suite.

High-risk domains may require stricter data handling, additional reviewers, or a fully internal execution environment. The central platform should make the safe path easier without pretending it can judge every product contract.

Measure After 30 and 90 Days

After 30 days, I would review:

• Percentage of confirmed production bugs with complete evidence packets
• Percentage with a candidate regression test
• Percentage with verified fail-before/pass-after proof
• Median time from confirmation to reproduction
• Engineer acceptance, rewrite, and rejection reasons
• Flaky tests introduced by the workflow
• Sensitive-data or permission violations

After 90 days, add:

• Percentage of confirmed bugs with merged verified regression coverage
• Repeat incidents for behaviors that should have been covered
• Escaped-defect rate in participating services
• Time from confirmed bug to safe fix
• Maintenance cost of generated tests
• Review and rework time
• Team understanding of the failures and tests

Do not use generated test count, lines of test code, prompts, or model acceptance rate as success metrics. Those measure activity.

The business value is a shorter learning loop: production exposes a failure, the team fixes it, and the delivery system permanently improves its ability to catch that failure before customers do.

Copyable Review Checklist

Use this before accepting any AI-generated regression test:

## Bug-to-regression-test review

### Defect

- [ ] The observed production failure is documented.
- [ ] Expected behavior is approved by an accountable person.
- [ ] The affected environment and version are recorded.
- [ ] Unknown conditions are explicit.

### Evidence

- [ ] Logs, traces, and identifiers are narrow and relevant.
- [ ] Secrets and unnecessary personal data are removed.
- [ ] The broken revision is preserved and reproducible.
- [ ] The test links back to the original defect.

### Test

- [ ] The assertion describes externally meaningful behavior.
- [ ] The test uses the lowest reliable layer.
- [ ] The fixture is minimal and maintainable.
- [ ] The test does not depend on unstable timing or excessive mocks.
- [ ] Existing coverage was checked for duplication.

### Proof

- [ ] The candidate fails on the broken revision.
- [ ] It fails for the expected reason.
- [ ] The identical test passes on the fixed revision.
- [ ] Relevant existing tests still pass.
- [ ] An engineer reviewed and approved the result.

The Bottom Line

AI can make test generation almost free. That does not make test volume valuable.

The better operating rule is that every confirmed production bug should leave the system harder to break in the same way. AI can collect the evidence, find the code path, draft the reproduction, prepare the test, and run the two revisions. Engineers retain authority over expected behavior, test quality, the fix, and the merge.

The mandatory proof is simple:

Broken revision: expected fail. Fixed revision: pass.

If a generated test cannot demonstrate both, it has not yet captured the lesson from production.

The same principle applies to the code fix itself: make the unit of AI-assisted work a small, evidence-backed change that a person can review. That complete workflow is covered in AI Coding Agents Should Take an Issue to a Reviewable PR.