Every team has a test that fails maybe one run in eight. Someone re-runs CI, it passes, the PR merges, nobody investigates.

Do that enough times and you get the real cost: people stop believing the suite. A red build becomes something you re-run rather than something you read. At that point the tests have stopped doing their job, even the ones that work.

Flaky tests are rarely random. They are almost always deterministic tests running against non-deterministic conditions. Find the condition and the flakiness goes away.

The five sources of non-determinism

Nearly every flaky integration test traces back to one of these.

Shared state between runs. Your test creates a user with test@example.com. It passed the first time. On the second run the row already exists, the insert fails on a unique constraint, and the test errors somewhere unrelated to what it was checking. Anything long-lived - a shared staging database, a persistent test account with a third party - accumulates the residue of every previous run.

Test order dependence. Test A creates a record that Test B quietly relies on. Run them alphabetically and everything is fine. Run them in parallel, or let your runner shuffle them, and B fails. B was never actually testing what you thought.

Timing and race conditions. The classic sleep(2) after an async operation. It works on your laptop and fails on a loaded CI runner where the same operation takes 2.3 seconds. Retry logic and fixed waits are how most teams hide this rather than fix it.

External dependencies. A third-party sandbox that rate limits you, times out under load, or behaves differently at 9am than at 3am. Your test failed and your code was fine.

Environment drift. The test ran against a database that was migrated last week, a service running a slightly older image, or a config value someone changed for debugging and forgot to revert.

Notice what these have in common. None are problems with the assertion. They are problems with what the test was standing on when it ran.

Why retries make it worse

The instinctive fix is to retry failed tests two or three times and only report a failure if all attempts fail.

That gets the build green. It also destroys the signal.

A test that fails one run in eight because of a genuine race condition in your code is telling you something real. Retry it three times and it now fails one run in five hundred - rare enough to ignore, frequent enough that it will happen in production. You have not fixed the race. You have hidden it and made it harder to reproduce.

Retries are reasonable as a temporary measure while you find the cause. They are not a fix, and treating them as one is how suites rot.

Isolation is the actual fix

If the problem is that conditions change between runs, the fix is to make every run start from identical conditions.

That means each test run gets:

Its own database, seeded to a known state, destroyed afterwards
Its own instances of your services, not shared with any other run
Its own mocked external dependencies, so no rate limits and no shared accounts
Its own network namespace, so parallel runs cannot see each other

Get that and four of the five causes disappear structurally. There is no shared state because nothing is shared. There is no order dependence because each test starts fresh. There are no third-party flakes because there is no third party. There is no drift because the environment is defined in a file next to the code.

Only genuine race conditions in your own code survive - and those are the ones you actually want the test to catch.

What this looks like in Dokkimi

Every dokkimi run creates an isolated Docker namespace: your services, a dedicated database, a headless browser, and interceptor sidecars capturing every HTTP call between services. When the run finishes, all of it is destroyed.

name: user-signup
items:
  - $ref: ../shared/web-app.yaml
  - $ref: ../shared/auth-service.yaml
  - $ref: ../shared/postgres-db.yaml
  - $ref: ../shared/mock-auth0-jwks.yaml

tests:
  - name: New user is created with correct defaults
    steps:
      - action:
          type: ui
          url: http://web-app:3000/signup
          subSteps:
            - action: type
              selector: '#email'
              value: 'test@example.com'
            - action: click
              selector: '[data-testid="submit"]'
            - action: waitForSelector
              selector: '[data-testid="welcome"]'

      - action:
          type: dbQuery
          database: postgres-db
          query: "SELECT plan FROM users WHERE email = 'test@example.com'"
        assertions:
          - path: $.response.data[0].plan
            operator: eq
            value: 'free'

test@example.com is safe to hardcode because that database did not exist ninety seconds ago and will not exist ninety seconds from now.

Because runs are isolated, tests can execute in parallel without interfering - which is the thing a shared staging environment can never offer, and a common reason teams run tests serially and slowly.

Diagnosing the ones that remain

When something still fails, the question is always the same: what state was the system actually in?

Dokkimi logs every database query the services make during a run and captures every HTTP call between them. So instead of guessing, you can look at what the service asked the database and what came back, in order.

The MCP server exposes this directly to your coding agent - get_failures for what broke, get_traffic and get_db_logs for the raw data, diagnose for automated root-cause analysis, and diff_traffic to compare a passing run against a failing one.

That last one is usually the fastest route to an answer. Two runs of the same test, one green and one red, diffed at the traffic level. The difference is the bug.

A practical order of operations

If you are dealing with a flaky suite right now:

Stop retrying. Let the failures be visible for a week so you can see which tests are actually unreliable and how often.
Check for shared state first. It is the most common cause and the easiest to confirm - run the suspect test twice in a row against the same environment and see if the second run fails.
Look for order dependence. Run the suite in a different order. Anything that breaks was depending on something it should not have been.
Isolate. Move to per-run environments so the first two categories cannot happen at all.
What remains is real. Genuine race conditions in your own code. Those are worth the time.

Getting started

brew install dokkimi/tap/dokkimi
# or
npm install -g dokkimi

dokkimi init
dokkimi doctor
dokkimi run

Requires Node 20+ and Docker. Free locally and in CI, with a GitHub Action so the same definitions run on every pull request.

Docs at dokkimi.com/docs.