cmux-test-bisect

cmux test bisect

PR CI runs selected tests, so a package suite can stay red on main for weeks with nobody noticing. This skill finds the commit behind each failure on CI, never on the Mac (no full cmux builds on laptops).

App-host (cmuxTests) failures on main have their own automatic bisect (#14510). Use this skill for the SwiftPM package suites under Packages/iOS and Packages/Shared that test-ios.yml can run (CMUXMobileCore, CmuxSyncStore, CmuxMobilePairedMac, CmuxMobileChanges, CmuxMobileShell, CmuxMobileShellModel).

Before you start

1. Get today's failing set from the red run:

gh run view <run> --repo manaflow-ai/cmux --log-failed, then grep ✘ Test .* failed after. Run the full suite twice if you have not already. A test that appears in only one run is a flake candidate, not a regression.

1. Search for an existing fix: gh search prs --repo manaflow-ai/cmux --state open '<test name>'.
2. Work in your own worktree from upstream/main. If git log stops at a

recent commit, the checkout is shallow: git fetch --shallow-since=<date> upstream main.

Run the bisect

T=scripts/ci/package_bisect.py
python3 $T --package CmuxMobileShell start --points 6 <old-sha>..upstream/main
python3 $T --package CmuxMobileShell status --wait     # up to 45 min
python3 $T --package CmuxMobileShell next --dispatch   # split each break window
python3 $T --package CmuxMobileShell cleanup           # delete probe branches

• Each probe pushes bisect/<name>/<sha10> (the name defaults to the package): the old commit with today's iOS

CI files laid over it and the package-lint gate dropped. Old commits carry old CI scripts, which is why a plain -f ref=<sha> dispatch fails early.

• Probes leave the runner on auto (idle owned mini first, Blacksmith

overflow). Do not pin a runner.

• adopt <sha> <run-id> counts a test-ios.yml run with

swift_package=<pkg> that already exists (a dispatch on main, or a probe run whose id start could not read) without dispatching again. A ci.yml run has no package job to read.

• --filter <regex> narrows the suite once you are chasing a few tests;

--paths changes which commits count as midpoint candidates (default: the iOS and Shared packages).

• --patch <sha> (repeatable) applies a fix commit to every probe. Use it

when older commits hang or fail to build for a reason you already know: bisect the question you have, not the known break.

• --bisect <name> keeps a second experiment (for example the same commits

with a --patch) beside the first. --package and --bisect go before the subcommand, and every later command for that bisect needs them too.

• State is shared by every worktree of the checkout, in

<git-common-dir>/package-bisect/<name>.json. Finished job logs are cached beside it per run attempt, so status --refetch costs one run lookup per probe and downloads only logs it has not seen.

Read the matrix

status prints one row per test and one column per probe, oldest first: X failed, . passed, - never ran at that probe, ? pending, E no test ran at all (compile or runner failure; check the log, rerun the run, then status --refetch; a rerun is a new attempt, so its log is read fresh).

Check the probe list first. A probe marked INCOMPLETE never printed its "Test run with" summary: the hung-test watchdog killed it or the job timed out, so every test after that point shows -. A - is never a pass: a suite that hangs early makes a whole cluster look like it "broke" at the commit that fixed the hang. Probe those commits again in a second bisect with the hang fix applied: python3 $T --package <pkg> --bisect <pkg>-patched start --patch <hang fix> <shas>.

Tests that break in the same window usually share one cause. Fix the cluster together.

Stale test or regression

Read the culprit PR, then decide per test:

• Stale test: the PR changed behavior on purpose and says so (PR body,

code comments, or other tests it updated in the same direction). Fix the test to express its original intent under the new behavior. Do not delete the assertion. If the old fixture now models the other case, keep a test for each.

• Regression: the test's intent still holds and nothing in the PR argues

it away. Fix the code, and keep the test as it was.

• When unsure, prefer the reading that keeps the test's name true, and say

which reading you chose in the PR.

Land it

• One small PR per culprit or cluster, pushed to the org remote

(git push mf <branch>, gh pr create --head <branch>).

• Credit the PR that changed the behavior (link it, and mention its author).
• Prove it with the focused suites: `gh workflow run test-ios.yml --repo

manaflow-ai/cmux --ref <branch> -f swiftpackage=<pkg> -f testfilter='<SuiteA|SuiteB>'`. Finish with one full-suite run on the last PR.

• Run cleanup so no bisect/ branches stay on the org remote.