A one hundred event communication trace went into a reducer. One event came out. The oracle accepted it.

The reducer is commcanary reduce, a ddmin-style minimizer. Delta debugging is the old, honorable idea: delete chunks of the input, keep any deletion that preserves a chosen property, repeat until nothing more can be removed. The property this oracle preserves is the decision. It compiles the surviving events, replays them across a set of backend configurations, and accepts the deletion if every pairwise ranking between configurations matches the ranking the full trace produced. Which configuration is faster than which, for every pair, under a declared tie tolerance.

The input was the synthetic adversarial trace that ships in the CommCanary repository. That trace is constructed so that queue-reset gaps and a few high-overlap tail windows invert a configuration ranking, the same shape of inversion an earlier reported campaign showed between an isolated benchmark and a decode workload. One hundred events, arranged so context matters. The reducer deleted ninety nine of them in six oracle calls, and all forty of the ranking decisions it checks survived.

This is a synthetic result, read from the repository’s own example scaffold, and I want to be precise about what it shows. It does not show that rankings are meaningless. It shows that a ranking is a projection. The oracle’s five backend configurations produce ten unordered pairs, and it checks each pair against four latency metrics, median, p95, p99, and mean. Forty bits of agreement, and a single well-placed event can carry all of them. If the only thing a minimizer must preserve is the decision, the minimizer will find the smallest artifact that happens to vote the same way, and that artifact can have almost nothing in common with the workload it came from.

The reducer is in the repository on purpose. It is the baseline the rest of the tool is built to refuse.

CommCanary is that rest. It takes a communication trace from a distributed LLM workload and distills it into a small replayable artifact, a canary in the mining sense, a cheap thing you send ahead so it fails before the expensive thing does. The artifact carries no model weights and no prompts, replays in minutes, and is meant to sit in front of a rollout: replay it under the candidate configuration, compare against the baseline report, and let the comparison exit nonzero when a regression threshold trips. The entire value of that artifact rests on one claim, that shrinking the trace did not change what it would have told you. So most of the tool is machinery for proving that claim, and for refusing to emit a canary when the proof fails.

neither benchmark nor workload

The reason the middle object needs to exist is that the two endpoints are both wrong in expensive ways.

At one end sit isolated collective microbenchmarks. A collective is a communication operation that every participant in a group must join, an all_reduce that sums gradients across GPUs for example, and tools like nccl-tests measure each collective by itself, in a loop, with nothing else running. That measurement is fast and it is also famously misleading. There is a long-standing NCCL issue, NVIDIA/nccl#513, describing the canonical shape: the isolated numbers look healthy while the real workload ships a regression on the order of twenty percent. The reported campaign I wrote about in idle gaps are not compute had the same structure, a configuration that placed sixth of eight in isolation and dead last in the workload.

The mechanism is erasure. An isolated benchmark deletes operation order, deletes the skew with which ranks arrive at a collective, deletes the compute that overlaps communication, deletes queueing, and deletes the rare windows where several of those pile up at once. Those are the contextual variables, and the regressions that matter tend to live in them.

At the other end sits the full reference run, which erases nothing and costs everything. It needs the model code, the weights, the data, and a cluster, which means it cannot run per candidate configuration in anything resembling CI, and it usually cannot be handed to another team at all because the weights and prompts cannot leave the building.

A canary occupies the space between. It is distilled from the actual workload’s trace, so order, skew, overlap, gaps, and tail windows are still in it. It carries none of the model, so it can move where the workload cannot. And it replays in minutes, so it can run on every configuration change instead of once a quarter. The one-event artifact from the opening also fits this description, which is exactly the problem. Being small and portable is easy. The hard part is the third property, that the small thing still answers like the big thing.

a decision is too easy to preserve

The natural gate for a regression canary is decision preservation, the same verdicts, the same rankings, the same tail behavior. The opening result says that gate is too weak on its own, because decisions are compressible far past the point where the workload survives. So CommCanary’s verifier does gate on decisions, and then keeps going.

verify-behavior replays a lossless normalized form of the source trace and the candidate canary across multiple backend configurations and reports separate statuses rather than one blended verdict. One status records the compiler-attested timing mode, lossless or bounded approximate. One records whether the cryptographic commitments binding the canary to its source trace recompute. One records whether the canary covers the whole normalized source or only a prefix, because a canary built from a subset is labelled as partial and is not allowed to carry a strong claim no matter how well it replays. One records behavioral fidelity, and one records whether the pairwise configuration rankings are preserved.

Behavioral fidelity is where the one-event artifact dies. The verifier compares the latency distribution at the median, the tail quantiles, the maximum and the mean, and alongside those the queue-wait distributions, the communication hidden by modeled overlap, the per-phase and per-operation breakdowns, and tail-event recall, whether the events that made the source trace’s tail expensive are still present and still expensive in the canary. All of it within declared tolerances. A single event preserves a ranking and none of this.

The gate is fail-closed, and that direction matters more than the checks themselves. Compilation in the strict mode refuses to emit an artifact that fails verification, rather than emitting it with a warning attached. When a trace has rank-local uncertainty the tool cannot resolve, the canary still replays, but its claim is downgraded to behaviorally unverified instead of quietly rounding up to verified. The failure mode of every proxy metric is silent divergence from the thing it stands for. A proxy that cannot prove agreement should say so in its own metadata, and this one does.

approximation with a receipt

Gating is only half the design. The other half is that the compression itself is auditable.

The compiler prefers exact representations wherever the trace allows them, run-length encodings for repetition and periodic encodings for regular structure, which cost nothing in fidelity. Where timing is irregular it falls back to ordered bounded intervals, and each of those carries an explicit per-field maximum error bound in the artifact. Compilation can be given an error budget, so many microseconds of skew error, so many of overlap error, and it fails closed when the trace cannot be represented within budget. There is a lossless timing mode for when no approximation is acceptable.

Every approximation also carries a SHA-256 commitment to the exact source segment it summarizes. A third party holding the original trace can recompute every claim the canary makes about it, which turns “trust my compression” into a checkable statement. The same instinct shows up in a smaller place: the compiler reports event-count compression and serialized-byte compression as separate numbers, because a canary with fewer events than its source can still be more bytes, and calling that compression would be a lie with units.

For minimization there is a behavior-search mode, and it is the structural answer to ddmin. It compiles the trace at every timing budget in a declared range, runs the full verification on each candidate, rejects the failures, and keeps the smallest passing artifact by serialized size. Then it goes one step finer, greedily lowering the budget for individual operation-signature groups, but only while source verification, behavioral verification, and ranking verification all continue to pass, so a quiet group of collectives is not forced to carry the same sample budget as the windows the ranking actually depends on. The search is greedy and I will not pretend otherwise, there is no per-window Pareto optimality claim here. The difference from the reducer in the opening is not cleverness. It is that the degenerate answer is unreachable, because the gate the search must satisfy includes the behavior the workload is made of.

the things a trace refuses to guess

An artifact whose whole value is trust cannot contain invented fields, so a surprising amount of CommCanary is refusal.

A trace must commit to one timing semantics. Either every event carries a timestamp and time orders the trace, or none do and input order plus explicit gaps govern, and the mixed case is accepted only when every event supplies its own gap. An event carrying a timestamp and a gap that contradict each other is rejected, not arbitrated. Merging captures from distributed processes is fail-closed the same way: duplicate rank contributions, conflicting metadata about the same collective, and partially measured latency signals all abort the merge. And arrival timestamps from different machines are combined into cross-rank skew only when an explicit clock calibration is supplied. Without one, the merged trace records skew as unknown, and compilation refuses to turn unknown into zero, because zero skew is a strong physical claim and the absence of a measurement is not evidence of it.

The import path follows the same rule. CommCanary can ingest a PyTorch profiler trace, the Kineto format, reading the collective metadata that torch 2.2 and later record. That import is observational and single-rank, and the imported workload says so in its own notes: no cross-rank skew is invented, no compute overlap, no measured exposed latency. A collective with no mapping in the canary format is imported as an explicitly custom operation rather than dropped or relabelled as its nearest cousin. Rank lists that arrive truncated are reconstructed only from an explicit stride description, and otherwise the import fails rather than fabricate group membership.

None of these refusals makes a benchmark number better. They make the artifact’s silence trustworthy, which for a regression gate is the more important property. A canary that quietly filled its gaps with plausible defaults would still replay, still produce reports, still rank configurations. It would also be unfalsifiable, and unfalsifiable is the failure mode this whole tool exists to avoid.

a simulator that says so

The replay engine at the center of all this is a deterministic scheduler simulator. It is not NCCL, it does not touch a GPU, and the reports it produces are claims about a model of contention and overlap, not about silicon. The README says this plainly and the design leans into it rather than apologizing for it.

Determinism is what makes the verification story work at all. Reports embed enough protocol and sample data that verify-report can replay the declared canary under the declared settings and compare the result bit for bit, so an edited number in a report fails validation instead of surviving as a screenshot. Ablations, replay runs with one preservation mechanism deliberately removed, arrival skew, compute overlap, ordering, the rare tail windows, are recorded in the replay protocol and covered by the same verification. They are simulator controls for asking which contextual variable a ranking depends on, and they are labelled as that, not as physical interventions.

Claims about real hardware go through a different door. A compiled canary exports to the PARAM comms-replay format, the trace-replay harness from facebookresearch/param, which executes collectives through real NCCL on real GPUs. The export expands the canary’s full program, motifs and repetition weights included, into one entry per logical occurrence with cumulative timestamps, so timestamped replay reproduces the inter-operation gaps the compiler worked to preserve. Point-to-point transfers export as matched send and receive entries because PARAM executes each side on its own rank, and operations with no PARAM equivalent fail the export closed unless explicitly skipped. The division of labor is clean: the simulator validates the compression and the regression logic, and the physical path is where hardware claims have to be earned.

the campaign that has not run

What has been earned so far is narrow, and the repository is unusually direct about this. The physical numbers in the paper trail come from one historical campaign on a single node with four A100 PCIe GPUs, and the raw attempt archive from that campaign was never committed, so the current verification gate cannot regenerate those numbers from retained evidence. I went through what that archive would have needed to contain in the previous post, and the conclusion has not changed: reported evidence can motivate the design, and it did, but it cannot be promoted to a reproducible result by proximity to source code.

So the honest status is this. The verifier can prove, fail-closed and recomputably, that the small trace behaves like the big one inside the simulator. Whether the simulator’s rankings behave like the machine’s is a separate claim, and it belongs to the manifest-bound, hash-verified physical campaign described in the artifact-evaluation guide, on multi-node and NVLink-class topologies the historical campaign never touched. That campaign has not run yet.

One more boundary, because the name invites the misreading. Model-free means the artifact contains no weights and no application code. It is a statement about what was removed, not a proof about what the remaining timing structure can leak, and the shape of a workload’s communication is itself information. The canary was built to be small enough to share. Whether it is safe to share is a question the compression proofs do not answer, and I am not going to pretend they do.