I compiled a ten event trace into a canary and the canary came out twice the size of the trace.

$ commcanary compile examples/traces/llama70b_tp8_trace.json \
    --output out/demo/workload.canary.json
compiled 10 trace events into 5 canary events; event ratio=2.0x, byte ratio=0.474x, timing=lossless_timing

Ten events in, five out, and the tool calls that a two times event ratio. Fine. Then it says the byte ratio is 0.474x, which if you have been reading compression tool output for any length of time reads as a win, and it is not a win. It is the reciprocal. The canary is 2.1 times larger than the trace it came from.

$ stat -f%z examples/traces/llama70b_tp8_trace.json out/demo/workload.canary.json
6390
12468

I want to be careful here because there are two byte counts in play and I confused them for about ten minutes. The on-disk numbers above are pretty printed JSON. The ratio the compiler prints uses canonical serialization, which is a different number:

$ python -c "import json; c=json.load(open('out/demo/workload.canary.json'))['compiler']; print(c['source_bytes'], c['canary_bytes'], c['byte_compression_ratio'])"
3608 7617 0.474

3608 canonical bytes of trace became 7617 canonical bytes of canary. 3608 divided by 7617 is 0.474. So the ratio is source over canary, and a value below one means the artifact grew.

This is the point where a tool that was trying to look good would report one compression number, pick the flattering one, and move on. CommCanary reports two, and the one it reports second is the one that makes it look bad on a toy input. That was enough to make me want to run the whole thing end to end and see what else it admits to.

Everything below runs on a MacBook, macOS 15.7.7, arm64, Python 3.14.6, commcanary 0.3.0 out of the repository at 02d894a. There is no GPU in this machine and nothing here touches NCCL. Every number is the deterministic simulator talking about its own model. That limitation is load bearing and I come back to it at the end.

two ratios because one would be a lie

The reason there are two numbers is that the two things being counted are not the same thing, and a canary can move them in opposite directions.

The event ratio counts logical events. Ten source events collapsed into five canary events because the compiler found repetition it could encode exactly, run length for repeated operations and periodic encodings for regular structure. That is real reduction in the thing that costs time at replay, because replay walks events.

The byte ratio counts serialized bytes. And the canary carries things the trace does not: per field error bounds, SHA-256 commitments to the source segments it summarizes, compiler attestations, the timing mode it committed to, the scheduler identity. On a ten event trace that metadata dwarfs the payload. You are watching fixed overhead lose to a small input.

Both numbers are true and neither one alone describes what happened. A tool reporting only the event ratio would be claiming a two times compression while doubling the bytes on disk. The README calls that a lie with units, which is the correct name for it.

Watch what happens with a slightly larger input:

$ commcanary compile examples/traces/llama70b_tp8_trace_long.json \
    -o out/demo/gated.canary.json
compiled 56 trace events into 33 canary events; event ratio=1.697x, byte ratio=0.529x, timing=lossless_timing

Fifty six events, and the byte ratio moved from 0.474 to 0.529. Still under one, still growing the artifact, but the fixed overhead is starting to amortize. I do not have a trace big enough here to find the crossover point where the byte ratio passes one, and I am not going to guess where it is. The direction of travel is the observation, not the location.

Also note the event ratio went the other way, 2.0 down to 1.697. The longer trace has less exactly encodable repetition per event than the short one. That is a property of the trace, not of the compiler.

what the thing is actually for

None of this matters if the artifact does not do a job. The job is regression gating, so the loop is compile once, replay under two configurations, compare.

$ commcanary replay out/demo/workload.canary.json \
    --output out/demo/baseline.report.json --include-samples
replayed 10 events: median=91.977 us p95=107.746 us p99=108.321 us hidden=16.27%

$ commcanary replay out/demo/workload.canary.json \
    --output out/demo/candidate.report.json --include-samples \
    --latency-floor-us 12
replayed 10 events: median=121.409 us p95=146.616 us p99=152.095 us hidden=13.25%

The --latency-floor-us 12 is standing in for a configuration change. It raises the floor on per operation latency in the model, which is the shape of what a worse interconnect setting does to you. Median went from 91.977 to 121.409 microseconds. The hidden fraction, communication the model believes is covered by overlapping compute, dropped from 16.27 percent to 13.25 percent, which is the second order effect: slower collectives do not just cost more, they also stop fitting underneath the compute you were hiding them behind.

Then the comparison:

$ commcanary compare out/demo/baseline.report.json out/demo/candidate.report.json \
    --output out/demo/comparison.json
comparison verdict: fail
- p99 regression 40.4% exceeds 15.0%
- p95 regression 36.1% exceeds 10.0%
- median regression 32.0% exceeds 8.0%
- phase 'decode' p99 regression 41.5% exceeds 15.0%
- phase 'prefill' p99 regression 16.8% exceeds 15.0%
- operation 'all_reduce' p99 regression 40.4% exceeds 15.0%
$ echo $?
1

Exit code 1. That is the whole CI story and it is unglamorous, which is correct. The interesting part is the breakdown. It did not just say “things got slower by 32 percent”, it said decode regressed 41.5 percent at p99 while prefill regressed 16.8 percent, and it named all_reduce as the operation carrying it. On a real trace that is the difference between a red build and a red build you can act on.

I should be precise about what that verdict is evidence of. It is evidence that the simulator, replaying this canary under two of its own backend settings, produces latency distributions that differ by those percentages. It is not a measurement of a machine.

two thresholds, and the one that stops the noise

Those 15, 10 and 8 percent numbers in the failure lines are defaults, and if you are putting this in front of a rollout you are going to want to move them. There are more knobs than I expected:

$ commcanary compare --help
  ...
  --p99-threshold-pct P99_THRESHOLD_PCT
  --p95-threshold-pct P95_THRESHOLD_PCT
  --median-threshold-pct MEDIAN_THRESHOLD_PCT
  --p99-absolute-threshold-us P99_ABSOLUTE_THRESHOLD_US
  --p95-absolute-threshold-us P95_ABSOLUTE_THRESHOLD_US
  --median-absolute-threshold-us MEDIAN_ABSOLUTE_THRESHOLD_US
  --hidden-drop-threshold-points HIDDEN_DROP_THRESHOLD_POINTS
  --breakdown-threshold-pct BREAKDOWN_THRESHOLD_PCT
  --breakdown-absolute-threshold-us BREAKDOWN_ABSOLUTE_THRESHOLD_US

(trimmed; --output, --html and --allow-mismatch also live in there)

Every latency threshold comes in a percentage form and an absolute microsecond form. I assumed the absolute one was an alternative, an either-or, so that you could gate on “5 percent or 50 microseconds, whichever trips first”. Wrong way round. Watch what happens when I leave the percentages at their defaults and raise only the absolute floors:

$ commcanary compare out/demo/baseline.report.json out/demo/candidate.report.json \
    -o out/demo/cmp_b.json \
    --p99-absolute-threshold-us 1000 \
    --p95-absolute-threshold-us 1000 \
    --median-absolute-threshold-us 1000 \
    --breakdown-absolute-threshold-us 1000
comparison verdict: warn
- latency regression is below the failure threshold but large enough to inspect
$ echo $?
0

The same 40.4 percent p99 regression that failed the build a moment ago now warns instead. The percentage thresholds have not moved. A regression has to clear the percentage threshold and the absolute one before it fails, and a 40 percent regression on a p99 of 108 microseconds is 44 microseconds of movement, nowhere near the 1000 microsecond floor I just set.

That is the right way round, and I should have guessed it. The absolute floor is the noise suppressor. Percentages are meaningless on small numbers: a collective that takes 3 microseconds and now takes 4 has regressed 33 percent and nobody cares. Without an absolute floor, a purely proportional gate turns every fast operation into a source of red builds, which is how teams learn to ignore the gate.

The other thing that output shows is a third verdict state I had not noticed. It is not pass and fail, it is pass, warn, and fail, and warn exits 0.

$ commcanary compare out/demo/baseline.report.json out/demo/candidate.report.json \
    -o out/demo/cmp_a.json \
    --p99-threshold-pct 50 --p95-threshold-pct 50 \
    --median-threshold-pct 50 --breakdown-threshold-pct 50
comparison verdict: warn
- latency regression is below the failure threshold but large enough to inspect
$ echo $?
0

Raising the percentages to 50 gets the same warn. So a regression that is real but under your declared bar still shows up in the output and in the JSON, it just does not stop the pipeline. That distinction is what stops people from setting the thresholds at infinity: you can be permissive about blocking without becoming blind.

--hidden-drop-threshold-points is the one I would reach for that is not a latency threshold at all. The hidden fraction went 16.27 to 13.25 in this comparison, a drop of 3.02 points, meaning three percent more of the communication stopped fitting under compute. On a real workload that number moving is an early signal that survives even when the latency numbers look calm, because losing overlap costs you before it costs you visibly.

the number I could not check

Which raises the obvious question. If the reports are the product, and the reports come from a model, what stops anyone from editing a report?

$ commcanary verify-report out/demo/baseline.report.json out/demo/workload.canary.json \
    -o out/demo/v-clean.json
report verification: model_recomputed
- canary: pass
- simulation_model: pass
- replay_protocol: pass
- backend: pass
- workload: pass
- canary_summary: pass
- metrics: pass
- by_phase: pass
- by_op: pass
- calibration: pass
- samples: pass
$ echo $?
0

model_recomputed is the header naming what level of assurance you just got, and the eleven lines under it are what got recomputed. Note that simulation_model and replay_protocol are checked as well as the numbers. A report that was produced by a different scheduler version, or replayed under a protocol the report does not declare, does not quietly compare as equal.

verify-report takes the report and the canary it claims to describe, re-runs the scheduler model over the embedded samples under the declared backend settings, and checks that every row still recomputes. This only works because the simulator is deterministic. That is the actual argument for the simulator, and it is a better argument than “it is cheaper than a cluster”.

So I edited the report. One field, halved:

import json
d = json.load(open('out/demo/baseline.report.json'))
def bump(o):
    if isinstance(o, dict):
        for k, v in o.items():
            if k == 'median_us' and isinstance(v, (int, float)):
                o[k] = v * 0.5
                return True
            if isinstance(v, (dict, list)) and bump(v):
                return True
    elif isinstance(o, list):
        for v in o:
            if bump(v):
                return True
    return False
bump(d)
json.dump(d, open('out/demo/tampered.report.json', 'w'))
$ commcanary verify-report out/demo/tampered.report.json out/demo/workload.canary.json \
    -o out/demo/v-tamper.json
commcanary: report by_op row 'all_reduce' median_us does not match samples
$ echo $?
3

It caught it, and it caught it in the right way. It did not say “verification failed”. It named the row, the operation, and the field. The samples are embedded in the report, so the number and the evidence for the number travel together, and a number that has drifted from its own evidence is detectable by anyone holding the file.

Note that a screenshot of a report proves nothing and a report file proves quite a lot. That asymmetry is the point.

the budget it would not meet

Up to here the compiler had said yes to everything. I wanted to see it say no, so I gave it an error budget it could not hit.

$ commcanary compile examples/traces/llama70b_tp8_trace_long.json \
    -o out/demo/tight.canary.json \
    --timing-sample-limit 4 \
    --max-skew-error-us 0.1 \
    --max-overlap-error-us 0.1
commcanary: timing fidelity max_skew_error_us=2.779 us exceeds budget 0.1 us
$ echo $?
3

Four timing samples per group is not enough to represent this trace’s skew within a tenth of a microsecond, and rather than write the artifact with a warning attached, it refused and exited 3. No file at out/demo/tight.canary.json.

The failure message carries the number it actually achieved, 2.779 microseconds, next to the number I asked for. That is the difference between a refusal you can act on and a refusal you have to bisect. I now know my budget is off by roughly a factor of thirty, and I can either raise it or spend more samples.

There is also --lossless-timing, which is the version of this for people who will not accept any approximation at all. Both of the compiles I ran earlier landed in lossless_timing mode without being asked, because the traces allowed it.

the search that looked like a win

So I loosened the constraint and let the compiler go looking for the smallest artifact that still passes verification.

$ commcanary compile examples/traces/llama70b_tp8_trace_long.json \
    -o out/demo/searched.canary.json \
    --behavior-search \
    --behavior-search-min-sample-limit 2 \
    --timing-sample-limit 128
behavior search: evaluating up to 127 uniform candidates plus per-group refinement
compiled 56 trace events into 33 canary events; event ratio=1.697x, byte ratio=0.238x, timing=bounded_approximate
approximation: gap<=3.731571429 us, skew<=3.187 us, compute-before<=3.643 us, pressure<=0.0, prefix-gap<=4.666714286 us

Look at that third line before the fourth. The mode moved from lossless_timing to bounded_approximate, and the artifact now carries five explicit per field error bounds. Gap error at most 3.731571429 microseconds. Skew at most 3.187. Pressure at most 0.0, meaning that field came out exact.

The bounds are not rounded for display. That is deliberate and I like it. A bound that has been prettied up to “about 3.7 us” is a bound you cannot recompute against, and recomputing against it is the entire promise: the canary carries SHA-256 commitments to the exact source segments each approximation summarizes, so somebody holding the original trace can check every one of those five numbers.

And the byte ratio went from 0.529 to 0.238.

I wrote a sentence claiming that as a 2.2 times improvement in artifact size. Then I went to check it, because 0.238 is worse than 0.529 under the source-over-canary convention I had just spent two paragraphs establishing, and a search for the smallest passing artifact producing a bigger artifact did not make sense.

It is bigger. The search made the file bigger.

$ python -c "import json; c=json.load(open('out/demo/searched.canary.json'))['compiler']; print(c['canary_bytes'], c['behavior_search']['selected_canary_bytes_without_search_metadata'])"
79387 35077

79387 canonical bytes in the file. 35077 of those are the artifact the search selected. The other 44310 bytes are the search’s own record of what it tried and why it rejected it.

The comparison that matters is 35077 against the 35810 that a plain compile of the same trace produces. The search bought 733 bytes, two percent, and then attached a ledger sixty times that size to explain how.

I do not think this is a bug and I do not think it is dishonest. selected_canary_bytes_without_search_metadata exists precisely so you can do the subtraction I just did, which means somebody anticipated this confusion and shipped the field to resolve it. But the headline byte ratio=0.238x is measuring the file including its own audit trail, and if you read that line and stop, you will conclude the opposite of what happened. I read that line and stopped. That is where the ten minutes went.

The honest summary is that behavior search is not a size optimization. It bought 2.4 percent. What it actually did was change the mode from lossless to bounded approximate under a gate, and produce the receipts.

a decision is cheap to preserve

Which brings me to why the gate exists, and the fastest way to see that is to run the thing the gate was built to refuse.

The repository ships a generic delta debugging reducer as a research baseline. Delta debugging is old and reliable: delete chunks of the input, keep any deletion that preserves a chosen property, repeat until nothing more comes out. The property this one preserves is the decision, meaning the pairwise configuration rankings.

The scaffold builds an adversarial hundred event trace first:

$ python examples/research_scaffolding.py
isolated ranking: isolated-fast-no-overlap > workload-overlap-friendly
full workload ranking: workload-overlap-friendly > isolated-fast-no-overlap
too-small canary status: failed fail
random baseline status: failed failed fail
frequency baseline status: failed failed fail
cluster baseline status: failed failed fail
verified canary status: behaviorally_verified pass
behavior-search canary: 16 samples, behaviorally_verified pass

The first two lines are the whole motivation for the project in six words of output. Under an isolated collective benchmark, isolated-fast-no-overlap wins. Under the decode-like workload, the order flips. Same two configurations, opposite answers, and the only difference is whether the measurement preserved operation order, arrival skew, queue reset gaps, and the tail windows where those pile up.

The next four lines are the negative controls, and all four fail. Random sampling of events fails. Frequency representative sampling fails. Clustering, which preserves event count and operation order and several timing medoids per signature, fails. These are not strawmen, they are the reasonable things you would try, and the verifier rejects all of them.

Now the reducer, pointed at that hundred event trace:

$ commcanary reduce out/research_scaffold/adversarial_decode.trace.json \
    -o out/demo/reduced.trace.json
reduction: oracle-call budget 256
ddmin reduced 100 -> 1 events in 6 oracle calls

One event. Six oracle calls. The whole thing took 0.13 seconds of wall clock on this laptop.

It is worth being exact about what the oracle checked, because I got this wrong on a first pass and had to go read the ledger:

$ python -c "import json; print(json.dumps(json.load(open('out/demo/reduced.trace.json'))['workload']['reduction'], indent=2))"
{
  "budget_exhausted": false,
  "configurations": [
    "baseline",
    "low_latency",
    "high_bandwidth",
    "overlap_friendly",
    "congested"
  ],
  "method": "ddmin_ranking",
  "oracle_call_budget": 256,
  "oracle_calls": 6,
  "original_events": 100,
  "ranking_metrics": [
    "median_us",
    "p95_us",
    "p99_us",
    "mean_us"
  ],
  "ranking_tie_tolerance_us": 0.001,
  "reduced_events": 1,
  "timing_sample_limit": null
}

Five configurations. Ten unordered pairs. Four ranking metrics per pair. Forty pairwise decisions in total, and one event carried every one of them.

Forty bits is not very much information. That is the actual lesson and it took me a while to see it as an information problem rather than a compression problem. A ranking is a projection of a workload down to a handful of bits, and once you have compressed something to forty bits, an enormous number of unrelated artifacts agree with it by construction. The minimizer is not being clever. It is finding one of the many things that vote the same way.

what the gate says about it

So point the real verifier at that one event artifact:

$ commcanary compile out/demo/reduced.trace.json -o out/demo/reduced.canary.json
compiled 1 trace events into 1 canary events; event ratio=1.0x, byte ratio=0.286x, timing=lossless_timing

$ commcanary verify-behavior out/research_scaffold/adversarial_decode.trace.json \
    out/demo/reduced.canary.json -o out/demo/reduced.behavior.json
behavior verification: failed
- representation fidelity: lossless_timing
- source verified: failed
- behavioral fidelity: fail
- configuration ranking: pass
- baseline: fail
- low_latency: fail
- high_bandwidth: fail
- overlap_friendly: fail
- congested: fail
- ranking: pass

Configuration ranking passes. Every single other check fails.

That output is the argument. Not a paragraph about why decision preservation is insufficient, just the tool reporting separate statuses and two of them disagreeing with the other eight. If verify-behavior returned one blended verdict, this artifact would come back as some middling score and somebody would ship it.

The separate statuses matter individually. source verified: failed means the cryptographic commitments binding this canary to the source trace do not recompute, which they cannot, because 99 percent of the source is gone. Digging into the JSON there is a source_coverage_status of partial_source, which is the flag that stops a subset canary from ever carrying a strong claim regardless of how well it replays.

Against the artifact the behavior search produced:

$ python -c "
import json
d = json.load(open('out/research_scaffold/behavior_search.behavior.json'))
keys = ['status','source_verified_status','behavioral_fidelity_status',
        'configuration_ranking_status','source_coverage_status']
print(json.dumps({k: d[k] for k in keys}, indent=2))"
{
  "status": "behaviorally_verified",
  "source_verified_status": "source_verified",
  "behavioral_fidelity_status": "pass",
  "configuration_ranking_status": "pass",
  "source_coverage_status": "full_source"
}

full_source, source_verified, and behavioral fidelity passing, which covers the latency distribution at median and tail quantiles, the queue wait distributions, the hidden communication, the per phase and per operation breakdowns, and tail event recall, meaning the events that made the source expensive are still present and still expensive.

The one event artifact and this one both preserve the ranking. Only one of them preserves the workload.

the flag that guards less than I thought

There is a compile flag called --require-behavior-verification, and the README describes it as intentionally stricter than field level fidelity: compilation fails unless the generated canary passes source verification, behavioral checks, and ranking verification. Reading that, I assumed it was the thing standing between a user and a bad canary, so I pointed it at the one event trace the reducer had just produced, expecting a refusal.

$ commcanary compile out/demo/reduced.trace.json \
    -o out/demo/gated_reduced.canary.json --require-behavior-verification
compiled 1 trace events into 1 canary events; event ratio=1.0x, byte ratio=0.275x, timing=lossless_timing
$ echo $?
0

It compiled. Exit 0. The gate that is supposed to be the strict one waved through the degenerate artifact I had spent two sections calling degenerate.

It took me longer than it should have to see why, and the answer is obvious in hindsight. The gate verifies the canary against its own source trace, and its own source trace is reduced.trace.json, the one event file. The canary is a faithful representation of that file. It genuinely does preserve everything about the input it was given. The input is the problem, and the compiler has no way to know that, because the hundred event trace the reduction came from is not an argument to this command.

So --require-behavior-verification answers “did compilation lose anything?” It does not answer “is this trace worth compiling?” Those are different questions and only the first one is a property of the compiler.

The second question is what verify-behavior is for, and it needs the original trace passed explicitly, which is exactly what I did two sections ago to get the source verified: failed result. Same artifact, different question, opposite verdict. Both correct.

I am flagging this because the flag name reads like a general safety net and it is not one. If you compile a subset of your workload with the gate on, it will pass, and the resulting canary will be labelled partial_source in its own metadata rather than rejected at compile time. The labelling is the honest part. But the refusal I expected happens one command later than I expected it, and if you are wiring this into automation, that is the difference between a gate that fires and a gate that does not.

the control that gets closest

The scaffold’s negative controls all fail, which is easy to be suspicious of. Synthetic adversarial trace, synthetic baselines, everything fails except the thing the authors built. So I ran the strongest control against the ordinary bundled example trace instead, the fifty six event one, nothing adversarial about it.

The cluster method is the strongest of the baselines. It keeps event count, operation order, operation signatures, and several deterministic timing medoids per signature. The only thing it throws away is exact burst and tail correlation, plus the source commitments.

$ commcanary baseline examples/traces/llama70b_tp8_trace_long.json \
    -o out/demo/cluster.trace.json --method cluster --cluster-count 8
wrote cluster baseline trace with 56 events: out/demo/cluster.trace.json

$ commcanary compile out/demo/cluster.trace.json -o out/demo/cluster.canary.json
compiled 56 trace events into 33 canary events; event ratio=1.697x, byte ratio=0.736x, timing=lossless_timing

Byte ratio 0.736, the best of anything I compiled in this whole session, because medoid timings compress better than real ones. Then:

$ commcanary verify-behavior examples/traces/llama70b_tp8_trace_long.json \
    out/demo/cluster.canary.json -o out/demo/cluster.behavior.json
behavior verification: failed
- representation fidelity: lossless_timing
- source verified: failed
- behavioral fidelity: fail
- configuration ranking: pass
- baseline: fail
- low_latency: fail
- high_bandwidth: fail
- overlap_friendly: fail
- congested: fail
- ranking: pass

Identical shape to the one event artifact. Ranking passes, everything else fails.

That is a much more convincing result than the scaffold’s version of it, because this control is genuinely reasonable. Preserving count, order, signatures, and per signature timing medoids is what a competent person would build if you asked them to shrink a trace and keep it representative. It survives the check most people would apply. It fails on tail correlation, which is where the regressions live.

The two artifacts that fail here could not be more different. One is a single event that discarded 99 percent of the trace. The other keeps every event and every operation in the right order. Same verdict, and the ranking passes in both cases, which by now should be unsurprising. Forty bits is not hard to hit.

the parts that refuse to guess

Getting a real trace into the tool is where I expected the friction, and it is where the design gets most opinionated.

I do not have a multi-GPU machine, so I hand wrote a PyTorch Kineto profiler trace against the shape the importer reads, which is record_param_comms events with a Collective name in their args. This is a constructed fixture, not a captured profile, and it lives in writing-notes/probes/canary/make_kineto.py. Five events: two allreduces, an allgather, a barrier, and a reduce, which CommCanary’s format has no mapping for.

$ commcanary import-kineto out/demo/kineto_clean.json \
    -o out/demo/imported.trace.json --workload-name llama70b-serve --phase decode
imported 4 collective events (skipped 1 control, 0 empty)

Four of five. The barrier is a control op and got dropped, and it said so. The reduce came through, with the op name, the custom flag, the byte count and the rank list:

$ python -c "
import json
for e in json.load(open('out/demo/imported.trace.json'))['events']:
    print(e['op'], e.get('custom_op', False), e['bytes'], e['ranks'])"
all_reduce False 16777216 [0, 1, 2, 3]
all_gather False 4194304 [0, 1, 2, 3]
reduce True 8388608 [0, 1, 2, 3]
all_reduce False 16777216 [0, 1, 2, 3]

Flagged custom_op rather than dropped, and rather than quietly relabelled as its nearest cousin. Mapping a reduce onto all_reduce would have produced a trace that compiles, replays, and ranks, and is wrong about what the workload does. The custom flag propagates: an operation with no PARAM equivalent fails the export closed later unless you explicitly skip it.

The imported trace also writes its own limitations into itself:

$ python -c "import json; print(json.load(open('out/demo/imported.trace.json'))['workload']['notes'])"
Imported from a single rank's PyTorch Kineto profiler trace. Single-rank
observational import: no cross-rank arrival skew, compute overlap, or measured
exposed latency is claimed. Skipped 1 control op(s), 0 zero-sized message(s),
and 0 nested duplicate record(s).

A single rank profile cannot see cross-rank skew. There is no rank to compare against. So the imported workload carries a note saying skew is not claimed, and downstream compilation will not turn that absence into zero. Zero skew is a strong physical claim and the absence of a measurement is not evidence for it.

Then I truncated a rank list, the way a profiler does when a group is large:

$ commcanary import-kineto out/demo/kineto_truncated.json -o out/demo/imported_bad.trace.json
commcanary: kineto process-group ranks are truncated or non-uniform and cannot be
reconstructed from a global rank start/stride; refusing to fabricate group membership

refusing to fabricate group membership. There is a Group size: 4 sitting right there in the args, and filling in [0, 1, 2, 3] from it would be correct most of the time. The importer will do exactly that reconstruction if you give it an explicit Global rank start and Global rank stride, because then it is arithmetic rather than a guess. Without them it stops.

That is the third refusal in this post, after the error budget and the tamper check, and they are all the same move. Fail closed, name the reason, do not round an absence up to a value.

There is a fourth if you count capture, which runs an instrumented command and collects a trace from it. I pointed it at a Python process that does not emit collectives, to see what it would do with nothing:

$ commcanary capture -o out/demo/captured.trace.json --workload-name smoke -- python -c "print('hello')"
hello
commcanary: target command did not write a trace; import commcanary.capture.record_collective or pass --allow-empty

An empty trace is a valid thing to want and an invalid thing to produce by accident, so it stops and names both the fix and the override in one line. I did not have a distributed workload to point it at, so that is as far as I took it.

the door to actual hardware

Everything so far has been the simulator agreeing with the simulator. The way out is the PARAM export, which turns a canary into a comms replay trace that facebookresearch/param executes through real NCCL on real GPUs.

$ commcanary export-param out/demo/workload.canary.json \
    -o out/demo/param_trace.json --dtype float32
exported 11 PARAM comms-replay entries: out/demo/param_trace.json

Eleven entries from a five event canary. The export expands the canary’s program back out, motifs and repetition weights included, into one entry per logical occurrence with cumulative timestamps, so a timestamped replay reproduces the inter-operation gaps the compiler worked to preserve. First entry is a process group init:

{
  "comms": "init",
  "global_ranks": [0, 1, 2, 3, 4, 5, 6, 7],
  "markers": ["commcanary:pg-init:tp0"],
  "pg_id": 0,
  "req": 0,
  "startTime_ns": 0,
  "world_size": 8
}

The expansion is the interesting bit. Compression toward five canary events and expansion back to eleven PARAM entries are not in tension, they are the same fact viewed from either end. Motifs are how the artifact stays small in storage while staying faithful in execution. Point to point transfers export as matched send and receive pairs, because PARAM runs each side on its own rank.

I cannot run the other end of this. It needs multiple GPUs and NCCL, and this machine has neither.

where it stops

I set out to write down how to use a tool and spent most of the time checking whether its numbers meant what I assumed. Four of them did not, in decreasing order of my own embarrassment: the byte ratio is inverted from the convention I expected, the behavior search byte ratio includes the search’s own ledger and hides that it saved 733 bytes, --require-behavior-verification verifies against the trace you hand it rather than the workload that trace came from, and the reduce oracle checks forty pairwise decisions across five configurations rather than the twenty eight I had in my head from an earlier draft.

Three of the four were recoverable from the artifacts, because the artifacts carry the fields you need to catch them. source_bytes next to canary_bytes. selected_canary_bytes_without_search_metadata next to canary_bytes. The full reduction ledger with its configuration list and its metric list. I did not have to trust a summary line in any of those cases, which is the property the whole design is chasing, and it held up under someone actively trying to misread it.

The fourth one is not in an artifact. It is in the gap between what a flag is named and what it checks, and the only way I found it was by expecting a refusal and not getting one. partial_source in the output metadata is the tool telling you, after the fact, that it knew. I would rather it refused at compile time, and I do not think the current behavior is wrong so much as under advertised.

What did not get tested is anything outside the model.

Every verification in this post is the deterministic scheduler checking its own arithmetic. verify-report recomputes bit identically because the model is deterministic. verify-behavior compares two replays under the same model. The ranking inversion in the scaffold is a property the scaffold was constructed to have. All of that is a real and checkable claim about compression preserving behavior, and none of it is a claim that the model’s rankings match a machine’s rankings.

The path for that claim exists. It runs through the PARAM export, onto multi-node and NVLink class topologies, under the manifest bound campaign written up in docs/artifact-evaluation.md. That campaign has not run. Until it does, the correct reading of every number in this post is that a simulator was internally consistent to eight decimal places about a model of contention it defined itself.

I would rather have the 733 bytes explained than the twenty percent regression assumed.