asquare.blog
Long-form, deeply technical writing on systems, GPUs, and distributed runtimes. Notes from work on HPX, CUDA, and the messy interior of modern compute.
Latest articles
- three hundred processes waited for one pid
HAMi issue 1662 blamed a lock file for a minute of startup latency. By the time I got there the lock had been replaced, and the expensive part was never the lock: it was a CUDA context created so a process could learn which PID the host calls it. One process carried from Kubernetes allocation to cuInit, then 300 released at once.
- the tensor was only the handle
A source-pinned reconstruction of PyTorch from one small training step: tensor identity, storage and views, ATen dispatch, modules, autograd, optimizers, modes, devices, transforms, compilation, export, data workers, distributed reducers, serialization, profiling, and custom operators.
- One collective call used two generations
A two-locality demo hangs before the hierarchy is even visible. Fixing it leads through HPX communicator servers, uneven trees, all-to-all routing, shared generation clocks, scans, flattened payloads, convergent failures, MPI shutdown, and one still-open endpoint repair.
- arrays become matrices
Six integers become a 2x3 matrix. Turning it sideways without moving them produces either the right answer or a very plausible wrong one, depending on two strides.
- my canary got bigger
Running CommCanary end to end on a laptop: a compression tool that doubled the size of its input, an error budget the compiler refused, a search whose own receipts outweighed what it saved, and the one number in the whole pipeline that turned out to mean what it said.
- one hundred events became one
A delta-debugging reducer deleted ninety nine of a trace's one hundred events and every configuration ranking survived. Why preserving the decision is the wrong gate on its own, and how CommCanary compresses a distributed LLM communication trace while proving the compression kept the answer.
- the address where the value lived
A pointer remembers the first element of a vector. One push_back later, the value survives and the pointer does not. Following the missing address through arrays, layout, lifetime, vector growth, arenas, and pools.
- the head pointer came back
A delayed dequeue sees the same head address twice and succeeds with a stale compare-and-swap. Following that returned address through FIFO invariants, helping, linearizability, hazard pointers, reclamation, and the actual limits of lock-free progress.
- the syscall the kernel refused
The same open call returns a file descriptor, and then returns Operation not permitted. Following the result through the arm64 trap, XNU's file authorization path, a process policy, Linux seccomp, and executable memory.
- idle gaps are not compute
A collective benchmark put one NCCL configuration safely in the middle. The workload put it last. Following that disagreement through trace capture, replay scheduling, CUDA streams, ranking statistics, missing evidence, queueing, and one source line that changed the explanation.
- Rings, Trees, and Butterflies: HPX Collectives
Eight ranks need the same sum. The HPX call leads into one communicator server, then into five executed all-reduce algorithms, a byte-and-message cost model, fresh MPI measurements, and a provenance-stamped cluster result that changes what the source alone seemed to imply.
- the page that was not there
A mapping succeeds with no resident pages, then one ordinary load per page enters the kernel 256 times. Following those loads through virtual addresses, arm64 translation, XNU faults, zero fill, protection, copy-on-write, file sharing, wiring, and the measurements that cannot reveal a TLB.
- how your computer fakes randomness
rand() returns the same numbers after every process start. The first value leads into state recovery, entropy, biased mappings, finite tails, forked streams, and benchmark results whose meaning changes when order and uncertainty are handled.
- one line to machine code
One line enters torch.compile. What comes back is a guarded region containing a library matrix multiply and a generated C++ loop, plus a crash that four threads can reproduce and one thread cannot.
- teaching a model
A rank-8 patch fit the training target while the frozen model forgot unrelated text. The loss said it learned. The held-out prompts forced a more careful answer.
- one gradient, backward
A thirty-line tape and PyTorch agree on one gradient. A finite difference does not. Following the discrepancy reaches the chain rule, reverse and forward accumulation, matrix backward, GPT-2, checkpointed activations, optimizer state, and the numbers fp16 cannot keep.
- one forward pass
GPT-2 put Paris fifth. Following the exact rows behind that result reaches byte-pair tokens, matrix products, attention, residual state, a growing KV cache, batch reuse, and the difference between simulating 4-bit error and running a packed kernel.
- how machines agree
One counter, two machines, one dropped packet, and the state that must survive retries, partitions, and a killed process before an acknowledged increment can be trusted.
- two machines talking to each other
What really happens when two machines talk to each other?
- where the time goes
One program that sums every byte of a 1 GiB file, and the same arithmetic made 4000x slower and then fast again, a walk through syscalls, the user/kernel boundary, 16 KiB pages, page faults, the page cache, mmap, P and E cores, sampling profilers, and using the roofline as a bottleneck detector.
- a file is just bytes
A column read that looked thirty-eight times faster led into Parquet pages, byte ranges, encodings, pruning, Arrow buffers, and the commit race hidden behind a table.
- the fast path out of python
Summing ten million numbers takes 290 ms in a Python loop and 3.4 ms in numpy. Following the gap reaches PyObject boxing, pairwise reduction, native buffers, ownership, the GIL, and the point where zero-copy stops being a sufficient claim.
- two threads to a task runtime
Two threads each add one to a counter a million times, and it comes out to 1,012,274, a different wrong number every run. Following that lost million down through the load-add-store race, atomics, the weakly-ordered memory model, false sharing, mutexes and deadlock, and up to why real systems hand you tasks and futures instead of raw threads.
- from a C loop to a GPU kernel
A walk from a slow 512x512 C matrix multiply through cache, SIMD, FMA, tiling, roofline thinking, Metal kernels, threadgroup memory, GEMM, reductions, softmax, attention, and fusion.
- sixteen million and one
A full walk through computer numerics from integer representation and IEEE-754 float bits to rounding, subnormals, cancellation, compensated summation, SIMD/FMA/BLAS, mixed precision, and quantization.
- from source to a running process
A C++ toolchain walk from preprocessing and templates to object files, ODR traps, static and dynamic linking, ABI symbols, relocations, GOT/stubs, lazy binding, dyld, and the first instruction of a process.
- Python is a C program
Five lines of Python followed all the way down: a 52 KB Mach-O launcher, tokens, AST, bytecode, _PyEval_EvalFrameDefault, PyObject, refcounts and immortals, cyclic GC, the GIL, free-threading, specialization, the experimental JIT, and the C API.
- std::move doesn't move anything
Value categories, RAII, move semantics, copy elision, smart pointers, templates, and concepts, traced from one line of libc++ source. With real benchmarks on an M4 and real HPX code throughout.