high performance computing on graphics processing units: hgpu.org

Efficient inter-GPU communication is very important for scalable distributed deep learning, yet the internal behaviour of NVIDIA’s Collective Communication Library (NCCL) at the GPU kernel level remains largely unexplored. Existing profiling tools observe only host-side call boundaries, giving no visibility into the individual send, receive, and reduce steps that constitute each collective operation.This thesis aims to address this by introducing a fine-grained modification embedded directly into the NCCL source. NCCL collectives can use different algorithms to perform their task, similarly they can use different protocols, which in turn use primitives to perform operations like send, receive or reduce. We implement a tracer, that wraps each collective as well as primitive step in order to get timing and metadata information. We try to introduce as little slowdown as possible, as we wish our traces and timings are as accurate as possible.Experiments were conducted on two GPU platforms spanning five collectives, two algorithms (Ring and Tree), three protocols (LL, LL128, Simple), and message sizes from 4KB to 1GB. The evaluation reveals several findings inaccessible to existing tools. Rather than treating a collective as a single black box, the tracing shows that it consists of a sequence of fine-grained steps (send, receive, and reduce), whose behavior varies significantly across conditions. This highlights that optimizing collectives requires understanding and improving individual steps, not just end-to-end execution.The results demonstrate that fine-grained kernel-level tracing can provide useful insight into NCCL’s internal execution behavior while introducing only modest runtime overhead. Beyond performance analysis, the collected measurements also provide a foundation for improving the realism of future GPU communication simulators.