Skip to content

collective-report.md

Latest commit

 

History

History
199 lines (154 loc) · 10.5 KB

collective-report.md

File metadata and controls

199 lines (154 loc) · 10.5 KB
Raw

Generate a collective-communication report

:description: Learn how to generate a multi-GPU collective-communication report with TraceLens to diagnose scaling issues and separate communication from skew.
:keywords: TraceLens, collective communication, NCCL, multi-GPU, distributed training, AllReduce, AllGather, bandwidth, skew, straggler, ROCm, PyTorch profiler

This topic shows how to analyze multi-GPU collective operations across ranks to diagnose scaling issues, separating true communication time from synchronization skew.

Before you begin

Confirm you have the following before continuing.

  • TraceLens installed.
  • Per-rank PyTorch profiler traces from a distributed run (one trace per rank).

Step 1: Generate the report

If your traces are in a single directory:

TraceLens_generate_multi_rank_collective_report_pytorch \
    --trace_dir /path/to/traces \
    --world_size 8

If the per-rank files follow a naming pattern, use --trace_pattern with a single * placeholder for the rank:

TraceLens_generate_multi_rank_collective_report_pytorch \
    --trace_pattern "/path/to/trace_rank_*_step_3.json" \
    --world_size 8

For arbitrarily named files (for example, traces written by torch.profiler.tensorboard_trace_handler, which are not named rank0_trace.json), use --trace_glob with --rank_regex to extract the rank from each filename:

TraceLens_generate_multi_rank_collective_report_pytorch \
    --trace_glob "/path/to/tensorboard/**/*.pt.trace.json.gz" \
    --rank_regex "rank\[(?P<rank>\d+)\]" \
    --world_size 16

--rank_regex must contain a named group rank (or a single capture group).

Expected output: An Excel report summarizing collective operations across ranks, with aggregation metrics (mean, median, min, max by default). If neither --output_xlsx_path nor --output_csvs_dir is given, the workbook is written to --trace_dir (or the lowest common directory of the resolved traces).

Excel output requires `openpyxl`; TraceLens installs it automatically if it's
missing. Use `--output_csvs_dir` to write CSV files instead.

Step 2: Read the report sheets

Each collective is assigned a collective_id (its Process Group name plus an index_in_group reflecting its order in the trace); this is how the tool correlates the same collective across ranks. The workbook contains a sheet per collective type and analysis. The most useful:

Sheet Granularity What it contains
nccl_summary_implicit_sync One row per (collective name, dtype, message size) Aggregated latency, skew, and bandwidth (algorithm and bus) for collectives that incur implicit sync (AllReduce, AllGather, ReduceScatter, balanced AllToAll).
nccl_summary_long One row per (collective name, dtype, message size) Duration aggregates (sum/mean/std/min/max) and counts across all ranks.
nccl_implicit_sync One row per collective Per-collective implicit-sync breakdown; per-rank timestamps, durations, and wait_time are spread across rank_<i>_* columns.
nccl_long One row per (collective, rank) Per-event, per-rank NCCL records (timestamps, durations, stream, message sizes) for drilling into specific ops.
nccl_summary_all2allv / nccl_all2allv Per (process group, dtype) / per (collective, rank) All-to-all-v metrics (throughput, wall time, size imbalance, skew), aggregated and per-rank.
nccl_all2allv_heatmap One row per rank, one column per rank Per rank-pair send volumes (only with --all2allv_heatmap).
straggler_summary One row per rank Aggregated straggler metrics — see Step 3: Find the straggler rank.

Each sheet is produced only when the corresponding collective type is present in the trace (for example, the all2allv sheets are skipped if the run has no all-to-all-v). The detailed per-event sheets (nccl_long, nccl_implicit_sync, nccl_all2allv) are controlled by --detailed_analysis, which is on by default.

Interpret communication versus skew

The implicit-sync sheets separate the time spent in actual data movement from the time a rank spends waiting for other ranks to arrive. The key metrics:

Metric Meaning
comm_latency Communication time for the collective, taken as the minimum duration across ranks — this strips out per-rank waiting so it reflects real data movement.
algo bw Algorithm bandwidth: message size / comm_latency.
bus bw Bus bandwidth (link utilization).
wait_time, skew in start time How much later some ranks arrive than the earliest rank — the synchronization cost.

High skew points to load imbalance upstream rather than a slow network; high comm_latency with low skew points to a bandwidth or topology limit.

Interpret all-to-all-v collectives (MoE and expert parallel)

All-to-all-v sends a different amount of data per rank, so there's no single message size and the ring/tree algo bw / bus bw formulas don't apply. The all2allv sheets instead report:

Metric Meaning
throughput (GB/s) Total data moved / wall-clock time. Compare against link bandwidth to gauge efficiency.
wall_time End-to-end time from the first rank entering to the last finishing.
size_imbalance Max rank's data ÷ mean. 1.0 = balanced; >> 1.0 = some ranks carry disproportionate load (common when some MoE experts are hotter than others).
max_rank_dur / min_rank_dur Spread in per-rank kernel time. A large spread together with high size_imbalance points to rebalancing expert routing or dropping tokens.
skew in start time How far apart ranks enter the collective — large skew means some ranks are blocked by upstream compute.

To diagnose low throughput, check size_imbalance:

  • size_imbalance ≈ 1.0: all ranks are sending roughly equal amounts of data, so the slow throughput isn't caused by one rank doing more work than others. Look for software or driver overhead instead (launch latency, kernel scheduling).
  • size_imbalance >> 1.0: some ranks are sending more data than others. This is common in Mixture-of-Experts models where certain experts attract more tokens. The busiest rank takes the longest, and all other ranks wait for it to finish before the collective can complete.

To see which specific rank pairs are exchanging the most data and identify the overloaded rank, add --all2allv_heatmap.

Step 3: Find the straggler rank

A straggler is the rank that consistently arrives last at collectives, forcing the others to wait in implicit synchronization. Open the straggler_summary sheet: it's sorted so the straggler is the first row (lowest total wait time — it arrives last, so it rarely waits itself).

Example from an 8-rank Llama 70B FSDP run (straggler_summary, truncated):

rank total_wait_time_us mean_wait_time_us times_arrived_last times_arrived_first pct_arrived_last num_collectives total_nccl_dur_us
4 27,195 55.7 420 6 86.1% 488 3,910,753
5 4,660,353 9,549.9 39 10 8.0% 488 8,463,896
... ... ... ... ... ... ... ...
0 13,358,753 27,374.5 3 320 0.6% 488 17,203,432

Rank 4 is the straggler: the lowest total wait time, last to arrive 86% of the time, and the lowest NCCL kernel duration (other ranks' durations are inflated by the time they spend waiting for it).

Column What it tells you
total_wait_time_us Sum of this rank's wait across all collectives. The straggler has the lowest value.
times_arrived_last / pct_arrived_last How often this rank was last. The straggler dominates these.
times_arrived_first How often this rank was first; the rank highest here pays the biggest sync penalty.
total_nccl_dur_us Time inside NCCL kernels; the straggler is typically lowest (others are inflated by wait time).

Once you know the straggler, merge its trace with a fast rank using trace fusion and open the result in Perfetto to see what compute or memory work delays the straggler before each collective.

Command-line options

The following table describes all available options.

Option Default Description
--trace_dir / --trace_pattern / --trace_glob — Mutually exclusive ways to locate the per-rank traces (directory, *-pattern, or glob). Exactly one is required.
--world_size required Number of ranks.
--rank_regex rank[\[\-_/]?(?P<rank>\d+) With --trace_glob, regex (named group rank or one capture group) used to extract the rank from each filename.
--output_xlsx_path auto Output Excel path. If omitted, written to the trace directory / lowest common directory of the traces.
--output_csvs_dir None Write each sheet as a separate CSV here instead of (or alongside) Excel.
--detailed_analysis on Emit the per-event detail sheets (nccl_long, nccl_implicit_sync, nccl_all2allv) in addition to the summaries.
--agg_metrics mean median min max Aggregation metrics in the summary sheets.
--gpus_per_node auto Adds node_id / node_span columns and labels each process group intra_node or inter_node. Auto-detected from trace deviceProperties if omitted.
--all2allv_heatmap off Add the nccl_all2allv_heatmap sheet (per rank-pair send volumes) — useful for spotting imbalanced all-to-all / MoE routing.
--use_multiprocessing off Load traces in parallel — a significant speedup when processing many large per-rank traces.
--max_workers os.cpu_count() Cap on worker processes (requires --use_multiprocessing).

Example combining topology labeling, the heatmap, and parallel loading:

TraceLens_generate_multi_rank_collective_report_pytorch \
    --trace_dir /path/to/traces \
    --world_size 8 \
    --gpus_per_node 8 \
    --all2allv_heatmap \
    --use_multiprocessing

Related topics