Photo by ThisisEngineering on Unsplash
- Hugging Face published a beginner-focused torch.profiler tutorial on May 29, 2026, arguing that dense, rectangle-filled traces — not lack of interest — are the main reason developers skip profiling.
- A 64×64 matrix multiplication trace shows roughly 1.78 milliseconds of cudaDeviceSynchronize overhead and a CPU-GPU lane offset of about 2.5 milliseconds, both artifacts of asynchronous execution rather than real compute cost.
- torch.profiler has effectively replaced the older autograd profiler, adding CPU, CUDA, and memory modes plus direct TensorBoard integration, per PyTorch's official documentation.
- torch.compile changes what a trace looks like entirely, fusing kernels so that a ProfilerStep#2 run often looks nothing like ProfilerStep#0 while warmup settles.
What's on the Table
1.78 milliseconds. That's how long a single synchronization call sits inside a trace of a 64×64 matrix multiplication — and according to Hugging Face, that one number is the moment most beginners quietly close the profiler tab and go back to guessing. As of July 17, 2026, that tutorial, published May 29, 2026 as Part 1 of a series, is still one of the more approachable entry points into a tool that PyTorch's own documentation describes mainly in reference-manual terms.
According to Hugging Face, the core problem isn't that profiling is unimportant. It's that "the traces are dense walls of colored rectangles. The events carry intimidating names," and nobody tells beginners which rectangles matter. The tutorial's opening line frames the stakes bluntly: "What you cannot profile, you cannot optimize." That's the whole pitch for why torch.profiler exists, and why skipping it means optimizing blind.
Side-by-Side: How torch.profiler and Autograd Profiler Differ
PyTorch's own documentation confirms torch.profiler is the direct successor to the older autograd profiler, and the difference is mostly in what you can see. The autograd profiler tracked operator-level CPU and CUDA timing well enough, but torch.profiler adds a memory-tracking mode and exports directly into TensorBoard, according to PyTorch's official docs at pytorch.org. That matters because a trace you can only read as a text dump gets ignored; a trace you can click through in a browser gets used.
Where the sources actually diverge is emphasis, not fact. Hugging Face's tutorial spends its energy on trace-reading pedagogy — walking through what a rectangle labeled cudaDeviceSynchronize actually represents. PyTorch's documentation, by contrast, stays in API-reference mode: it lists CPU, CUDA, and memory profiling modes without walking a reader through what any of it looks like on screen. That gap is exactly why a third-party tutorial had an opening to exist in the first place.
The two overhead numbers from the 64×64 example illustrate why beginners misread traces in the first place. Neither number reflects the matrix multiplication's actual math — both are artifacts of how CPU and GPU work queues talk to each other.
Chart: Overhead measured in Hugging Face's 64×64 matmul trace, published May 29, 2026 — neither figure represents actual GPU compute time.
For lower-level kernel analysis beyond what torch.profiler shows, NVIDIA's Developer Blog points to Nsight Systems and Nsight Compute as complementary tools that dig into individual CUDA kernel launch latency — a layer torch.profiler summarizes but doesn't fully expose on its own.
Photo by Javier Esteban on Unsplash
The AI Angle
None of this is academic for anyone training or serving large language models. Every millisecond of synchronization overhead multiplied across millions of inference calls is real infrastructure cost, which is why PyTorch's Blog frames the profiler's evolution alongside torch.compile as part of the same performance push: automatic kernel fusion under torch.compile changes trace shape entirely, so a ProfilerStep#2 run can look structurally different from ProfilerStep#0 simply because warmup hasn't finished settling. Teams tuning tokens-per-second throughput on transformer models can't separate compiler behavior from raw compute cost without reading a trace correctly first.
Which Fits Your Situation
Don't turn on memory profiling on your first pass. Per PyTorch's documentation, each mode adds its own overhead to the trace, and beginners reading a cluttered timeline is exactly the failure mode Hugging Face's tutorial is trying to fix.
When a rectangle labeled cudaDeviceSynchronize eats 1.78 milliseconds in a tiny 64×64 op, that's expected asynchronous-execution behavior, not a performance regression to chase.
Because kernel fusion reshapes the trace, a profile taken before torch.compile tells you little about the compiled model's real bottlenecks — run the profiler again post-compilation, and skip the first step or two while warmup settles.
Frequently Asked Questions
How do I use torch profiler in PyTorch?
Wrap the code you want measured in a torch.profiler.profile context manager, specifying which activities (CPU, CUDA) to record, then export the result to TensorBoard or print a summary table — the exact API is documented at pytorch.org/docs/stable/profiler.html.
What is the difference between torch profiler and autograd profiler?
Autograd profiler was the earlier, narrower tool for operator-level CPU/CUDA timing. torch.profiler is its successor, adding a memory-tracking mode and native TensorBoard export, according to PyTorch's official documentation.
How do I visualize PyTorch profiler results?
The most common route is exporting the trace to TensorBoard, where operations appear as colored rectangles across CPU and GPU lanes — the exact format Hugging Face's May 29, 2026 tutorial walks through step by step.
How does torch compile affect profiling traces?
torch.compile fuses multiple kernels together, so compiled traces look structurally different from eager-mode traces, and early ProfilerStep entries (like ProfilerStep#0) can misrepresent steady-state performance until warmup finishes.
On balance, the more useful takeaway from this tutorial isn't the specific millisecond figures — it's that most of what looks alarming in a fresh trace is asynchronous plumbing, not a bug in your model. Reading traces correctly, our analysis suggests, is a bigger unlock for most teams than any single optimization technique.
Disclaimer: This article is editorial commentary based on publicly available technical documentation and does not constitute professional engineering advice. Research based on publicly available sources current as of July 17, 2026.