performance profiling parallel

Performance Profiling is the measurement and analysis of where a parallel program spends time and resources — identifying bottlenecks that limit performance and guiding optimization efforts to maximum effect.

Profiling Workflow

1. Hypothesis: Where is the bottleneck? (CPU compute? Memory? GPU kernel? Communication?)
2. Instrument: Enable profiling — minimal overhead tools preferred.
3. Collect: Run with profiler attached → gather data.
4. Analyze: Identify top time consumers, hotspots, stalls.
5. Optimize: Fix bottleneck.
6. Verify: Measure speedup, ensure no regression.

GPU Profiling Tools

NVIDIA Nsight Systems:
- System-wide timeline: CPU threads, CUDA kernels, memory transfers, NVLink.
- Shows GPU utilization, transfer overlap, synchronization gaps.
- CLI: `nsys profile --trace=cuda,nvtx ./app`

NVIDIA Nsight Compute:
- Kernel-level analysis: Throughput, occupancy, instruction mix, memory bandwidth.
- Roofline model view: Is kernel compute-bound or memory-bound?
- Source-level metrics: Which lines have most cache misses.

CPU Profiling Tools

Intel VTune:
- Hotspot analysis, threading, memory access patterns.
- Microarchitecture analysis: Front-end stalls, back-end stalls, cache misses.
- Platform: Windows and Linux, all Intel and AMD CPUs.

Linux perf:
- Sampling profiler: `perf record -g ./app` then `perf report`.
- Hardware counters: cache-misses, branch-misses, cycles, instructions.
- Flame graphs: Hierarchical call-stack visualization.

Memory Profiling

• Valgrind Massif: Heap memory usage over time.
• CUDA memcheck / compute-sanitizer: GPU memory errors.
• Heaptrack: Fast heap profiler with stack unwinding.

Key Metrics to Examine

• GPU: SM utilization, memory bandwidth utilization, occupancy, warp efficiency.
• CPU: Instructions per cycle (IPC), cache miss rate, vectorization ratio.
• MPI: Communication time fraction, synchronization overhead, load imbalance.

Amdahl's Law in Practice

• Profile first: Optimize the 20% of code that takes 80% of time.
• Common mistake: Optimize clean code that contributes < 1% of runtime.

Performance profiling is the scientific method for parallel optimization — without measurement, optimization is guesswork; with proper profiling, optimization effort can be directed to where it matters most, achieving maximum speedup per engineering hour invested.