Skywork/Matrix-Game-2.0
Image-to-Video
•
Updated
•
114
Tolga Cangöz
tolgacangoz
AI & ML interests
AIGC
Recent Activity
liked
a model
1 day ago
Skywork/Matrix-Game-2.0
updated
a model
2 days ago
tolgacangoz/MAGI-1-T2V-4.5B-distill-Diffusers
liked
a model
2 days ago
sand-ai/MAGI-1
Organizations
liked
a
model
1 day ago
updated
a
model
2 days ago
liked
a
model
2 days ago
reacted to
a-r-r-o-w's
post with 🧠
6 days ago
Post
1986
You would've implemented the 3-loop matrix multiplication many times as a ML practitioner, but the naive implementation is terrible for GPU performance. Modern GPUs achieve peak performance through careful memory access patterns and minimizing scheduling overhead.
In naive matmul (MxK . KxN), the computation happens in tiles - both for the output matrix and for how you read chunks from the input matrices. Each thread-block processes one output tile by loading corresponding tiles from input (for sum-reduction across K dimension), performing the computation, then terminating. The GPU launches many thread-blocks and schedules them across available streaming multiprocessors (SMs). When an SM finishes one tile, it gets assigned a new thread-block for the next uncomputed tile. This way, multiple output tiles are computed in parallel across the SMs, but we pay the cost for launching thread-blocks each time a new tile is computed.
Persistent matmul changes this approach. Instead of launching thread-blocks to compute some output tiles, computing the results on SMs in parallel, and repeating until all output tiles are computed, you launch only as many thread-blocks as you have SMs available (typically 80-132 on modern GPUs). These thread-blocks stay alive until all output tiles are computed, looping through multiple tiles sequentially. Each persistent thread-block may handle multiple output tiles.
The key benefit is the reduced thread-block launch latency. This persistence strategy, combined with other optimizations like coalesced memory loads/stores, block-tiling, warp-tiling, warp-specialization, double-buffering, ping-pong scheduling and other tricks, helps achieve peak performance. More on this in the future!
Code snippet for testing: https://gist.github.com/a-r-r-o-w/28339b442d164084506c0967029968a8
(Bonus: Since I've wanted to learn Manim for a while, this was a great opportunity to make a visualization for Naive VS Persistent matmul. Enjoy ✨)
In naive matmul (MxK . KxN), the computation happens in tiles - both for the output matrix and for how you read chunks from the input matrices. Each thread-block processes one output tile by loading corresponding tiles from input (for sum-reduction across K dimension), performing the computation, then terminating. The GPU launches many thread-blocks and schedules them across available streaming multiprocessors (SMs). When an SM finishes one tile, it gets assigned a new thread-block for the next uncomputed tile. This way, multiple output tiles are computed in parallel across the SMs, but we pay the cost for launching thread-blocks each time a new tile is computed.
Persistent matmul changes this approach. Instead of launching thread-blocks to compute some output tiles, computing the results on SMs in parallel, and repeating until all output tiles are computed, you launch only as many thread-blocks as you have SMs available (typically 80-132 on modern GPUs). These thread-blocks stay alive until all output tiles are computed, looping through multiple tiles sequentially. Each persistent thread-block may handle multiple output tiles.
The key benefit is the reduced thread-block launch latency. This persistence strategy, combined with other optimizations like coalesced memory loads/stores, block-tiling, warp-tiling, warp-specialization, double-buffering, ping-pong scheduling and other tricks, helps achieve peak performance. More on this in the future!
Code snippet for testing: https://gist.github.com/a-r-r-o-w/28339b442d164084506c0967029968a8
(Bonus: Since I've wanted to learn Manim for a while, this was a great opportunity to make a visualization for Naive VS Persistent matmul. Enjoy ✨)
- 3 replies
upvoted
a
paper
8 days ago
liked
a
model
16 days ago
upvoted
a
collection
16 days ago
Update README.md
#2 opened 29 days ago
by
tolgacangoz
Update README.md
#3 opened 29 days ago
by
tolgacangoz
Update README.md
#2 opened 29 days ago
by
tolgacangoz
Update README.md
#2 opened 29 days ago
by
tolgacangoz
Update README.md
#2 opened 29 days ago
by
tolgacangoz
Update README.md
#2 opened 29 days ago
by
tolgacangoz
Update README.md
#3 opened 29 days ago
by
tolgacangoz
Update README.md
#2 opened 29 days ago
by
tolgacangoz
Update README.md
#3 opened 29 days ago
by
tolgacangoz
Update README.md
#2 opened 29 days ago
by
tolgacangoz
Update README.md
#2 opened 29 days ago
by
tolgacangoz
liked
2
models
about 1 month ago