Posts on gau-nernst's blog

World's fastest (panel) QR factorization on B200

Sun, 27 Sep 2026 19:20:00 +0800

GPU MODE recently hosted a series of leaderboards for linear algebra kernels. I participated in the first one, QR decomposition, and claimed the 2nd place. I think it’s a breath of fresh air, a departure from the usual LLM-oriented kernels most people program for nowadays.

As usual, my submission is open-sourced at gau-nernst/gpu-mode-kernels. The final submission is codex/submission.py.

Introduction to QR decomposition

I don’t want to touch too much on this as I’m no expert on the topic (in fact, I only learned about QR decomposition, mostly from Codex, during the competition). There are excellent resources online explaining this, such as the one from Michael. Instead, I will focus on my thought process tackling the crux of the challenge: chained dependency.

Using TMA in CuteDSL

Mon, 25 May 2026 04:00:00 +0700

Some time ago, I explored using CuteDSL to write some kernels for vLLM to enjoy the JIT benefits (ability to specialize on more parameters, don’t need to recompile vLLM from source during development, etc…). I have always known that we can write CUDA C++ way (SIMT style) in CuteDSL: there are cute.arch.thread_idx()/cute.arch.block_idx() (equivalent of threadIdx/blockIdx in CUDA C++), PTX (or its NVVM equivalent) can be used freely in device code. However, Tensor Memory Accelerator (TMA) is a special exception. Typically we need to encode tensor descriptors on the host side using CUDA driver API cuTensorMapEncodeTiled(), but this is not exposed in CuteDSL!

tcgen05 for dummies

Sun, 21 Dec 2025 22:26:00 +0800

tcgen05 is the set of PTX instructions to program Tensor Cores on the latest NVIDIA Blackwell GPUs (sm100, not to be confused with consumer Blackwell sm120). At the time of writing, I couldn’t find a Blackwell tutorial in plain CUDA C++ with PTX, even though such exist for Ampere (alexarmbr’s and spatters’) and Hopper (Pranjal’s). So let’s write one, documenting my process of learning tcgen05 and reaching 98% of CuBLAS speed (on M=N=K=4096 problem shape)!

My first Multi-GPU kernel: Writing All-to-all for AMD MI300X

Sun, 02 Nov 2025 21:43:00 +0800

Last month, I participated in the AMD Distributed Challenge, hosted by GPU MODE. This was very exciting for me as it was the first time I learned how to write a multi-GPU kernel! Although I had a brief understanding of how DDP and FSDP worked under the hood via collective primitives like all-reduce and reduce-scatter, I didn’t know it was possible to perform remote memory access directly inside a kernel! It opens up a lot of opportunities for multi-GPU optimizations, especially overlapping compute with inter-GPU communications.

Use NVRTC to explore MMA instruction variants

Mon, 08 Sep 2025 08:44:00 +0800

Recently I tweeted about realistic Speed-of-Light (SOL) of 5090 and RTX PRO 6000 for some dtypes, and mobicham asked me about FP8 MMA with FP16 accumulation. I of last year would turn to Triton for this - it’s trivial to change the accumulation dtype of tl.dot(). However, I roughly know how to write a fast matmul kernel now, so why not do it myself! In addition, I have been tinkering around with torch.cuda._compile_kernel(), which compiles CUDA kernels super fast via NVRTC. This seems ideal for JIT-compiled kernels and trying out all possible variants of the MMA instruction.

Writing Speed-of-Light Flash Attention for 5090 in CUDA C++

Sat, 23 Aug 2025 15:25:20 +0800

In this post, I will walk through how I learned to implement Flash Attention for 5090 in CUDA C++. The main objective is to learn writing attention in CUDA C++, since many features are not available in Triton, such as MXFP8 / NVFP4 MMA for sm120. I also feel this is a natural next step after learning about matmul kernels. Lastly, there are many excellent blogposts on writing fast matmul kernels, but there is none for attention. So I want to take this chance to write up something nicely.