AI performance optimization GEMM kernels
In the rapidly evolving field of artificial intelligence, performance optimization is key to advancing the capabilities of deep learning models. One area of focus is the optimization of General Matrix Multiplication (GEMM) kernels, which are fundamental to the workloads of large language models (LLMs).
This blog post explores a significant advancement in this domain: the development of an optimized Triton BF16 Grouped GEMM kernel, designed specifically for Mixture-of – Experts (MoE) models. This innovation plays a crucial role in accelerating both training and inference processes, offering substantial speed improvements over traditional methods, especially regarding AI performance optimization, especially regarding General Matrix Multiplication, particularly in GEMM kernels in the context of AI performance optimization, especially regarding General Matrix Multiplication in the context of GEMM kernels. At the heart of many AI models, including MoE architectures like DeepSeekv3, GEMM operations are pivotal.
These operations involve the multiplication of input activation matrices by weight matrices and are central to the computational demands of AI workloads. In MoE models, the tokens are dynamically routed to different experts, resulting in numerous independent GEMMs.
A Grouped GEMM kernel, capable of executing these operations collectively in a single kernel launch, reduces overhead and enhances GPU utilization, particularly in AI performance optimization, including GEMM kernels applications. Recent optimizations have demonstrated speedups of up to 2.62 times over traditional methods, such as manual PyTorch loop implementations on NVIDIA H100 GPUs (PyTorch, 2025).
AI performance optimization kernels
The design of efficient kernels is crucial for maximizing the performance of NVIDIA GPUs, which utilize streaming multiprocessor units (SMs) for load, store, and compute operations. In traditional kernel designs, a new threadblock (CTA) is launched for each tile of work.
However, persistent kernels keep CTAs “alive” and dynamically assign them new tasks until the entire GEMM operation is complete, including AI performance optimization applications in the context of General Matrix Multiplication in the context of GEMM kernels. This approach minimizes launch overhead, improves cache reuse, and addresses scheduling imbalances, such as wave quantization, which occur when tasks are not evenly divisible among GPU resources (Colfax Research, 2025). By applying this persistent kernel strategy to Grouped GEMM kernels, significant performance gains can be achieved.
The Triton kernel is configured to match the number of SMs on an H100 GPU, allowing for a single wave of computation that processes the entire matrix multiplication task, especially regarding AI performance optimization, including General Matrix Multiplication applications. This approach ensures that all Triton programs are continuously active on the SMs, eliminating repeated launches and maintaining a single continuous wave of computation.
L2 cache management optimization
Efficient cache management is another critical factor in kernel performance. In Triton, programmers can control the order in which output tiles are computed, allowing for optimization of L2 cache performance.
Two main strategies exist: linear tile ordering and grouped launch ordering. The latter, which holds bands of rows in cache and processes them in a column-major fashion, has shown superior performance, including AI performance optimization applications, including General Matrix Multiplication applications in the context of GEMM kernels, especially regarding AI performance optimization in the context of General Matrix Multiplication in the context of GEMM kernels. This approach increases cache performance for both input matrices, enhancing data reuse and reducing latency.
In tests, the grouped launch ordering strategy demonstrated a 1.33 times speedup and a 60% increase in L2 cache hit rates compared to linear launch orders (Triton, 2025). This method improves temporal locality by reordering program launches to better utilize input activations and expert weights, enhancing arithmetic intensity and reducing kernel latency.
Tensor Memory Accelerator optimization
Another innovative technique in optimizing GEMM kernels involves using the Tensor Memory Accelerator (TMA) on NVIDIA Hopper GPUs. This unit facilitates efficient load/store operations on tensors, freeing up SM resources while data is transferred from global to shared memory.
For MoE models, where the selection of experts occurs dynamically at runtime, a modified approach to TMA utilization is necessary, especially regarding AI performance optimization, particularly in General Matrix Multiplication. By dynamically creating local TMA descriptors based on runtime selections, the Grouped GEMM kernel can effectively target TMA loads to the appropriate expert weights. This method ensures that data is not read incorrectly, maintaining the integrity of the computation and enhancing overall efficiency.
Triton GEMM AI performance optimization
The newly optimized Triton Grouped GEMM kernel has been rigorously benchmarked against a baseline Triton kernel to isolate the benefits of the discussed optimizations. These benchmarks reveal up to a 1, particularly in AI performance optimization in the context of General Matrix Multiplication, especially regarding GEMM kernels.50 times speedup over the baseline, highlighting the significant improvements made possible through persistent kernel design, grouped launch ordering, and TMA utilization.
Furthermore, when integrated into frameworks like torchtitan, the optimized kernel demonstrates substantial end-to – end performance gains, underscoring its potential to accelerate the training and inference of complex AI models.
AI performance optimization efficiency
The advancements in GEMM kernel optimization represent a critical step forward in the field of AI, particularly for MoE models that demand high computational efficiency. By leveraging persistent kernel designs, optimizing cache performance, and utilizing advanced hardware features like the TMA unit, these innovations offer significant speedups and efficiency gains in the context of AI performance optimization, particularly in General Matrix Multiplication in the context of GEMM kernels.
As AI continues to evolve, such optimizations will be essential in pushing the boundaries of what these models can achieve, ultimately leading to more powerful and capable AI systems.
