high performance computing on graphics processing units: hgpu.org

Writing custom GPU kernels using OpenAI Triton is a common approach for authoring deep learning operators. However, deploying models containing Python @triton.jit kernels into production C++ inference runtimes-specifically Microsoft ONNX Runtime (ORT) and NVIDIA TensorRT (TRT)-presents architectural limitations. Standard ONNX graph export mechanisms cannot trace imperative Python Triton calls, dynamic launch grid functions, implicit layout semantics, in-place memory mutability, dynamic runtime host scalars or low-level PTX entry parameter signatures. In this paper, we present KernelLens, a framework for compilation, C++/CUDA code synthesis and zero-copy runtime execution. KernelLens bridges PyTorch models containing custom Triton kernels directly to ONNX Runtime and TensorRT custom C++ plugins without manual C++ development. We present a formalization of KernelLens's consolidated 4-phase compiler architecture: (1) Analysis & Manifest Assembly via dual-pass tracing and AST inspection; (2) ONNX Graph Transformation with custom node injection (triton_custom:: ); (3) C++/CUDA Code Synthesis & Hardening with 16-byte memory alignment guards and PTX ISA version clamping; and (4) Zero-Copy Runtime Binding. We detail key architectural innovations in KernelLens, including TensorRT C++ plugin dynamic creators with explicit symbol registration, automated Blackwell (sm_120a) PTX ISA version clamping to satisfy CUDA driver JIT requirements, primary CUDA context retention and executable stack security clearance for Linux kernels. We evaluate KernelLens across research model kernels from LLaMA 3, Mistral, Qwen 2.5, PaLM and Liger-Kernel (Fused RMSNorm, SwiGLU, Rotary Position Embeddings and Fused Cross-Entropy Loss). KernelLens achieves exact floating-point numerical parity (MaxDiff = 0.00e+00) against eager Triton execution while enabling C++ inference deployment.