The OoO VLIW JIT Compiler for GPU Inference

Paras Jain, Xiangxi Mo, Ajay Jain, Alexey Tumanov, Joseph E. Gonzalez, Ion Stoica
UC Berkeley, MIT
arXiv:1901.10008 [cs.DC], (31 Jan 2019)


   title={The OoO VLIW JIT Compiler for GPU Inference},

   author={Jain, Paras and Mo, Xiangxi and Jain, Ajay and Tumanov, Alexey and Gonzalez, Joseph E and Stoica, Ion},

   journal={arXiv preprint arXiv:1901.10008},



Download Download (PDF)   View View   Source Source   



Current trends in Machine Learning (ML) inference on hardware accelerated devices (e.g., GPUs, TPUs) point to alarmingly low utilization. As ML inference is increasingly time-bounded by tight latency SLOs, increasing data parallelism is not an option. The need for better efficiency motivates GPU multiplexing. Furthermore, existing GPU programming abstractions force programmers to micro-manage GPU resources in an early-binding, context-free fashion. We propose a VLIW-inspired Out-of-Order (OoO) Just-in-Time (JIT) compiler that coalesces and reorders execution kernels at runtime for throughput-optimal device utilization while satisfying latency SLOs. We quantify the inefficiencies of space-only and time-only multiplexing alternatives and demonstrate an achievable 7.7x opportunity gap through spatial coalescing.
Rating: 3.0/5. From 2 votes.
Please wait...

* * *

* * *

HGPU group © 2010-2024 hgpu.org

All rights belong to the respective authors

Contact us: