We report a viability result: an entire Llama-class decoder, written in pure Rust and compiled to PTX by NVIDIA’s own experimental first-party Rust\u2192PTX backend (cuda-oxide), runs FP4-quantized weights on a consumer NVIDIA RTX 5070 Ti and generates coherent English text. On a real TinyLlama-1.1B model quantized to MXFP4, the engine sustains roughly 181 tokens\/s of decode throughput at steady state (median over a clean-GPU re-measurement; 188.6 tok\/s in the original session, which the clean run reproduced) – within about 3.7x, or roughly 27%, of llama.cpp’s decode throughput (\u2248664 tok\/s, Q4_K_M, hand-tuned CUDA with CUDA graphs) on the same GPU, on a first, unoptimized cut. Along the way we measured the actual programmable surface of consumer Blackwell (sm_120) for Rust kernels and found it richer than secondary sources imply: TMA, thread-block clusters with distributed shared memory, and native FP4 block-scaled tensor-core MMA all execute, while the tcgen05 5th-generation tensor-memory MMA pipeline correctly does not. We do not claim supremacy; llama.cpp is faster today, and we say where and why. The contribution is that the path exists end to end – a coherent LLM, compiled from Rust by NVIDIA’s compiler, on a gaming GPU – and that it lands close on the first attempt, with a clear optimization runway.<\/p>\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
We report a viability result: an entire Llama-class decoder, written in pure Rust and compiled to PTX by NVIDIA’s own experimental first-party Rust\u2192PTX backend (cuda-oxide), runs FP4-quantized weights on a consumer NVIDIA RTX 5070 Ti and generates coherent English text. On a real TinyLlama-1.1B model quantized to MXFP4, the engine sustains roughly 181 tokens\/s of […]<\/p>\n","protected":false},"author":351,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[11,89,3],"tags":[1782,14,20,2199,193,2149],"class_list":["post-30998","post","type-post","status-publish","format-standard","hentry","category-computer-science","category-nvidia-cuda","category-paper","tag-computer-science","tag-cuda","tag-nvidia","tag-nvidia-geforce-rtx-5070-ti","tag-ptx","tag-rust"],"views":374,"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/posts\/30998","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/users\/351"}],"replies":[{"embeddable":true,"href":"https:\/\/hgpu.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=30998"}],"version-history":[{"count":1,"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/posts\/30998\/revisions"}],"predecessor-version":[{"id":31001,"href":"https:\/\/hgpu.org\/index.php?rest_route=\/wp\/v2\/posts\/30998\/revisions\/31001"}],"wp:attachment":[{"href":"https:\/\/hgpu.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=30998"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hgpu.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=30998"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hgpu.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=30998"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}