Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent
Atrex-Bench addresses the problem that existing GPU kernel benchmarks use synthetic or curated workloads, not production traces. The method samples 30 operators and 440 shapes from full-cluster production inference traces, weighting each by observed GPU time and card-hours. Experimental evidence shows the best vanilla coding agent reaches only ~10% of the hardware roofline on production operators, with much of the pass rate from PyTorch fallbacks. The co-released Atrex-Kernel-Agent (AKA) converts zero-FlyDSL fallbacks into real kernels matching hand-tuned baselines, demonstrating that profile-driven optimization is critical for production readiness.