WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms
The problem is that LLMs' ability to perform temporal reasoning over digital waveforms, a critical bottleneck in design verification, remains unexplored despite their strong code generation capabilities. The method introduces WaveformQA, an open-source benchmark with 360 programmatically grounded questions across eight difficulty categories, including multi-signal correlation and event ordering. Experimental evidence shows frontier LLMs achieve reasonable accuracy on simple queries but degrade on complex temporal and multi-step questions due to context window and reasoning limitations, with event-time JSON representation outperforming VCD format. This matters because WaveformQA provides a reproducible, extensible framework for evaluating and improving LLM temporal reasoning in real hardware verification tasks.