Filtered by: Transformer × Clear all

Attention by Synchronization in Coupled Oscillator Networks

Fabio Pasqualetti, Taosha Guo 2026-06-14
Attention Language Model Training Transformer × Hardware Network

The problem is that softmax attention requires exponentiation and global reduction, which are energy-expensive on von Neumann hardware and lack a natural physical analog. The method replaces softmax with Kuramoto synchronization dynamics, where queries are fixed anchors on a sphere and free oscillators equilibrate to encode attention weights via cosine similarity. Experimental evidence shows that at oscillator dimension 2, oscillator attention outperforms softmax on keyword spotting (+1.00 pp) and subject-verb agreement (+5.27 pp), while on causal language modeling it closes the perplexity gap as dimension increases, from +11.09 to +2.98 on WikiText-2 and from +2.39 to +0.57 on TinyStories. This matters because it provides a mathematically grounded blueprint for accurate attention on energy-constrained physical substrates without requiring exponentiation or global reduction.

PDF

GF-DiT: Scheduling Parallelism for Diffusion Transformer Serving

Xinwei Qiang, Yifan Hu, Shixuan Sun, Jing Yang 2026-06-14
Transformer × GPU Runtime Scheduling Parallelism Serving

The problem is that existing Diffusion Transformer (DiT) serving systems use static parallelism for each request, which is inefficient due to heterogeneity across requests, execution stages, and system conditions. GF-DiT introduces a policy-programmable runtime that dynamically adapts parallelism via an asynchronous execution abstraction and group-free collectives for low-overhead online GPU reallocation. Experimental evaluation in vLLM-Omni shows GF-DiT improves throughput by up to 6.01×, reduces mean latency by up to 95%, and lowers SLO violation rates by up to 90% compared to fixed-pipeline execution. This matters because it enables efficient, elastic DiT serving that treats GPU parallelism as a schedulable resource, significantly improving performance and service quality for image and video generation workloads.

PDF