[Submitted on 4 Feb 2026 (v1), last revised 22 Jul 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:Large Language Models (LLMs) incur substantial memory and computation costs. Prior works reduce FP-INT arithmetic overhead by converting linear-layer activations to block floating point (BFP), but retain FP activations in attention layers due to accuracy concerns. We propose Harmonia, an algorithm-hardware co-design framework that enables BFP representation and computation across both linear and attention layers. Harmonia first explores BFP configurations to balance model accuracy and activation compression. It then combines asymmetric bit allocation with hybrid offline-online outlier smoothing to compress the KV cache from FP16 to 4-bit-mantissa BFP with less than 1% average accuracy loss on LongBench. To efficiently support all-layer BFP computation, Harmonia incorporates a reconfigurable PE unit for mixed data formats and precisions, a real-time FP16-to-BFP converter, and a flexible tiling-aware dataflow that reduces external memory traffic. Evaluations across eight widely used LLMs show that Harmonia achieves 3.84x higher area efficiency, 2.03x better energy efficiency, and 3.08x speedup on average, with maximum improvements of 5.05x, 3.90x, and 4.62x, respectively.

Submission history

From: Xinyu Wang [view email]
[v1] Wed, 4 Feb 2026 14:22:08 UTC (28,667 KB)
[v2] Sun, 22 Feb 2026 03:10:11 UTC (18,300 KB)
[v3] Wed, 22 Jul 2026 11:57:18 UTC (19,277 KB)