[Submitted on 14 Jul 2026 (v1), last revised 25 Jul 2026 (this version, v4)]

View PDF HTML (experimental)

Abstract:Modern edge system-on-chips (SoCs) combine CPUs, integrated GPUs (iGPUs), and neural processing units (NPUs), yet existing LLM runtimes typically make coarse device-level decisions or optimize operators in isolation. As a result, they underutilize heterogeneous resources, particularly on unified-memory platforms where performance depends on both device placement and task-graph coordination. We present HeteroMosaic, a heterogeneity-first scheduling framework for edge LLM inference. HeteroMosaic first uses a heterogeneous roofline model to identify when combining iGPU and NPU execution is beneficial. It then decomposes inference into dependency-preserving micro-batches that expose cross-accelerator overlap and applies trace-guided co-optimization of scheduling and device allocation under practical effects such as memory contention, DVFS, device variation, and NPU runtime overheads. We implement HeteroMosaic in PyTorch C++ and evaluate it on three AMD Ryzen AI platforms spanning NPU-heavy, balanced, and iGPU-heavy designs. On the balanced platform, HeteroMosaic achieves up to 1.73X speedup over an iGPU baseline, 1.78X over an NPU baseline, and 2.05X over frameworks such as llama dot cpp, while reducing energy by up to 45.3%. It also improves performance over prior heterogeneous edge AI solutions by up to 2.35X.

Submission history

From: Gregory Hyegang Jun [view email]
[v1] Tue, 14 Jul 2026 14:56:14 UTC (3,787 KB)
[v2] Wed, 15 Jul 2026 16:01:24 UTC (3,788 KB)
[v3] Thu, 16 Jul 2026 01:46:28 UTC (3,798 KB)
[v4] Sat, 25 Jul 2026 21:10:17 UTC (3,788 KB)