Building an AI Runtime Operating System for Commodity Hardware

For the last few months I've been working on something that started as a simple question:

Why do we still treat AI inference as "load an entire model into GPU memory and hope it fits"?

My development machine certainly doesn't make life easy.

  • Dell Precision 5520
  • NVIDIA Quadro M1200 (4 GB VRAM)
  • 32 GB RAM
  • Ubuntu

Like many developers experimenting with open-source LLMs, I quickly ran into the same wall everyone eventually hits:

CUDA Out of Memory

Enter fullscreen mode Exit fullscreen mode

At first I assumed the answer was simply "buy a bigger GPU."

But after digging through projects like llama.cpp, Ollama, TensorRT-LLM, Colibri, and various memory streaming experiments, I started wondering whether we were solving the wrong problem.

The Problem Isn't Just Memory

Most inference engines think about execution like this:

Application
    ↓
Inference Engine
    ↓
GPU

Enter fullscreen mode Exit fullscreen mode

If the model fits, great.

If it doesn't...

You're usually done.

Modern runtimes have become incredibly efficient, but they're still primarily focused on executing tensors as quickly as possible.

What if we instead treated inference more like an operating system treats processes and memory?

From Runtime to Runtime OS

That idea eventually became UGR.

Instead of replacing inference engines like llama.cpp, UGR sits above them as an orchestration layer.

Application
    │
OpenAI-Compatible API
    │
Planner
    │
Runtime Intelligence Engine
    │
Scheduler
    │
Virtual Resource Manager
    │
Adaptive Memory Fabric
    │
Driver Abstraction Layer
    │
llama.cpp / TensorRT / Vulkan

Enter fullscreen mode Exit fullscreen mode

The execution backend becomes interchangeable.

The runtime becomes responsible for deciding how execution should happen.

Core Ideas

Rather than managing a single GGUF file, UGR treats AI workloads as collections of resources.

That includes things like:

  • Tensors
  • KV cache
  • MoE experts
  • Embeddings
  • LoRA adapters
  • Execution state

These resources can move between different memory tiers depending on runtime conditions.

VRAM
  ↓
RAM
  ↓
NVMe
  ↓
Remote Storage
  ↓
Archive

Enter fullscreen mode Exit fullscreen mode

Instead of asking:

Can this model fit into VRAM?

The runtime asks:

How should these resources be orchestrated across available hardware?

Hardware Shouldn't Dictate Capability

One thing I kept running into was that modern AI software often assumes modern hardware.

Reality is different.

There are countless developers, researchers, universities, and small teams running inference on older GPUs or mixed hardware environments.

Rather than optimizing for a single GPU architecture, I wanted UGR to detect hardware capabilities and adapt its execution strategy accordingly.

Simulation Before Execution

One feature I'm particularly excited about is the simulation engine.

Instead of loading a model and hoping for the best:

ugr simulate gemma-4-31b.gguf

Enter fullscreen mode Exit fullscreen mode

The runtime predicts:

  • Expected memory usage
  • Estimated throughput
  • Bottlenecks
  • Scheduler choice
  • Whether the model is even runnable

...before execution begins.

Current Status

UGR is still very much a research project.

Currently implemented:

  • Multi-backend runtime
  • GGUF parser
  • Adaptive Memory Fabric
  • Virtual Resource Manager
  • OpenAI-compatible API
  • Hardware capability detection
  • CLI tools
  • Model registry

Still under active development:

  • Runtime Intelligence Engine
  • Simulation engine improvements
  • MoE expert paging integration
  • TensorRT backend
  • Vulkan backend

Why Open Source?

I decided to open source the project because I don't believe architectures improve in isolation.

I also don't have access to high-end hardware.

Most of my development has been on a machine with a 4 GB Quadro GPU, so I would love feedback from people running larger GPUs or heterogeneous systems.

If nothing else, I hope some of the ideas around resource virtualization and runtime orchestration spark interesting discussions.

GitHub

⭐ UGR (AI Runtime Operating System)

https://github.com/UGEM-io/UGR

I'd genuinely appreciate feedback, bug reports, architectural criticism, or ideas for improving the runtime.

Final thoughts

When I started this project, I thought I was trying to solve a memory problem.

Today I think I'm trying to solve a runtime orchestration problem.

Whether UGR ultimately becomes a useful research platform or something more, I'm excited to see where the community takes it.