Cover image for The Local AI Developer (Part 2): A Reality Check After Real-World Testing

David Vellé Abel

This is Part 2 of my series on building a Local AI Developer Stack. If you missed the initial setup guide, you can read it here but I warn you that it might be already obsolete: The Local AI Developer (Part 1).

It has been two months since I published that initial guide. The setup in Part 1 looked great and promising on day one, so I decided to use it daily to give it a test in real-world conditions.

Here is a reality check of what works, what doesn't, and the optimizations required to make it truly usable.

Reality Check

I have tried my setup with different agents, LLMs, and combinations, but to be honest, it was hard to make it work as expected right out of the box. These were the main struggles:

 Speed

This was an expected trade-off, but I was not expecting this level of slowness. I am talking about 4 to 5 tokens per second. To put that in context, this is equivalent to ~180 words per minute. An average human reader consumes between 200 and 300 words per minute. When the machine types slower than you can read, the developer experience suffers.

 Out of Memory Errors

It was naive to only calculate the RAM needed for the LLM weights. The context window, the sessions, the system prompts—everything wants its tiny bit of RAM, and they all have to share the machine with my IDE, browser, and background processes.

It was common to see the LLM crash with an Out of Memory (OOM) error after 20 minutes of deep thinking.

Quality

It was usual to see the LLM running in circles but to be honest, that was a configuration problem on my side: using the wrong LLM for the task or setting the wrong context window. You end up with an agent that has no memory and starts overthinking the same ideas without actually executing them.

Can we improve it?

To fix these issues, I had to change my approach from "maximum power" to "maximum efficiency." There are a couple of parameters you can play with:

 Select the Proper Model

I know we all want the biggest frontier model working for free on our machine, but we are running laptops, not mainframes, and we need to deal with their limits.

Smaller, more focused models exist. They have a significantly lower memory footprint, which leaves more space for context, sessions, and your actual daily apps. All models benefit from working with small chunks of code, and I have found smaller, specialized models to be "good enough" for small units of work.

Check Your Runtime

When I wrote Part 1, I started with Ollama. Since then, I found out there are a lot of runtimes and LLM servers, and they are all optimized in different ways.

Trying different runtimes to find the one that best fits your specific hardware can be a game changer. (I will delve into a highly optimized running setup specifically for my M4 Pro in a later post).

Context Management

I know this is a hot topic right now, but it really matters in local development. As I stated earlier, we cannot rely on humongous context windows like we do with cloud servers. We need to find the sweet spot between usability (so the agent remembers what it is doing) and memory usage.

Standard context optimization techniques are mandatory here: keep your sessions focused, compact your context, and don't let the agent read files it doesn't absolutely need.

Conclusion

The biggest lesson I learned over these last two months is that local AI development is not a drop-in replacement for the cloud; it is an entirely different discipline. You cannot throw infinite compute at a problem. Instead, you have to act like a systems engineer—balancing memory constraints, managing context hygiene, and matching the right model size to the right task. Once you embrace those constraints, building a completely private, autonomous local stack isn't just feasible; it is incredibly rewarding.