Your AI agent framework probably isn't your security problem (7,020 trials say so)

If you've picked LangChain over CrewAI (or vice versa) partly for "security reasons," this preprint is worth five minutes.

I ran a controlled evaluation — 6 LLMs, 6 agent execution conditions, 5 attack families, 7,020 payload-verified trials — to isolate what actually explains security outcomes in agentic AI systems. The headline: framework choice explains about 0.06% of the variance (not statistically significant, p ≈ 0.70). Attack family explains ~29%. Model explains ~4%.

In plain terms: which framework you build on barely matters. What attack you're facing matters a lot. If your threat model assumes "framework X is inherently safer," this data doesn't back that up.

This is one of four connected preprints I've published on agentic AI security evaluation methodology — the other three dig into a more foundational problem: how reliable are the automated detectors (refusal classifiers, prompt-injection classifiers) that most red-teaming pipelines use to generate labels in the first place? Short answer: less reliable than commonly assumed, and I show exactly where they break.

All four preprints are open (CC BY 4.0) and the evaluation code is open-source:

Preprints: https://figshare.com/authors/Waqar_Javed/24479225
Eval framework GitHub: https://github.com/AgentSafeLabs/safelabs-eval
Eval framework Website: https://agentsafelabs.com/