[Submitted on 8 Apr 2026 (v1), last revised 27 Jul 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:Agent skills combine natural-language instructions with executable code while inheriting an agent's filesystem, credential, and network access. Attacks can span prose and files, whereas regex and code-only analyzers cover only one modality. SkillSieve applies three progressively deeper layers: recall-oriented regex, AST, and metadata triage; four parallel LLM security sub-tasks; and an independent three-model jury with debate on disagreement.
We evaluate 49,592 real ClawHub skills, a 390-skill labeled benchmark, and 100 adversarial samples across five evasion techniques on a 440 USD ARM board. The full pipeline achieves F1 = 0.929 (precision 0.912, recall 0.945) at an average cost of $0.006 per skill. An optional XGBoost fast path reduces Layer-2/3 calls by 32% with a 1.7-point F1 decrease while preserving recall. On 52 Feishu/Lark packages, Layer 2 reclassifies 13 of 14 Layer-1 flags as safe after contextual analysis; we also deploy the system as a Feishu chat bot. Code, labels, and aggregate results are open-sourced.

Submission history

From: Zongyou Yang [view email]
[v1] Wed, 8 Apr 2026 00:58:48 UTC (94 KB)
[v2] Tue, 26 May 2026 15:22:47 UTC (179 KB)
[v3] Mon, 27 Jul 2026 11:35:54 UTC (199 KB)