Building 60 MCP Servers Using the Five Elements Framework: A Build-in-Public Journey

Most people focus on adding more features to their AI agents. I discovered better evaluation loops matter more.

Over the past several months, my team and I have been building tancoai.com — a platform hosting 60+ MCP (Model Context Protocol) servers and Skills. What started as "let's build a few useful AI tools" turned into a deep exploration of what makes a Skill actually reliable in production.

Along the way, we developed something we call the Five Elements Framework (连珠五环). It's not a library or a package — it's a way of thinking about Skill design that forced us to stop adding features blindly and start measuring whether what we built actually worked.

This article is an honest account of what happened when we applied this framework across 60+ servers. Including the parts where it didn't go well.

The Problem We Faced

When we started, the approach was straightforward: identify a useful task, build a Skill for it, test it a few times, ship it. Repeat.

This worked for the first 10 or so Skills. They passed basic tests. They returned plausible-looking outputs. But something felt wrong.

We call it the "works but feels wrong" syndrome. A Skill would pass our manual test cases — we'd type a prompt, get a reasonable response, and mark it done. But when we looked at real usage logs, the picture was different:

  • Trigger failures: Skills activated on prompts they shouldn't have, or failed to activate when they should.
  • Context gaps: Skills retrieved irrelevant information or missed the context that actually mattered.
  • Format drift: The same Skill would return JSON 80% of the time and prose 20% of the time, breaking downstream consumers.
  • No feedback loop: We had no way to know if a Skill was getting better or worse over time.

The breaking point came when we analyzed 100+ failed Skill implementations. Not crashes — "successful" executions that produced wrong or unhelpful results. The patterns were striking:

Failure Type Occurrence
Trigger mismatches (Skill ran on wrong input type) 67%
Retrieval problems (relevant context existed but wasn't surfaced) 54%
Reasoning errors (logic sound in isolation, failed on edge cases) 43%
Output format issues (structure inconsistent across runs) 31%
Any feedback mechanism at all 0%

That last number was the wake-up call. We were building Skills with no way to know if they were getting better.

Introducing the Five Elements Framework

We didn't set out to create a "framework." We set out to stop making the same mistakes. But after a while, the patterns we kept checking condensed into five elements:

1. Trigger — Does it activate at the right moment?

Before anything else, a Skill needs to know when to run. This sounds trivial, but it's where most silent failures begin. A fact-checking Skill that activates on every question containing a URL is too aggressive. One that only activates on explicit "fact-check this" commands is too passive.

We test triggers with a corpus of "should fire" and "should not fire" inputs and measure precision and recall separately. A Skill isn't allowed to ship until both exceed 90%.

2. Retrieval — Is context found accurately?

Most Skills need external context — API data, documents, previous conversation history. The question isn't "can it retrieve?" but "does it retrieve the right things?"

We measure this with a hit-rate metric: for a set of test cases with known-relevant context, what percentage of the time does the Skill surface that context? Before the framework, we didn't track this at all. Turns out, several of our Skills were confidently answering questions using completely irrelevant source material.

3. Reasoning — Is decision logic sound?

This is the hardest element to test because "reasoning" is fuzzy. Our approach: for each Skill, we define 10-20 test cases with known-correct outcomes and measure pass rate. We also include adversarial cases — inputs designed to trigger common reasoning errors.

The key insight: reasoning quality isn't about the model being smarter. It's about the prompt, the context, and the constraints working together. A well-structured prompt with the right context can make a weaker model outperform a stronger one with poor structure.

4. Output — Is response format reliable?

This was our most embarrassing finding. We assumed our Skills returned consistent formats. They didn't. JSON fields would be missing, types would shift between string and number, markdown would sometimes include code fences and sometimes not.

Now, every Skill has an output schema test. We run the Skill 100 times on varied inputs and check that 100% of outputs conform to the declared schema. If even one run fails, the Skill doesn't ship.

5. Feedback — Can we measure improvement over time?

This is the element that ties everything together. Without feedback, the other four are just snapshots. With it, they become a trend.

Our feedback system tracks:

  • Per-Skill accuracy over time (test case pass rate by version)
  • Real-world usage patterns (which inputs are most common, where do users retry)
  • Failure clustering (grouping failures by root cause element)

When we deploy a new version of a Skill, we can see whether Trigger accuracy went up or down, whether Retrieval improved, whether Output consistency held. This is the difference between "I think it's better" and "it's measurably better on dimension X."

How They Work Together

The five elements aren't a checklist — they're a system. A Trigger failure means the Skill never gets a chance to Retrieve. A Retrieval failure means Reasoning operates on wrong data. An Output failure means Feedback can't be collected properly (because downstream systems break). Feedback informs improvements to all four.

The framework's value isn't in any single element. It's in the discipline of checking all five before shipping.

Building 60 Servers: What Actually Happened

Here's an honest timeline:

Weeks 1-4: Built the first 10 Skills using trial-and-error. No framework. No eval. Just vibes. About 6 of those 10 Skills had serious issues that we only discovered weeks later.

Weeks 5-8: The analysis. We stopped building and started looking at what was broken. This is when we analyzed the 100+ failed implementations and started forming the framework. Productive? Not in terms of new Skills. But it changed everything that came after.

Weeks 9-16: Rebuilt with the framework. We went back through the first 10 Skills and applied all five elements. This was painful — several Skills needed fundamental redesigns, not tweaks.

Weeks 17-24: Scaled to 60+ Skills. With the framework in place, building new Skills became faster. Not because the framework writes code, but because it tells you when you're done. Each element has a pass/fail threshold. You're not done when it "looks good." You're done when Trigger >= 90%, Retrieval >= 85%, Reasoning >= 90%, Output = 100%, and Feedback is wired up.

Key Failure Point: The Fact-Checking Skill (辩真)

Our fact-checking Skill initially had 70% accuracy on our test suite. The problem wasn't reasoning — it was Retrieval. The Skill was pulling context from the wrong sources. After restructuring the retrieval pipeline (and adding the retrieval hit-rate metric), accuracy went to 92%. Same model, same prompt structure. Just better context.

# Before: retrieval was implicit — whatever comes back, we use
def fact_check(claim):
    context = search_web(claim)
    return llm.verify(claim, context)

# After: retrieval is tested, filtered, and honest about gaps
def fact_check(claim):
    candidates = search_web(claim, top_k=10)
    relevant = filter_by_relevance(candidates, claim, threshold=0.7)
    if len(relevant) < 2:
        return {"status": "insufficient_sources", "claim": claim}
    return llm.verify(claim, relevant)

Enter fullscreen mode Exit fullscreen mode

That threshold=0.7 and the len(relevant) < 2 check came directly from the Retrieval element analysis. Before, the Skill would confidently verify claims using a single low-relevance source. After, it would honestly report when it couldn't find enough relevant context.

The Honest Struggles

  • Bots vs. real traffic. Early on, our usage data was dominated by crawlers and automated tools. It took weeks to separate real usage signals from noise. We eventually added client-side behavioral signals to filter this.
  • Documentation debt. Building 60 Skills means 60 sets of docs. We fell behind badly. The eval reports partially solve this — they serve as living documentation of what each Skill actually does and how well.
  • Maintenance burden. 60 Skills don't maintain themselves. When an upstream API changes, 3-4 Skills might break simultaneously. The Feedback element helps catch this faster, but it's still a constant effort.
  • The temptation to skip. Around Skill #45, we got tired. The framework felt like overhead. We shipped two Skills without full eval. Both came back with issues within a week. The framework exists because discipline is hard to maintain manually.

Lessons Learned

What surprised us most: The biggest accuracy improvements didn't come from better models or better prompts. They came from better Trigger and Retrieval — the elements before reasoning. We spent so much energy optimizing prompts that we missed the fact that our Skills were running on wrong inputs and retrieving wrong context.

Advice for builders:

  • Start with Feedback. Even if you can only measure one thing, measure it consistently. You can't improve what you can't track.
  • Don't build a "framework" on day one. Build 5-10 things, notice your patterns, then formalize.
  • Output consistency matters more than output quality. A Skill that always returns valid JSON with 85% accuracy is more useful than one that returns brilliant prose 95% of the time but breaks your pipeline 5% of the time.

When to use this framework vs. keep it simple: If you're building 1-3 Skills, ad-hoc testing is fine. If you're building 10+, the lack of structure will catch up with you. If you're building 50+, you need this or something like it — there's no way to hold 50 Skills' quality in your head.

What's Next

We're continuing to build in public. The eval reports for all Skills are available on tancoai.com, showing both strengths and weaknesses — no spin. When a Skill scores poorly on an element, it's visible. When we improve it, the trend is visible too.

The framework itself is open source. We're not claiming it's the only way to think about Skill design — but it's the way that worked for us after 60+ servers worth of trial and error.

GitHub: https://github.com/tancoai/lianzhu-skill

We welcome issues, discussions, and contributions. If you've built Skills and hit similar walls, we'd love to hear about your approach. Disagreement is especially welcome — the framework evolved through arguments, and it should keep evolving.

**Website with live demos and eval dashboards: [https://tancoai.com](https://ta