How we tackled 精准翻译与语言识别 (precise translation and language identification) for AI-powered book translation.

The Problem: Garbage In, Garbage Out

When we first launched LectuLibre, our AI book translation service, we thought the hardest part would be fine-tuning LLM prompts for literary quality. But we quickly discovered a more fundamental hurdle: if the source language of an uploaded book is misidentified, no amount of prompt engineering can salvage the translation.

Users upload EPUBs and PDFs from all over the world. Some contain metadata specifying the language, but many don't. Others are multilingual books, or have prefaces in a different language. Our initial language detection using Python's langdetect library was correct only about 85% of the time on real-world uploads. That 15% error rate meant entirely garbled translations, frustrated users, and wasted LLM API credits.

We needed something far more robust—what we internally call 精准翻译与语言识别 (precise translation and language identification). Here’s how we built it.

The Language Detection Pipeline: From 85% to 98% Accuracy

Our first instinct was to try heavier models like fastText's pre-trained language identification model, which is known for high accuracy. But when we tested it on book excerpts, we hit a new problem: short paragraphs or dialogues in one language embedded in a book of another language (e.g., French phrases in an English novel) would throw off chunk-level detection.

We realized that we needed a two-tier approach: book-level language detection with confidence scoring, and per-chunk verification before translation.

Combining Multiple Detectors with Voting

We created a LanguageDetector class that runs several detectors and picks the majority vote, with a fallback to user-specified language when available. The detectors we use are:

  • fastText with the official lid.176.bin model (loaded once, not per request)
  • langdetect, which is lightweight and good for long texts
  • cld3 (Compact Language Detector 3) from Google, which works well on short snippets
  • A custom fine-tuned fastText model on 10,000 book excerpts we curated from multilingual EPUBs (we'll open source this soon)

For a given text, we take the top-1 prediction from each detector, and if at least three of them agree, we trust that result. If there's a tie, we fall back to the one with the highest overall confidence (from the model's probability). If the user explicitly set a source language during upload, we honor that, but we still run detection and warn if a mismatch is strong.

Here's a simplified version of our detector:

import fasttext
import langdetect
import cld3

class LanguageDetector:
    def __init__(self):
        self.ft_model = fasttext.load_model('lid.176.bin')

    def detect(self, text, user_lang=None):
        # Clean text: replace newlines, keep only first 10000 chars
        clean = text.strip().replace('\n', ' ')[:10000]
        votes = []
        # fastText
        pred_ft = self.ft_model.predict(clean, k=1)
        lang_ft = pred_ft[0][0].replace('__label__', '')
        conf_ft = pred_ft[1][0]
        votes.append((lang_ft, conf_ft))
        # langdetect
        try:
            lang_ld = langdetect.detect(clean)
            votes.append((lang_ld, 1.0))  # langdetect doesn't give confidence
        except Exception:
            votes.append(('unknown', 0.0))
        # cld3
        pred_cld = cld3.get_language(clean)
        if pred_cld.is_reliable:
            votes.append((pred_cld.language, pred_cld.probability))
        else:
            votes.append(('unknown', 0.0))

        # Voting: count occurrences, pick max with tie-break on confidence
        lang_counts = {}
        for lang, conf in votes:
            lang_counts[lang] = lang_counts.get(lang, 0) + 1
        majority_lang = max(lang_counts, key=lambda l: (lang_counts[l], 
                            sum(c for v_l, c in votes if v_l == l)))
        if majority_lang != 'unknown' and lang_counts[majority_lang] >= 3:
            # High confidence, possibly override user_lang with warning
            if user_lang and user_lang != majority_lang:
                # log warning, but use detection
                return majority_lang
            return majority_lang
        # If no consensus, fallback to user setting or the highest confidence from all
        if user_lang:
            return user_lang
        # Pick the highest confidence individually
        best = max(votes, key=lambda x: x[1])
        return best[0]

Enter fullscreen mode Exit fullscreen mode

In production, we load the fastText model at startup to avoid per-request overhead. We also cache detection results per book (based on a hash of the first 10k characters plus random sampling) to speed up subsequent operations.

Handling Multilingual Books

For books that contain multiple languages (e.g., a language textbook with side-by-side translation), we don't require detection at the book level. Instead, we first try to identify the dominant language (the one with >80% of the pages). If the user requests translation, we only translate chunks that match the source language and leave the rest untouched. This requires per-chunk language identification. Our pipeline runs detection on each chunk (around 1000 tokens) before sending it to the LLM, and skips chunks that are clearly not the target language. That prevents the LLM from halting on foreign phrases.

Chunking for LLMs: Preserving Context Without Breaking the Bank

Large language models are great at translation, but they have token limits and cost per token. Books are long, and sending them whole would bust context windows and API budgets. We needed a chunking strategy that keeps enough context for accurate translation while minimizing token waste.

Dynamic Chunking with Overlap

We use TikToken (OpenAI's tokenizer) to count tokens (also works for Claude's tokenizer which is similar). We set a maximum chunk size of 3,500 tokens (to leave headroom for the system prompt and response). We also maintain a 200-token overlap with the previous chunk to provide context continuity. This is crucial for sentences that span paragraph boundaries.

import tiktoken

def chunk_text(text, max_tokens=3500, overlap=200):
    enc = tiktoken.get_encoding("cl100k_base")
    tokens = enc.encode(text)
    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + max_tokens, len(tokens))
        chunk = tokens[start:end]
        # if we're not at the beginning, prepend overlap from previous chunk
        if start > 0:
            overlap_start = max(0, start - overlap)
            prepend = tokens[overlap_start:start]
            chunk = prepend + chunk
        chunks.append(enc.decode(chunk))
        start = end
    return chunks

Enter fullscreen mode Exit fullscreen mode

We then pass each chunk to the LLM with a system prompt that includes the source and target languages, and instructions to preserve formatting (markdown, special characters). We use Anthropic's Claude 3 Haiku for cost efficiency, but fall back to Claude Sonnet for complex passages when the model returns a confidence score below a threshold.

Async Translation Pipeline with Rate Limiting

Translating a full book might require 50+ chunks. We run these in parallel using asyncio, but we must respect API rate limits (for Anthropic, it's 5 requests per second on our tier). We implemented a token bucket rate limiter and semaphore:

import asyncio
from aiolimiter import AsyncLimiter

rate_limiter = AsyncLimiter(5, 1)  # 5 requests per 1 second
sem = asyncio.Semaphore(10)  # max 10 in-flight

async def translate_chunk(chunk, src_lang, dst_lang):
    async with sem:
        async with rate_limiter:
            # Call Claude API
            ...

Enter fullscreen mode Exit fullscreen mode

We also batch large books into groups of chunks and process them with a small number of workers. This keeps the API usage smooth and prevents 429 errors.

Quality Control and Fallback

Even with good language detection and chunking, LLMs sometimes produce translations that are literal and miss literary nuance. For academic books, that might be fine, but for novels, we needed a way to improve fluency. We added a post-processing step where we run each translated chunk through a second LLM call (a smaller model like Claude Haiku with a higher temperature) to "polish" the language only if the user selected the "literary" translation mode. This adds cost but significantly improves readability. We found that around 30% of users opt for literary mode.

Results and Lessons Learned

After deploying the multi-model language detector, our language identification accuracy on a test set of 5,000 book excerpts jumped from 85% to 98.2%. Misidentifications dropped to rare edge cases (e.g., very short texts in dialect). In production, false language detection errors fell to near zero, and user complaints about garbled translations ceased.

The chunking strategy with overlap reduced API retries due to incomplete sentences by 40%, and the literary post-polish improved average user readability ratings (from A/B tests) by 15% on fiction books.

We also saved about 20% on API costs because the better language detection meant we weren't wasting credits on incorrect translations, and the dynamic chunking allowed us to pack more text per API call.

Key Takeaways for Developers

  • Don't trust a single detector; ensemble methods are far more reliable for language ID.
  • Always run per-chunk verification before translating to catch mixed-language content.
  • Overlap is essential when chunking for LLM translations—200 tokens of context made a big difference in our tests.
  • Rate limiting and async are your friends; they turn a fragile pipeline into a robust one.

Open Questions

We're still exploring: How to handle ancient languages or highly specialized jargon? Is there a way to dynamically adjust chunk size based on sentence boundaries? We'd love to hear from the community—especially anyone who's tackled translation of technical manuals or poetry.

LectuLibre's translation pipeline continues to evolve, and we're planning to open-source our language detection ensemble and curated dataset. If you're building something similar, reach out!