Chanyeong Yun

I was testing a RAG ingestion pipeline and noticed something painful: ingesting a single document took nearly 50 seconds.

Checking the metrics revealed that the CPU was doing almost nothing. The app wasn't compute-bound — it was simply sitting idle, waiting for sequential HTTP network round-trips to return.

By converting the blocking embedding module into an asynchronous workflow, processing time dropped from 49.61 seconds to 1.56 seconds with zero infrastructure changes.

Benchmark Setup

  • Model: Amazon Titan Text Embeddings V2 (AWS Bedrock)
  • Dataset: 33 text chunks from a single document
  • Region: us-east-1
  • Test: Sequential blocking requests vs. concurrent asynchronous requests

Results

Approach Time Difference
Sequential (requests blocking loop) 49.61s Baseline
Concurrent (aiohttp + asyncio.gather) 1.56s 31.8× faster

Why the Gap Was So Massive

Sequential (Blocking I/O)

Each request must complete before the next one starts.

chunk 1 ──> request ──> ⏳ wait (~1.5s) ──> response
chunk 2 ──> request ──> ⏳ wait (~1.5s) ──> response
chunk 3 ──> request ──> ⏳ wait (~1.5s) ──> response
...
Total Time ≈ (33 chunks × ~1.5s) ≈ 49.6s

Enter fullscreen mode Exit fullscreen mode

Since network latency dominates the runtime, the event loop sits completely unused.

Concurrent (Non-blocking Async)

Requests are dispatched concurrently; total time collapses down to the duration of the slowest single request.

chunk 1  ──> request ──────────────> response
chunk 2  ──> request ──────────────> response
chunk 3  ──> request ──────────────> response
...        (in flight concurrently)
Total Time ≈ slowest single request ≈ 1.56s

Enter fullscreen mode Exit fullscreen mode

The Code Change

Before: Sequential Execution 

import requests

def generate_embeddings(texts: list[str]) -> list[list[float]]:
    all_embeddings = []
    for text in texts:
        response = requests.post(url, headers=headers, json={"inputText": text})
        vector = response.json().get("embedding", [])
        all_embeddings.append(vector)
    return all_embeddings

Enter fullscreen mode Exit fullscreen mode

After: Concurrent Execution 

import asyncio
import aiohttp

async def fetch_embedding(session: aiohttp.ClientSession, text: str) -> list[float]:
    async with session.post(url, headers=headers, json={"inputText": text}) as res:
        data = await res.json()
        return data.get("embedding", [])

async def generate_embeddings_async(texts: list[str]) -> list[list[float]]:
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_embedding(session, text) for text in texts]
        return await asyncio.gather(*tasks)

Enter fullscreen mode Exit fullscreen mode

Note: If you're using boto3 instead of raw HTTP requests, look at aioboto3, or offload the blocking SDK calls with asyncio.to_thread() to get similar non-blocking behaviour.

How It Scales

Chunks Sequential (est.) Concurrent (est.)
31 49.61s 1.56s
100 ~160s (2.7 min) ~2–3s
500 ~800s (13 min) ~3–5s
1,000 ~1,600s (26 min) ~5–10s

The more chunks you ingest, the more the concurrent approach pays off — batch ingestion jobs that took half an hour now finish in seconds.

Key Takeaways

  • Check for idle waiting first. Before scaling up infrastructure or adding complex worker queues, verify whether your pipeline is simply blocked on I/O.
  • Mind the service quotas. Bedrock has per-region request-rate limits — as chunk counts grow into the thousands, cap concurrency (e.g. asyncio.Semaphore) before you hit them.
  • Low effort, high impact. Changing just one file (embedder.py) removed the embedding stage as a bottleneck in the pipeline.