I Built a Job-Scraping Bot to Escape Cloud Consulting (And It's Open Source)

I Built a Job-Scraping Bot to Escape Cloud Consulting (And It's Open Source) From manually...

Dev.to · Jul 22, 2026

I Built a Job-Scraping Bot to Escape Cloud Consulting (And It's Open Source)

From manually copy-pasting cover letters to a Python pipeline that hunts Dutch tech roles while I sleep — the architecture, the failures, and why I need your help.

The Setup

I had applied to 73 jobs in 4 weeks.

Not casual browsing. I mean:

Custom cover letters per role
CV tweaks for Platform Eng vs SRE vs Cloud Security
LinkedIn cold messages to hiring managers
Tracking everything in a Notion database

The result? Ghosted by 61. Rejected by 8. Phone screen with 4.

Meanwhile, I was working full-time at Siemens, studying Go for kube-radar, and maintaining a homelab that was literally on fire one Tuesday because a Cilium policy update bricked the cluster network. I was burning out on the process of job searching instead of actually getting better at it.

So I did what any engineer would do: I automated the pain away.

That's how job-digest started — a Python pipeline that scrapes job boards, filters by my exact criteria, ranks roles by match quality, and delivers a daily summary to my Telegram. It didn't get me hired overnight. But it got me interviewed by companies I actually wanted to work for — and it taught me more about web scraping, async Python, and data pipelines than any course.

This is the story of building it. The code is public. I want collaborators. If you're job-hunting in tech and you know Python, read on.

Why This Problem?

Cloud consulting taught me how to build systems. It did not teach me how to sell myself.

When I decided to relocate to Amsterdam 🇳🇱 and target backend/platform roles, I faced three problems no recruiter warns you about:

1. The signal-to-noise ratio is brutal

Job boards are flooded with "Cloud Engineer" roles that are actually MSP helpdesk jobs, "DevOps" roles that mean "you're the only ops person for 40 devs," and "Kubernetes" roles at companies running Docker Compose on a single EC2 instance.

I needed filtering that understood my stack: Go, Kubernetes internals, Cilium/eBPF, Azure + AWS multi-cloud, security compliance (NIS2/DORA).

2. Timing asymmetry

The best roles at Adyen, Booking.com, ASML — they go live on Tuesday morning and have 200 applicants by Thursday afternoon. I was checking boards manually on Sunday evenings. I was always late.

3. Context switching kills deep work

Every 20-minute "let me check LinkedIn" session destroyed my flow state. I wanted one daily digest at 7 AM. Read it on the train. Apply to the top 3 roles. Then close the tab and write Go code.

The Architecture

┌─────────────┐    ┌──────────────┐    ┌──────────────┐    ┌─────────────┐
│  Scrapers   │───►│   Enricher   │───►│    Ranker    │───►│   Notifier  │
│ (async)     │    │ (LLM + rules)│    │ (scoring)    │    │ (Telegram)  │
└─────────────┘    └──────────────┘    └──────────────┘    └─────────────┘
     │                                              │
     ▼                                              ▼
 LinkedIn Jobs                              SQLite cache
 Indeed                                     (dedup + history)
 Glassdoor
 We Work Remotely
 RemoteOK

Enter fullscreen mode Exit fullscreen mode

Design Decisions

Decision	Chose	Over	Because
Language	Python 3.11 + asyncio	Go	Rapid prototyping, BeautifulSoup/httpx ecosystem, LLM SDKs
Storage	SQLite	PostgreSQL	Zero infra, portable, single file. WAL mode for concurrent reads.
Scraping	httpx + BeautifulSoup	Selenium/Playwright	Faster, lighter, cheaper. JavaScript-rendered boards get headless fallback.
Rate limiting	Adaptive backoff (exponential + jitter)	Fixed delays	Respects robots.txt without being slower than necessary.
LLM enrichment	OpenAI GPT-4o-mini	Local LLM	Cost: ~$0.02 per 100 jobs. Quality worth it for extraction.
Delivery	Telegram Bot API	Email	I live in Telegram. Push notification = instant awareness.
Scheduling	cron on homelab RPi	GitHub Actions	Free, persistent, no cold start. Runs at 06:00 CET daily.

The Pipeline (Simplified)

# 1. COLLECT — async scrapers, one per source
async def scrape_linkedin(query: str, location: str) -> list[RawJob]:
    # httpx + BeautifulSoup
    # Extract: title, company, location, description, salary, url, posted_at
    # Returns list of RawJob dataclasses

# 2. NORMALIZE — unify schema across sources
def normalize(raw: RawJob) -> Job:
    # Map "Amsterdam, North Holland" → "Amsterdam"
    # Extract seniority: Junior / Mid / Senior / Staff from title
    # Detect remote policy: onsite / hybrid / remote
    # Parse salary ranges from description text (regex + LLM fallback)

# 3. FILTER — hard rules (cheap, deterministic)
def passes_filter(job: Job) -> bool:
    return (
        job.location in TARGET_CITIES  # Amsterdam, Berlin, Dublin, etc.
        and job.seniority in {"Senior", "Staff", "Principal"}
        and any(kw in job.description.lower() for kw in TECH_KEYWORDS)
        and job.company not in BLOCKLIST  # agencies, body shops
    )

# 4. ENRICH — LLM judges fit quality
def enrich_with_llm(job: Job) -> EnrichedJob:
    prompt = f"""
    Role: {job.title} at {job.company}
    Description excerpt: {job.description[:2000]}

    Candidate profile: 6 years cloud/platform, K8s internals (Cilium, Talos),
    Go learner, Azure+AWS, security compliance (NIS2/DORA).
    Target: Platform Engineer or Backend Engineer in Amsterdam/Berlin/Dublin.

    Tasks:
    1. Is this role genuinely backend/platform work (not IT support, not MSP)?
    2. Does the tech stack mention K8s, Go, distributed systems, or cloud-native?
    3. Visa sponsorship likely? (big tech yes, small local shop probably no)
    4. Score 0-100 for fit.
    5. Is it worth applying? yes/no/maybe.
    """
    response = openai.chat.completions.create(...)
    return EnrichedJob(..., llm_score=response.score, verdict=response.verdict)

# 5. RANK — composite score
def score(job: EnrichedJob) -> float:
    return (
        job.llm_score * 0.5 +
        salary_score(job.salary_max) * 0.2 +
        company_tier_score(job.company) * 0.2 +  # Tier 1 = Adyen/Booking/ASML/etc.
        recency_score(job.posted_at) * 0.1        # Fresh postings score higher
    )

# 6. NOTIFY — formatted Telegram message
async def send_digest(jobs: list[EnrichedJob]):
    msg = format_telegram(jobs[:10])  # top 10 only
    await telegram_bot.send_message(chat_id=MY_CHAT, text=msg)

Enter fullscreen mode Exit fullscreen mode

The Telegram Output

🔍 Daily Job Digest — 2026-07-22
18 new roles | 4 match your filters | Top 3 below

━━━━━━━━━━━━━━━━━━━━
🥇 94 pts — Adyen — Platform Engineer
   📍 Amsterdam 🇳🇱  |  💰 €90-120K  |  🏠 Hybrid
   🛠 Go · K8s · Kafka · Terraform
   📝 "Build internal primitives for developer platforms..."
   ✅ LLM: "Strong match. Go-heavy platform team. Sponsors HSM."
   🔗 https://careers.adyen.com/jobs/...

🥈 88 pts — Booking.com — SRE
   📍 Amsterdam 🇳🇱  |  💰 €85-110K  |  🏠 Hybrid
   🛠 K8s · Prometheus · Python/Go · Cilium
   ✅ LLM: "Excellent infra depth match. Large K8s fleet."

🥉 81 pts — N26 — Platform Engineer
   📍 Berlin 🇩🇪  |  💰 €80-110K  |  🏠 Hybrid
   🛠 Go · K8s · Terraform · Microservices
   ✅ LLM: "Good fit. Fintech, EU Blue Card likely."

📊 Actions: /apply N26  |  /dismiss Adyen  |  /details Booking

Enter fullscreen mode Exit fullscreen mode

5 Failures Before It Worked

Failure 1: LinkedIn Rate-Limited Me in 6 Minutes

My first scraper used a single httpx session with a 2-second delay between requests. I was proud of the backoff. LinkedIn was not impressed.

They returned a 429 after ~30 requests and started serving CAPTCHA pages. I added:

Rotating User-Agent headers (from a pool of 50 real browser strings)
Proxy rotation via free proxy lists (terrible idea — 80% were dead)
Randomized delays (1-5 seconds with jitter)

The fix that actually worked: scrape LinkedIn's public job search JSON endpoint instead of HTML parsing. It returns structured data at /jobs-guest/jobs/api/jobs and is significantly more stable. I also dropped proxy rotation and just respected a 5-second delay. LinkedIn is aggressive but fair.

Lesson: Don't fight the frontend. Find the API the frontend uses.

Failure 2: The LLM Hallucinated a Role at Google That Didn't Exist

I blindly trusted GPT-4o-mini's extraction for the first week. It invented a "Kubernetes Platform Engineer" role at Google Amsterdam with a salary of €500K. The URL 404'd. I had already told my wife about it.

The fix: every scraped job gets a live URL validation (HEAD request, follow redirects). If the URL doesn't return 200, it's dropped before enrichment. LLMs are pattern matchers, not fact checkers.

Lesson: LLMs are enrichment, not validation. Never trust their facts without verification.

Failure 3: I Applied to the Same Job 4 Times

My SQLite schema initially keyed jobs by (title, company). Then a company reposted the same role with a slightly different title. I applied twice. The second time, the recruiter replied: "You already applied last week."

The fix: content hashing. I now hash the normalized job description (after removing whitespace and dates). Same description = same job, even if the title changes. Deduplication is fuzzy, not exact.

Lesson: Job boards are messy. Exact matching is naive.

Failure 4: Running It on My Laptop Meant It Died When I Closed the Lid

I initially ran the script manually: python main.py, wait 8 minutes, get results. But I kept forgetting. And when I closed my MacBook, the script died.

The fix: a headless Raspberry Pi 4 in my homelab runs a cron job at 06:00 CET. It has a persistent SQLite file on an external SSD. The Pi also runs my DNS (Pi-hole), my monitoring (Prometheus), and now my job search. One device, many duties.

Lesson: Automations that depend on you remembering to run them are not automations.

Failure 5: I Got 18 "Matches" and Felt Overwhelmed

Even with filtering and scoring, getting 18 "good" roles in one digest meant I applied to none. Decision paralysis. I would read all 18, open 12 tabs, and close the laptop exhausted.

The fix: the digest shows top 3 by default. To see 4-10, you reply /more. To see the rest, /all. This tiny UX change meant I actually applied to the top 3 instead of bookmarking 12.

Lesson: More data is not better. More decisions is worse.

Results (So Far)

I won't claim this got me hired. I'm still interviewing. But here's what changed:

Metric	Before	After
Time spent browsing jobs/day	45 min	5 min reading digest
Applications/week	~8 (sporadic)	~12 (consistent)
Roles matching my stack	~30% of browsed	~85% of digest
Time-to-apply after posting	3-4 days	<24 hours
Relevant recruiter conversations/month	1-2	4-5

The real win: mental bandwidth. I stopped treating job search as a thing I do when I'm tired at 10 PM. It became a system that runs without me.

Companies where the pipeline surfaced roles I wouldn't have found manually:

A stealth-mode fintech in Amsterdam hiring Go platform engineers (not on LinkedIn, found via We Work Remotely)
A CDN company in Berlin rebuilding their edge on Cilium + eBPF (mentioned in a blog post the scraper indexed)
A Series B startup in Dublin with ex-Google SREs building a K8s-native security product (literally my exact niche)

The Code

git clone https://github.com/beltagyy/job-digest.git
cd job-digest
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cp config.example.yaml config.yaml
# Edit: add your OpenAI key, Telegram bot token, target cities, tech keywords
python main.py --run-now

Enter fullscreen mode Exit fullscreen mode

What you get:

Modular scrapers (add a new source in ~50 lines)
SQLite cache with WAL mode
LLM enrichment with prompt templates
Telegram bot interface with simple commands
Docker support for headless deployment

What you need:

Python 3.11+, an OpenAI API key (~$5/month at my volume)
A Telegram bot (free via @botfather)
A server that runs cron (RPi, VPS, old laptop)

I Need Collaborators

This project helps me, but it's built for my profile and my targets. To make it useful for others, I need help with:

1. More scrapers

AngelList / Wellfound (startup roles)
Hacker News "Who is Hiring?" (monthly thread parser)
Greenhouse / Workday ATS backends (many companies use these and have predictable JSON APIs)
Country-specific boards (StepStone.de, Indeed.nl, IrishJobs.ie)

2. Better filtering

Visually parse "years of experience" requirements and flag "possibly underqualified" vs "definitely qualified"
Detect visa sponsorship mentions from description text (pattern-based, not just LLM)
Filter out roles from recruiting agencies vs direct employers

3. Smarter enrichment

Local LLM option (Ollama/Llama 3.1) for people who don't want OpenAI costs
Company research: pull funding stage, Glassdoor rating, team size from public APIs
Salary benchmarking: compare offered range to levels.fyi / Glassdoor data

4. Alternative frontends

Discord bot (many communities live there)
Email digest (for non-Telegram users)
Web dashboard (Streamlit or similar) for browsing history and stats

5. Testing & reliability

Scrapers break when job boards change HTML. We need snapshot tests (save HTML, assert extraction still works).
Mock LLM responses for unit tests (expensive to hit OpenAI in CI).

If any of this sounds like you — especially if you're job-hunting in EU tech and want a tool that actually understands your stack — open an issue or PR.

Even if you're not technical: try it, tell me what job board you want added, and I'll prioritize it.

3 Lessons That Transfer Beyond Job Searching

1. Automate the boring, not the important

I automated discovery and filtering. I did not automate applying or interviewing. The bot finds roles. I still write every cover letter by hand and research every company before applying. The pipeline handles noise; I handle signal.

2. Your side project's tech stack should teach you something

I chose Python + asyncio because I already knew Python but had never built a production async pipeline. I learned:

asyncio.gather for parallel scrapers
SQLite WAL mode and connection pooling
Prompt engineering for structured LLM output (JSON mode, function calling)
Rate limiting and backpressure

If your side project doesn't stretch you, it's just work you don't get paid for.

3. Ship before it's perfect

The first version didn't have LLM enrichment. It was just a list of job titles and URLs sorted by posting date. It was ugly. It still saved me 30 minutes a day.

I added scoring later. Then filtering. Then the Telegram bot. Each increment made it better, but the core value (stop manually browsing job boards) existed on day 1.

What's Next

Feature	Status
Core pipeline (scrape → filter → notify)	✅ shipped
LLM enrichment	✅ shipped
SQLite dedup + history	✅ shipped
More scrapers (HN, Wellfound, StepStone)	🔄 in progress
Company enrichment (funding, Glassdoor)	📋 planned
Web dashboard	📋 planned
Resume-to-job matching score	📋 planned

**
🚀 Try it:** https://github.com/beltagyy/job-digest
**
🤝 Collaborate:** Open an issue with the tag help-wanted or DM me

**

If you're job-hunting right now — what's the most frustrating part? Drop it in the comments. If it's common enough, I'll add it to the pipeline.

TL;DR

I applied to 73 jobs in 4 weeks, got exhausted by the process, and built a Python bot (job-digest) to automate job-board scraping, filtering, and ranking.
It's a pipeline: async scrapers → LLM enrichment → scoring → daily Telegram digest. Runs on a Raspberry Pi at 6 AM.
It cut my daily job-browsing time from 45 minutes to 5, and surfaced roles at companies I actually wanted (Adyen, Booking, N26) that I would have missed manually.
Open source and looking for collaborators: github.com/beltagyy/job-digest. I need help with scrapers, filtering rules, local-LLM support, and a web dashboard.

Read the original source

#Backend

I Built a Job-Scraping Bot to Escape Cloud Consulting (And It's Open Source)

I Built a Job-Scraping Bot to Escape Cloud Consulting (And It's Open Source)

The Setup

Why This Problem?

The Architecture

Design Decisions

The Pipeline (Simplified)

The Telegram Output

5 Failures Before It Worked

Failure 1: LinkedIn Rate-Limited Me in 6 Minutes

Failure 2: The LLM Hallucinated a Role at Google That Didn't Exist

Failure 3: I Applied to the Same Job 4 Times

Failure 4: Running It on My Laptop Meant It Died When I Closed the Lid

Failure 5: I Got 18 "Matches" and Felt Overwhelmed

Results (So Far)

The Code

I Need Collaborators

3 Lessons That Transfer Beyond Job Searching

1. Automate the boring, not the important

2. Your side project's tech stack should teach you something

3. Ship before it's perfect

What's Next

**

TL;DR

0 Comments