The same job is posted on three boards. Here's how to deduplicate job listings properly.

Search any regional job board for a role and you will meet the same position three times, under three slightly different titles, from three sources.

The company entered it once. The boards each reformatted it. If you are aggregating listings, you now have a deduplication problem, and it's harder than it looks: the naive fixes either merge two genuinely different jobs or fail to merge two copies of the same one.

Why exact matching fails

The same posting, as three boards render it:

E-Commerce Executive              | Example Trading Co | Dubai, United Arab Emirates
E-commerce Executive              | Example Trading Co | Dubai - UAE
E-Commerce Executive (FMCG)       | EXAMPLE TRADING CO | Dubai

Enter fullscreen mode Exit fullscreen mode

Nothing matches exactly. Case differs, the location string differs, one board appended a category, one uppercased the company. An exact-match key finds zero duplicates here and you publish the same job three times.

Why fuzzy matching also fails

The obvious next step is fuzzy string similarity on the title. Set a threshold, merge anything above it. This works until it meets real data:

Senior Accountant     vs  Senior Accountant     — same job? Maybe. Two different teams?
Sales Manager (Dubai) vs  Sales Manager (Abu Dhabi) — 92% similar, definitely different jobs
Data Analyst I        vs  Data Analyst II       — 95% similar, different seniority, different pay

Enter fullscreen mode Exit fullscreen mode

Titles are short. Short strings are similar to each other by accident. Any threshold loose enough to catch "E-Commerce Executive" and "E-commerce Executive (FMCG)" is also loose enough to merge two roles in different cities.

What actually works: a normalized composite key

Match on the pieces that identify a job, normalize each aggressively, and require all of them:

import re

def normalize(text):
    """Lowercase, strip everything that isn't alphanumeric."""
    return re.sub(r"[^a-z0-9]+", "", (text or "").lower())

def job_key(job):
    return "|".join([
        normalize(job["title"]),
        normalize(job.get("company") or job.get("location") or ""),
    ])

Enter fullscreen mode Exit fullscreen mode

E-Commerce Executive and E-commerce Executive both collapse to ecommerceexecutive. Company does the same. Sales Manager (Dubai) and Sales Manager (Abu Dhabi) stay apart because the parenthetical survives normalization.

Falling back to location when the company is missing matters more than it sounds: some boards omit the employer for confidential postings, and keying on title alone would merge every "Sales Manager" in the country into one row.

Merge, don't discard

When two records collapse to the same key, don't pick one and throw away the other. Different boards carry different fields. Take the union:

def merge(records):
    merged = {}
    for job in records:
        key = job_key(job)
        if key not in merged:
            rec = dict(job)
            rec["sources"] = [job["source"]]
            rec["allLinks"] = {job["source"]: job["url"]}
            merged[key] = rec
            continue

        rec = merged[key]
        rec["sources"] = sorted(set(rec["sources"] + [job["source"]]))
        rec["allLinks"][job["source"]] = job["url"]
        # Fill gaps from whichever source has the field
        for field in ("company", "location", "postedDate", "salaryText", "description"):
            if not rec.get(field) and job.get(field):
                rec[field] = job[field]
    return list(merged.values())

Enter fullscreen mode Exit fullscreen mode

One board gives you a salary range, another an accurate posted date, a third the full description. The merged record is better than any single source, and sources tells the reader where the job appears — genuinely useful, because applying through the board with the fewest applicants is a real strategy.

Keep the merge rate honest

The number worth watching is how many records collapsed. Publish it:

58 raw postings → 51 records (7 merged)

Enter fullscreen mode Exit fullscreen mode

If that ratio ever hits something implausible — 200 postings collapsing to 12 — your key is too loose and you are merging distinct jobs. If it never merges anything, your normalization isn't doing its job. In my own runs a cross-board merge rate around 10 to 15 percent looks right for a single keyword; much higher and I go read the rows.

Which is the general lesson. Print the rows behind any aggregate before you trust it. Every deduplication bug I've shipped looked perfectly reasonable as a percentage and obviously wrong as a list of titles.

Don't dedupe before you filter

Order matters. If you cap results per source before merging, you throw away the copy that had the salary field. If you filter by location before normalizing, you drop Dubai - UAE while keeping Dubai. Fetch fully, normalize, merge, then filter and cap.

The ready-made version

I maintain a scraper that searches the three largest regional job boards in one run and merges duplicates exactly this way, returning one record per role with every source it appears on. Pricing is per merged record, so cross-posted jobs are billed once rather than three times.

It's on my Apify page