Before you write a scraper for a company's careers page, ask a cheaper question: is the data even reachable?

I checked 42 of the most sought-after employers in one region. Seven were reachable. The other thirty-five were not, and no amount of clever parsing would have changed that.

Here is how the split works, and how to test any company in a single request.

Two kinds of applicant tracking system

Companies don't type jobs into job boards. They enter each role once into an applicant tracking system, and that system pushes copies outward.

Modern ATS platforms publish an open job board API. Greenhouse, Lever, Ashby and SmartRecruiters all serve their job data as JSON to anyone who asks. No key, no login, no browser. They do this deliberately: companies want their roles discovered and syndicated.

Enterprise ATS platforms do not. Taleo, Oracle Cloud Recruiting, Phenom and iCIMS render jobs through session-bound, JavaScript-heavy portals with no public feed. Some of them will happily block you for trying.

That distinction predicts almost everything about whether scraping a given company is going to be pleasant or miserable.

The result: 7 of 42

The 35 unreachable companies weren't small. They included the region's largest airline, its national oil company, its biggest property developers, its major banks, and its best-known retail groups. Household names, all of them running enterprise portals with no public feed.

The seven reachable ones had something in common. They were the newer, faster-moving companies: a fintech, a delivery platform, a real-estate developer that recently modernised, a clean-energy firm, a booking platform. Companies that adopted their ATS in the last several years, when Greenhouse and Lever were the default choice.

This is not a knock on the larger companies. Enterprise HR systems solve problems that a startup doesn't have. It just means their job data isn't available at the source, which is why aggregators show incomplete or stale listings for exactly the employers people search for most.

Check any company in one request

Fetch the careers page and look for the ATS signature in the HTML:

import re
import requests

SIGNATURES = [
    ("greenhouse",      r"boards\.greenhouse\.io/([\w-]+)"),
    ("lever",           r"jobs\.lever\.co/([\w-]+)"),
    ("ashby",           r"jobs\.ashbyhq\.com/([\w.-]+)"),
    ("smartrecruiters", r"careers\.smartrecruiters\.com/([\w-]+)"),
    ("workday",         r"([\w-]+)\.(wd\d+)\.myworkdayjobs\.com/([\w-]+)"),
    # Present but not scrapeable — worth naming so you stop early:
    ("taleo",           r"taleo\.net"),
    ("oracle",          r"oraclecloud\.com|/hcmUI/CandidateExperience"),
    ("phenom",          r"phenompeople"),
    ("icims",           r"icims\.com"),
]

def detect(careers_url):
    html = requests.get(careers_url, timeout=20).text
    for ats, pattern in SIGNATURES:
        m = re.search(pattern, html)
        if m:
            return ats, m.groups()
    return None, None

Enter fullscreen mode Exit fullscreen mode

If it comes back taleo, oracle, phenom or icims, stop. You are looking at days of brittle browser automation for data that will break next quarter. Spend the time on the companies you can read properly, and get the rest from job boards.

Then read the jobs

Once you know the platform, it's one call:

# Greenhouse
requests.get("https://boards-api.greenhouse.io/v1/boards/stripe/jobs?content=true").json()["jobs"]

# Lever
requests.get("https://api.lever.co/v0/postings/spotify?mode=json").json()

# SmartRecruiters (paginated, 100 per page)
requests.get("https://api.smartrecruiters.com/v1/companies/Visa/postings?limit=100").json()["content"]

Enter fullscreen mode Exit fullscreen mode

Ashby is an unauthenticated GraphQL POST. Workday is a POST too, and it has one trap that costs everybody a day: limit cannot exceed 20. Ask for 100 and it returns an empty array rather than an error, which looks exactly like "no more results." It also throttles fast paging, so if your loop treats a failed page as the end of the list, you'll silently collect a fraction of the jobs.

Don't guess the slug

The board token in those URLs is not the company name. It's whatever the company typed when they set up the account. Read it off the real careers URL rather than guessing.

I learned this the tedious way: guessing slugs across those 42 employers had roughly a 10% hit rate, and two of the "hits" were the wrong company entirely. One slug belonged to a New York firm that happened to share a name fragment with the company I wanted. Another belonged to an education company rather than the marketplace with a similar name.

Always print the locations of the jobs you get back before you trust that you found the right board.

The no-code version

I packaged all five platforms into a free scraper: it auto-detects the ATS, normalizes the payloads into one schema, and adds seniority and function detection. Free, because these APIs need no proxies and cost nothing to call.

All my scrapers are here — the ATS one is free to run.

The code is open source under MIT if you'd rather read it than run it.

Be reasonable about it

These endpoints are public because companies want their jobs found. Cache your results, pause between requests, and don't hammer a small company's board on a cron. The reason all of this works without keys is that nobody has yet had a reason to lock it down.