If you last touched domain lookups a few years ago, the thing you remember is
WHOIS: a plaintext protocol on port 43 that returned an unparseable wall of text
in a different shape from every registry.

It is effectively gone. RDAP replaced it, it is RFC 9083, it returns JSON, and
every gTLD registry is required to run one. This is one of the rare cases where
the standards process produced something strictly better and nobody noticed.

Here is what a lookup looks like now, what you can actually get out of it after
GDPR, and the one trap that had my own tool confidently reporting registered
domains as free to buy.

What RDAP gives you, and what it does not

Post-GDPR, RDAP returns no registrant personal data for virtually every TLD. If
you are looking for who owns a domain, stop: that information is not there and
no legitimate source has it.

What is there is the operational picture, and that turns out to be the more
useful half:

  • registrar, registration date, expiry date, status codes, DNSSEC
  • whether the name is locked against transfer

Combine that with a DNS lookup and you get the thing people actually want, which
is what a company runs on:

domain        registrar                expires       age mail       dns         dmarc
stripe.com    SafeNames Ltd.           2027-09-11   30.9 google     aws-route53 reject
linear.app    CloudFlare, Inc.         2030-05-09    8.2 google     cloudflare  reject
vercel.com    Amazon Registrar, Inc.   2029-10-04   26.8 google     vercel      quarantine
github.io     -                        -               - -          aws-route53 -
angel.co      -                        -               - google     cloudflare  reject
notion.so     -                        -               - -          cloudflare  quarantine

Enter fullscreen mode Exit fullscreen mode

That is a real run. Mail provider and DNS provider come from MX and NS records,
DMARC policy from a TXT lookup at _dmarc.<domain>. For sales qualification,
security auditing or competitive research, those three columns are worth more
than any registrant field ever was.

Note the bottom three rows have DNS but no registrar and no expiry. Hold that
thought.

The code 

import json
import os
import urllib.request

ACTOR = "glitchbound~domain-scraper"
API = "https://api.apify.com/v2"


def profile(domains):
    payload = {
        "domains": domains,
        "includeDns": True,
        "recordTypes": ["A", "MX", "NS", "TXT"],
        "includeEmailSecurity": True,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)
    # A domain that could not be looked up comes back as a row with an `error`
    # field rather than failing the run, and those are not charged.
    return [r for r in rows if not r.get("error")]


for r in profile(["stripe.com", "linear.app", "vercel.com"]):
    print(f'{r["domain"]:14} {r.get("mailProvider") or "-":10} '
          f'{r.get("dnsProvider") or "-":12} dmarc={r.get("dmarcPolicy") or "-"}')

Enter fullscreen mode Exit fullscreen mode

That runs an Apify Actor I maintain,
Domain Scraper, because the
tedious parts are the RDAP bootstrap, the vcard unpacking and the provider
fingerprinting rather than the HTTP call. If you would rather do it yourself,
https://rdap.org/domain/<name> is the whole API and it needs no key.

The trap: a 404 is ambiguous

rdap.org is a router. It reads IANA's bootstrap registry, finds the RDAP
server for your TLD, and redirects. A 404 back from it looks like it means one
thing: no registry holds this name, therefore the domain is available.

That is what my tool assumed. It was wrong, and the failure is ugly:

rdap.org/domain/github.io    404
rdap.org/domain/notion.so    404
rdap.org/domain/segment.io   404
rdap.org/domain/angel.co     404

Enter fullscreen mode Exit fullscreen mode

All four are registered. All four returned 404.

The reason is that IANA's bootstrap does not cover every TLD. It lists RDAP
services for around 1,200 of them, and some of the most common startup TLDs are
not in there:

.io   .co   .de   .me   .so   .gg   .sh

Enter fullscreen mode Exit fullscreen mode

For those, rdap.org has nowhere to send the query, so it 404s for every name
under them, registered or not. My Actor turned that into isAvailable: true and
would have told you github.io was free to register.

On a tool whose headline feature is availability checking, that is the worst
possible way to be wrong: confident, plausible, and acted upon. Nobody
double-checks a clear answer.

The fix, in two parts

Ask whether there is a registry at all. Fetch
https://data.iana.org/rdap/dns.json once, build the set of TLDs that have an
RDAP service, and only treat a 404 as "available" when the TLD is in it.

import json, urllib.request

def rdap_tlds():
    d = json.load(urllib.request.urlopen("https://data.iana.org/rdap/dns.json"))
    return {t.lower().lstrip(".") for entry in d["services"] for t in entry[0]}

Enter fullscreen mode Exit fullscreen mode

Let DNS settle the rest. A name with NS records has been delegated by the
registry, and a registry does not delegate a name nobody registered. So for the
TLDs with no RDAP, NS records prove registration even though RDAP cannot.

The reverse does not hold, and this is the part worth being careful about: a
registered domain can have no NS at all. So "no NS and no RDAP" is not
"available", it is unknown, and it should be returned as null with an
explanation rather than as a guess. A null is a fact the caller can handle. A
plausible wrong answer is not.

Why this generalises

The bug was not really about RDAP. It was about reading an absence as a
definite answer.

404 means "I have nothing for you". Whether that is because the thing does not
exist, or because you asked somewhere that was never going to know, is not in
the response. Any time you convert a missing result into a positive claim, check
that you actually asked something capable of answering.

I have now hit the same shape three times in a month in different tools: an API
silently ignoring a filter it does not support, a page parser taking the first
number in a plausible range, and this. All three produced output that looked
completely normal.

What it costs

Charged per domain returned, at $2.00 per 1,000 on the free tier, so a hundred
domains is 20 cents plus a $0.02 Actor-start event. Domains removed by the
built-in filters, like onlyRegistered or expiringWithinDays, are not charged,
and neither are lookups that fail.

Notes

  • Some ccTLDs run thin RDAP servers that return fewer fields. You get whatever the registry publishes rather than an error.
  • DNS answers are returned sorted. Resolvers round-robin, and unsorted output makes a scheduled run look like it changed every time when nothing did.
  • Availability comes from the registry itself, not a reseller API, so there is no incentive for anyone to tell you a name is taken when it is not.

Code

Runnable examples are here:
github.com/danielhagever/apify-data-actors

Every example input in that repo is generated from the Actor's own input schema,
so it is valid by construction rather than by my memory of the field names.