Lucy Green

Tired of burning cash on expensive SEO suites just to run a simple competitor analysis? Me too. That’s why I built a lean workflow using the Chrome DevTools Network tab and a lightweight, free Python script to map out any page’s internal link structure—no paid subscriptions required.

Most people don’t realize the DOM alone can be misleading. JavaScript frameworks often inject placeholder links that differ from the actual href attributes rendered after hydration. The real source of truth? The network waterfall. When you scrape the fetch/XHR requests from a live page, you get the raw data that drives the site’s navigation.

Here’s a quick snippet to grab internal links from a page’s response:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urlparse, urljoin

def get_internal_links(url):
    try:
        response = requests.get(url, timeout=10)
        soup = BeautifulSoup(response.text, 'html.parser')
        domain = urlparse(url).netloc
        internal_links = set()
        for a_tag in soup.find_all('a', href=True):
            href = a_tag['href']
            full_url = urljoin(url, href)
            parsed = urlparse(full_url)
            if parsed.netloc == domain and parsed.scheme in ('http', 'https'):
                internal_links.add(full_url)
        return list(internal_links)
    except Exception as e:
        return f"Error: {e}"

# Usage
url = "https://example.com"
links = get_internal_links(url)
print(f"Found {len(links)} internal links on {url}")

Enter fullscreen mode Exit fullscreen mode

This script filters out external links, ads, and social media icons. But it still misses dynamic routes loaded via API calls. To catch those, you’d hook into the Network tab using a headless browser like Playwright—but that’s a tutorial for another day.

Why does this matter? SEO audits often get stuck on homepage metrics. But internal link depth signals content priority to search engines. If your “Services” page is three clicks deep and only linked from a footer, it’s practically invisible.

I’ve been using this approach to spot orphaned pages and broken breadcrumbs on client sites. If you want to scale this into a full crawl—including backlink gaps and keyword mapping—you’ll eventually need a tool that automates the heavy lifting. That’s where SerpSpur comes in as a solid Semrush alternative. It gives you the same site audit, backlink analyzer, and keyword tracking without the enterprise price tag.

But for today, just run the script on your own site. You’ll be surprised what you find. Drop your results in the comments—I’d love to see what you uncover.