Billy

When you scrape at any real volume, the bottleneck is rarely your code — it's the target site's rate limiting and IP bans. Rotating residential proxies solve this by routing each request through a different real-user IP. Here's how to wire them into requests and Scrapy in Python, including the sticky-session trick most tutorials skip.

The proxy URL format

A residential proxy is just an authenticated HTTP/SOCKS endpoint. With a pool gateway, you target a country and control session behavior through the username, not separate endpoints:

http://USERNAME_country-us_session-a1b2c3_lifetime-30:[email protected]:1000

Enter fullscreen mode Exit fullscreen mode

  • country-us — exit country (ISO code)
  • session-a1b2c3 — a sticky-session id; reuse it to keep the same IP, change it to rotate
  • lifetime-30 — how many minutes that session's IP stays fixed

Basic request through a rotating proxy 

import requests

USER = "USERNAME"
PWD = "PASSWORD"

def proxy(country="us", session=None, lifetime=30):
    tag = f"_country-{country}"
    if session:
        tag += f"_session-{session}_lifetime-{lifetime}"
    url = f"http://{USER}{tag}:{PWD}@proxy.gproxy.net:1000"
    return {"http": url, "https": url}

# New IP on every call (no session id):
r = requests.get("https://api.ipify.org?format=json", proxies=proxy(), timeout=30)
print(r.json()["ip"])

Enter fullscreen mode Exit fullscreen mode

Run it in a loop and you'll see a different IP each time — the gateway rotates automatically when no session id is present.

Sticky sessions: keep one IP across requests

Some flows (login, multi-step checkouts, paginated results behind a cookie) break if your IP changes mid-session. Pin the IP by passing a stable session id:

import uuid

sess = uuid.uuid4().hex[:8]        # one id for the whole flow
p = proxy(country="de", session=sess, lifetime=30)

s = requests.Session()
s.proxies.update(p)
s.get("https://example.com/login")
s.post("https://example.com/login", data={...})   # same exit IP

Enter fullscreen mode Exit fullscreen mode

When you want a fresh IP, generate a new session id.

Scrapy integration

Scrapy reads the proxy off each request's meta. A tiny middleware rotates a fresh session per request:

# middlewares.py
import uuid

class ResidentialProxyMiddleware:
    USER = "USERNAME"
    PWD = "PASSWORD"

    def process_request(self, request, spider):
        sess = uuid.uuid4().hex[:8]
        cc = request.meta.get("country", "us")
        request.meta["proxy"] = (
            f"http://{self.USER}_country-{cc}_session-{sess}_lifetime-10:"
            f"{self.PWD}@proxy.gproxy.net:1000"
        )

Enter fullscreen mode Exit fullscreen mode 

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ResidentialProxyMiddleware": 350,
}

Enter fullscreen mode Exit fullscreen mode

Set request.meta["country"] = "gb" in a spider to geo-target specific requests.

A few practical notes

  • Retry on proxy errors. Residential IPs occasionally drop; wrap requests with retries (urllib3 Retry, or Scrapy's RETRY_ENABLED) so a bad exit doesn't kill a job.
  • Respect the target. Rotating IPs is for spreading legitimate load and geo-testing — not for hammering a site past its terms. Add delays and honor robots.txt.
  • SOCKS5 works the same way on port 1002 if you need it (socks5://...).

Getting credentials

You need a residential pool that supports country targeting and sticky sessions. I use GProxy — the username-based targeting above is exactly its format, and it's usage-based ($0.90/GB) rather than a fixed monthly seat, which suits burst scraping jobs. Grab a key from the dashboard, drop it into USER/PWD, and the snippets above run as-is.

Happy (responsible) scraping.