Web scraping in Laravel with Larascraper

I do a fair amount of scraping in Laravel, and it always splits into two worlds. Guzzle for the pages that are just HTML, and a mess of Puppeteer scripts shelled out from PHP for the ones that render in JavaScript. Two toolchains, two mental models, glue in between.

Larascraper collapses that into one. You write a class, you describe what you want, and it runs through a real headless browser or plain HTTP with the same code. This post walks through the whole thing.

How to Install

composer require edulazaro/larascraper

Enter fullscreen mode Exit fullscreen mode

The browser driver drives Puppeteer, so it pulls a headless browser the first time you scrape a JavaScript site. The HTTP driver needs nothing extra.

A scraper is a class

You extend Scraper and implement handle(). Inside it, $this->scrape($url) starts a fetch, and a terminal decides what comes back.

use EduLazaro\Larascraper\Scraper;
use EduLazaro\Larascraper\ScraperResponse;

class BikeScraper extends Scraper
{
    protected function handle(string $url): ScraperResponse
    {
        return $this->scrape($url)->run();
    }
}

Enter fullscreen mode Exit fullscreen mode

Call it from anywhere:

$response = BikeScraper::run('https://example.com/bikes/4');

$response->data;     // the raw HTML
$response->success;  // true or false
$response->error;    // the error message if it failed

Enter fullscreen mode Exit fullscreen mode

BikeScraper::run() always hands you a ScraperResponse; with no crawler, $response->data is the raw HTML. That is rarely what you want. You want structured data, and for that you write a Crawler.

Crawlers turn a page into data

A Crawler gets a small query builder over the DOM via $this->filter():

use EduLazaro\Larascraper\Crawler;

class BikeCrawler extends Crawler
{
    protected function handle(): array
    {
        return [
            'name'  => $this->filter('h1.title')->text(),
            'price' => $this->filter('.price')->text(),
            'specs' => $this->filter('.spec')->each(fn ($node) => $node->text()),
        ];
    }
}

Enter fullscreen mode Exit fullscreen mode

Chain it onto the fetch and you get the parsed array back:

protected function handle(string $url): ScraperResponse
{
    return $this->scrape($url)->crawl(BikeCrawler::class)->run();
}

BikeScraper::run('https://example.com/bikes/4')->data;
// ['name' => 'Black Eagle', 'price' => '2,499', 'specs' => [...]]

Enter fullscreen mode Exit fullscreen mode

The input is generic. Usually it is an HTML string, but $this->filter($selector, 'xml') treats it as XML, and $this->raw() gives you the untouched input for when you would rather reach for regex, simplexml or json_decode.

Driving a JavaScript page

A lot of the web is empty without JavaScript, so Larascraper runs a real browser by default and lets you act on the page before reading it: click, type, wait, scroll.

protected function handle(string $url): ScraperResponse
{
    return $this->scrape($url)
        ->type('#search', 'black eagle')
        ->press('Enter', waitForNavigation: true)
        ->waitForSelector('.results')
        ->crawl(BikeCrawler::class)
        ->run();
}

Enter fullscreen mode Exit fullscreen mode

The actions run in the browser, in order, and the crawler sees the final DOM. There is a full set: click(), select() (one value or an array for multi-selects), check() and uncheck() for checkboxes hidden inside widgets, hover(), scroll(), waitForSelector(), waitForNavigation().

When a page is plain HTML, skip the browser entirely:

$this->scrape($url)->driver('http')->run();

Enter fullscreen mode Exit fullscreen mode

Same shape, no browser. Actions need the browser, so mixing them with the HTTP driver throws.

Waits that do not fail the run

By default a wait that times out fails the scrape, which is usually correct. But an empty result set is a valid outcome, not an error. Make the wait optional:

->waitForSelector('.results', ['optional' => true, 'timeout' => 8000])

Enter fullscreen mode Exit fullscreen mode

Now an empty search continues instead of blowing up, and the crawler just returns nothing. You can also race several selectors and take whichever lands first:

->waitForSelector(['.results', '.no-results'])

Enter fullscreen mode Exit fullscreen mode

Retry until it works

Captchas, sessions that need warming up, flaky pages: some flows only work on the second or third try. repeatUntil() loops a block until a condition holds, always bounded so it can never hammer a server:

use EduLazaro\Larascraper\Support\Condition;

$this->scrape($url)
    ->repeatUntil(
        Condition::selectorMissing('#captcha'),
        fn ($b) => $b
            ->solveCaptcha('#captcha-img', '#captcha-input')
            ->clickAndWait('#verify'),
        max: 5,
        delay: 1500,
    )
    ->crawl(ResultCrawler::class)
    ->run();

Enter fullscreen mode Exit fullscreen mode

A throw inside one attempt counts as a failed attempt, not a dead run: the loop re-checks the condition and tries again, up to max. A genuine misconfiguration, like an unknown captcha solver or a 4xx from a bad API key, aborts immediately instead of burning retries.

Solving Captchas

Simple image captchas go through OCR, no API cost:

->solveCaptcha('#captcha-img', '#captcha-input')

Enter fullscreen mode Exit fullscreen mode

The OCR packages are optional; install them once with php artisan larascraper:install --captcha. For the distorted ones that tesseract cannot read, there is an OpenAI vision solver you opt into per call:

->solveCaptcha('#captcha-img', '#captcha-input', ['solver' => 'vision'])

Enter fullscreen mode Exit fullscreen mode

It reads OPENAI_API_KEY from the environment. OCR stays the default, so you only pay for the hard ones.

Downloading files

For a PDF or any binary produced by a click or a form submit, use capture():

$response = $this->scrape($url)
    ->clickAndWait('a.download')
    ->capture(['expect' => 'application/pdf'])
    ->file();

$response->file;         // the bytes
$response->contentType;  // application/pdf

Enter fullscreen mode Exit fullscreen mode

Using Spiders and concurrency

One scraper fetches one page. For thousands, you write a Spider. It has a handle() where you call your scrapers, and pool() runs them concurrently:

use EduLazaro\Larascraper\Spider;

class CatalogSpider extends Spider
{
    protected int $concurrency = 20;

    public function handle(): array
    {
        $ids = range(1, 5000);

        return $this->pool(
            $ids,
            BikeScraper::class,
            fn ($response, $id) => $response->success ? $response->data : null,
            $this->concurrency,
        );
    }
}

$bikes = array_filter(CatalogSpider::run());

Enter fullscreen mode Exit fullscreen mode

PHP is single-threaded, so this leans on Fibers. Each scraper suspends when it fetches, the Spider batches all the in-flight requests into one wave, sends them together, then resumes each Fiber with its response. Real concurrency lives on the HTTP driver, and a shared cookie Session threads through every request so a login survives the whole crawl.

Wrapping up

That is the entire surface: a Scraper to fetch, a Crawler to parse, browser actions when the page needs them, a Spider when one page is not enough. The same class shape whether you go through a browser or plain HTTP.

👉 Package on Packagist.
👉 Source on GitHub.

Written against version 3.3. If you try it and something is missing, open an issue.