Puppeteer vs Playwright vs Selenium for scraping

The short answer, before anything else: for new scraping projects in 2026, Playwright is the default; Puppeteer keeps Chrome-only Node pipelines; Selenium stays wherever language breadth or existing Grid infrastructure demands it. The differences that decide are the protocol (WebDriver's extra hop against direct CDP), the languages, and the stealth ecosystems. ego (lite) lets the agent run many actions in one JavaScript pass inside the page instead of one command per round trip. One scenario cuts across all three identically: data behind logins you own. Each starts a clean browser, so each hands you login scripting and cookie maintenance.

Every scraping team eventually holds this three-way debate, usually with one member defending the Selenium suite they've run for a decade, one who came up on Puppeteer, and one who read that Playwright won.

All three positions have merit, because the tools were built in different eras for different jobs: Selenium (2004) for cross-browser testing breadth, Puppeteer (2017) for direct Chrome control, Playwright (2020) for modern-web reliability.

How do the three architectures differ?

Selenium speaks the WebDriver protocol: your script sends JSON-over-HTTP commands to a browser-specific driver binary, which relays them to the browser. That indirection is why Selenium runs everywhere (any browser that ships a driver, including legacy ones), and why every single command pays a network round trip through an intermediary.

Puppeteer and Playwright skip the middleman and speak Chrome DevTools Protocol (Playwright uses equivalent direct channels for Firefox and WebKit): a persistent connection, lower per-command latency, and access to browser internals WebDriver never exposed, like network interception without a proxy.

For a test suite running dozens of steps, the protocol difference is a rounding error. For a scraper executing millions of commands across thousands of pages, per-command overhead and connection stability compound into real throughput differences, which is why scraping infrastructure drifted CDP-ward years before the testing world did.

One fairness note on trajectory: the gap is narrowing. WebDriver BiDi, the W3C's bidirectional successor protocol, is landing across the Selenium ecosystem and already powers things like console and network capture in Selenium-based tooling such as mcp-selenium's diagnostics tool. The architecture argument above describes 2026's shipping reality, not a law of nature.

Protocols are destiny at scale.

How do they compare on the scraping matrix?

Five rows cover what scraping teams actually ask. Each cell includes the limitation, because that's the half that decides.

The waiting row is worth seeing in code, because it's the row that produces 2 a.m. pages. The same guarded click in each:

# Selenium: you compose the wait
WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.CSS_SELECTOR, "#load-more"))).click()

// Puppeteer: you remember the wait
await page.waitForSelector('#load-more', { visible: true })
await page.click('#load-more')

// Playwright: the wait is the action
await page.click('#load-more')

Multiply that difference by every interactive element in a 30-site scraper portfolio, written partly by juniors or coding agents, and you've explained most real-world flake-rate gaps between the three stacks.

What are the pros and cons of each?

Selenium.

Pros: unmatched language and browser breadth, twenty years of infrastructure patterns (Grid), and the largest install base of automation skills in the industry. Cons: the WebDriver hop costs latency at scraping scale, waits are entirely your discipline, and the modern scraping tooling wave (agent integrations included) ships elsewhere first.

Puppeteer.

Pros: direct CDP speed on Chrome, the most battle-tested stealth classic, and the deepest pile of scraping recipes on the internet. Cons: JavaScript only, Chrome-first with Firefox in beta, and manual waiting that turns hurried code into flaky scrapers.

Playwright.

Pros: auto-waiting kills the flake class the other two make you manage, browser contexts make parallel session fleets cheap, five official languages, and the current stealth-fork generation targets it. Cons: youngest scraping folklore of the three, heavier install (it manages its own browser builds), and no legacy browser story at all.

Which should you pick for scraping in 2026?

By situation, since that's how the decision actually arrives. Starting fresh with no constraints: Playwright, for auto-waiting plus contexts plus the active stealth scene; that combination is why it's become the default recommendation across scraping guides.

Node shop, Chrome targets, screenshot-heavy: Puppeteer remains excellent, and its per-capture speed advantage is real.

Team writes Ruby or PHP, or scraping rides on existing Selenium Grid infrastructure: Selenium, without embarrassment; the WebDriver tax is invisible until your volumes are large, and infrastructure you've already amortized beats infrastructure you haven't built.

The trap to avoid is migrating a working stack for identity reasons. All three scrape competently; switching costs are real; move when a specific limit (language, flake rate, parallelism cost) actually binds, and measure that limit on your own targets before the rewrite, not after.

Which framework should a QA engineer learn?

For a new QA automation learner in 2026, start with Playwright unless your team already standardizes on Selenium or Cypress. Playwright gives one modern API across major browsers, built-in auto-waiting, tracing, and parallel contexts; Selenium remains the better investment when job requirements or an existing Grid require Java, Python, C#, Ruby, or legacy browser coverage.

• Choose Playwright for a greenfield web suite and learn locators, fixtures, traces, and CI parallelism first.
• Choose Selenium when cross-language portability, WebDriver-standard tooling, or a large existing estate matters more than a shorter setup.
• Choose Cypress when its in-browser runner and team workflow are the deciding constraint; it is a separate trade-off, not a drop-in replacement for either framework.

The durable skill is test design: deterministic fixtures, accessible locators, isolation, and useful failure artifacts. Framework syntax changes faster than those principles.

How should each framework handle bot walls and CAPTCHAs?

None of Selenium, Puppeteer, or Playwright reliably defeats Cloudflare, CAPTCHA, WAF, or rate limits, and this comparison does not provide bypass instructions. A block is a signal to reduce load, use an official API or licensed feed, request permission, or stop the collection—not to escalate evasion.

• Respect robots directives, terms, consent requirements, rate limits, and the site's account policy; identify your client where appropriate.
• Use backoff, caching, incremental checkpoints, and a bounded request budget for permitted public data.
• For data behind an account you own, use the supported login flow or an approved export. Do not replay cookies, rotate identities, or spoof fingerprints to evade a control.

Stealth plugins and realistic mouse movement may change a browser fingerprint, but they are not a compliance or reliability guarantee. Treat CAPTCHA and 403 responses as a workflow state that needs a human or a different data source.

How do they handle logins, infinite scroll, and pagination?

All three can handle login flows, infinite scroll, and cursor or page-number pagination, but reliable scraping depends on state modeling rather than a single selector. Wait for a meaningful state change, capture the current cursor or URL, and checkpoint each page before continuing.

• Login: prefer a dedicated test account or supported session bootstrap, keep secrets outside source control, and stop for MFA or consent instead of attempting to bypass it.
• Infinite scroll: scroll in bounded increments, wait for new item IDs, deduplicate, and stop when the count or sentinel stops changing.
• Pagination: persist the next cursor, validate row counts, and make retries idempotent so a timeout does not duplicate records.

Playwright's auto-waiting reduces incidental flake; Selenium and Puppeteer can be equally correct when explicit waits target application state instead of fixed sleeps.

What changes at large scraping scale?

At tens of thousands of pages per day, the framework is only one part of the system. Queue URLs, enforce per-domain budgets, reuse browser contexts safely, persist checkpoints, and separate extraction workers from validation and export so one slow site cannot stall the fleet.

• Use Playwright when browser contexts, tracing, and modern parallelism are the priority; use Selenium Grid when your organization already operates a broad, multi-language fleet.
• Use Puppeteer for a focused Chrome/Node pipeline when its direct CDP model and team expertise outweigh cross-browser needs.
• Add observability for queue age, success rate, bytes, browser minutes, retries, block responses, and cost per valid record.

Scale is not permission to ignore a site's controls. If a target cannot support the requested rate, negotiate access or choose an API, feed, or licensed provider instead of adding more parallel browsers.

Which framework fits each scraping use case?

Pick the framework that matches the dominant constraint: Playwright for new multi-browser extraction and test-like workflows, Puppeteer for Chrome-only Node jobs, and Selenium for language breadth or an established Grid. The cheapest migration is often keeping a stable incumbent and fixing its actual bottleneck.

How do you keep browser tests and scrapers stable?

Stability comes from waiting on observable state, using resilient locators, isolating data, and retaining a trace or equivalent artifact for every failure. Do not hide flake by increasing global timeouts or blindly retrying a mutating scrape.

• Prefer role, label, test-id, or stable data attributes over generated CSS classes and screen coordinates.
• Use bounded retries only for known transient failures, and mark a test flaky when retries conceal a real regression.
• Pin browser and framework versions, run a small smoke flow after upgrades, and retain traces, console logs, and response status for diagnosis.

For scrapers, add schema validation and duplicate detection after extraction. A page that loaded successfully but returned an empty or shifted schema is a data-quality failure, not a successful run.

How do you survive browser and anti-bot updates?

Treat browser updates, layout changes, and new access controls as expected drift. Keep selectors and extraction rules in a small versioned layer, monitor representative pages, and canary upgrades before rolling them across the fleet.

• Pin dependencies for reproducibility, but schedule security and browser updates; verify release notes and run smoke tests before unpinning.
• Alert on sudden changes in status codes, item counts, field null rates, latency, and CAPTCHA or consent frequency.
• When a site changes its policy or markup, pause the job, review permission and contract terms, then update the parser or source—not a stealth workaround.

A maintained fallback can be an official API, export, or licensed dataset. Reliability is the ability to recover transparently, not the ability to keep scraping after every control changes.

What about your own logged-in sources?

One scenario cuts across all three identically: the data lives behind logins you own (vendor portals, dashboards, member communities). Selenium, Puppeteer, and Playwright all start clean browsers, so all three hand you the same chores: script the login, persist the cookies, patch it when 2FA or session policy changes.

That maintenance is the part ego (lite) deletes. It imports your Chrome profile in one click, so the agent starts from a browser you are already signed into, which avoids the extra challenges that blank profiles, repeated logins, and unfamiliar sessions tend to trigger.

That route is measured on Real-World Bench, a public 31-task benchmark against live production sites, and the tasks look like the scraping work teams actually assign rather than demo pages: cross-comparing Metacritic's highest-rated Action RPGs on PS5 against PC, five games per platform with Metascores, or collecting the top posts about index funds from r/personalfinance sorted by top over the past year. Selenium and Puppeteer were not among the tools measured; the harness and dataset are open atcitrolabs/ego-browser-benchmark-framework.

Here's what that looks like in practice, from a recorded session against a live page: the task written as JavaScript and piped in, with only the extracted fields coming back, not a page dump.

ego-browser nodejs <<'EOF'
const task = await egoBrowser.newTaskSpace('evidence-egobrowser-hn')
console.log({ taskSpaceId: task.id })

await task.page.goto('https://news.ycombinator.com/', { waitUntil: 'load', timeout: 20000 })
const title = await task.page.title()
const topStory = await task.page.locator('.athing .titleline > a').first().innerText()
const points = await task.page.locator('.subtext .score').first().innerText().catch(() => null)
console.log({ title, url: task.page.url(), topStory, points })
EOF

# Real output
{
  "taskSpaceId": 13
}
{
  "title": "Hacker News",
  "url": "https://news.ycombinator.com/",
  "topStory": "Qwen 3.8 27B",
  "points": "412 points"
}

It doesn't replace any of the three for large-scale public scraping or CI test infrastructure; it replaces the login-scripting half of the job for sources where the account is yours.

See ego (lite) vs Selenium, or download ego (lite) for Mac, free.

FAQ

Which is better, Selenium or Puppeteer?

For Chrome-only scraping in a JavaScript codebase, Puppeteer: direct CDP control without the WebDriver hop. For anything needing other languages, other browsers, or existing Grid infrastructure, Selenium. They're rarely both viable for the same team, which makes this the easier of the pairwise calls.

What are Playwright vs Selenium pros and cons in short?

Playwright: auto-waiting, cheap parallel contexts, modern tooling; but no legacy browsers and younger folklore. Selenium: widest languages and browsers, mature Grid; but per-command WebDriver overhead and fully manual waiting. New scraping projects lean Playwright; established Selenium estates rarely benefit from moving.

Is Selenium too slow for web scraping?

Not too slow, just taxed: the JSON-over-HTTP driver hop adds per-command latency that CDP tools skip. At small and medium volumes it's unnoticeable; at millions of commands it's a real line item, and that's the scale where teams migrate.

Which of the three works best with AI coding agents?

Playwright, by infrastructure: its MCP server and CLI are first-party and maintained, while Puppeteer's reference MCP server is deprecated and Selenium's is community-run. Auto-waiting also makes agent-written Playwright scripts flake less. Selenium shops can still connect agents through the community mcp-selenium; our Selenium MCP guide covers that setup.

Is Playwright replacing Selenium?

In new-project defaults, largely yes; in installed base, no. Selenium's language breadth, Grid infrastructure, and two decades of institutional knowledge keep it running enormous estates that have no reason to move. The accurate statement is that the frontier shifted, not that the incumbent died.

Do any of the three handle CAPTCHAs or bot walls automatically?

No. Stealth plugins reduce fingerprint signals; none defeats protocol-level detection reliably, and CAPTCHA solving means third-party services with their own terms. Scope your scraping to targets that tolerate it, and use your own authenticated access (not automation tricks) for sources where you have accounts.