Apify Alternatives for Large-Scale Web Scraping (1M+ Pages)

At million-page scale, the cost driver is rarely the crawler itself. It is proxy bandwidth, failed retries, and browser rendering. So when you evaluate Apify alternatives, comparing sticker prices is not enough: what matters is how each tool handles those costs in practice.

Simply moving off Apify to another platform does not automatically make scraping cheaper. A more effective first step is to separate the pages that genuinely need a browser. If an API or a plain HTTP request already returns the data you need, use it. Only tasks that involve JavaScript rendering, clicking through pagination, form submission, or a signed-in session need to go through a browser.

Once you split the workload this way, it becomes much clearer which tool fits which job. Apify, Firecrawl, or a self-hosted crawler suit large volumes of public, directly accessible data. When the workflow reaches the steps that genuinely depend on browser state and interaction, ego (lite) can take over.

The reason is that these tasks are no longer just about fetching a page. The agent has to work with rendered content, the current browser state, and page interaction, and the next action may change based on what the page returns. ego (lite) runs directly in a real browser, so the agent can read the current page, continue the interaction, and hand control back to you naturally when a step needs human judgement or takeover.

So for large-scale jobs, the better plan is usually not to find one tool that replaces the whole Apify workflow. It is to pick the right execution mode per task type: keep the parts that an API or HTTP call can do simple, hand the bulk public crawl to a dedicated crawler, and use ego (lite) for the steps that genuinely depend on a browser environment and interaction. That keeps cost under control and avoids paying browser runtime for tasks that never needed a browser.

What does a million-page crawl cost on Apify?

Roughly $430 to $1,300 on the numbers below, and more than 80% of that is proxy bandwidth rather than crawling.

The rates behind that arithmetic are published on Apify's pricing page.

How compute units are counted is documented at Apify's usage and resources documentation.

That estimate is built from Apify's published rates, not from a bill or a quote. It assumes a million target URLs, a Cheerio-style HTTP pass for the load-bearing majority of them, browser rendering on about 100,000 pages that genuinely need JavaScript, and residential proxies for the whole run.

One distinction matters more than the rest when you plan at this volume. Request volume is not page volume. If 85% of your requests succeed on the first try, a million target pages is closer to 1.15 to 1.3 million requests after retries. You pay for the requests, and you pay for their bandwidth, even the ones that come back empty.

Our cost model, line by line.

Read this as an arithmetic exercise on published prices. We are pricing Apify's rates against a defined workload, and the workload assumptions are stated so you can swap in your own.

Two numbers in that table deserve a second look, because they are where budgets actually go wrong.

The first is the 1024MB line. New Apify users routinely assume that paying for more memory costs more. It doesn't, in CU terms: a job that needs twice the memory finishes in half the time, and the product of the two is what you are billed for. Memory is a scheduling knob, not a cost lever.

The second is the proxy line. Note that the platform literally cannot be the dominant cost here: at $0.20 to $0.13 per CU depending on tier, you would need to burn 2,000-plus compute units to match a single moderate bandwidth bill.

If your crawl is expensive, you are paying for bandwidth, not for orchestration.

How does Apify's compute unit model actually bill?

One compute unit is 1GB of memory held for one hour, measured at one-second granularity, and it is the only unit the platform charges crawling against.

Memory times duration. A run holding 1024MB for an hour spends exactly one CU. The same run sized at 4096MB for fifteen minutes also spends one CU. That is why the CU column in our model moves so little when you change worker sizes — the work is the work.

A few properties of the model are worth internalizing:

1. Cheerio is not a small optimization. Apify's own guidance puts a Cheerio-style HTTP pass up to 20 times faster than the same job in a browser. That factor is what keeps 170 CU for a million pages in reach.
2. Concurrency, not worker size, sets your wall clock. Per-plan limits run from 25 concurrent runs on Free up to 256 on Business, with combined memory ceilings of 16,384MB, 65,536MB, 262,144MB, and 524,288MB across those tiers. At 32 concurrent tasks and roughly 3,200 pages per minute, a million pages is about 5.2 hours.
3. Some actors have hard memory floors. A browser actor will not run below 1024MB and the Google Maps scraper needs 4096MB or more. 4096MB is the pragmatic default when you would rather not tune per-actor.
4. Credits do not roll over. Unused monthly credits expire. If your crawling is bursty, one enormous run a quarter instead of a steady drip, you are paying for capacity you did not use.

None of this is an argument against Apify. It is one of the few platforms whose unit economics you can reason about from published documentation, and the CU model is genuinely honest: it charges for work rather than for rows. The problem is what it doesn't include.

Proxies. Rendering-heavy actors. Storage that keeps accruing while a dataset sits there. Those are the line items that decide whether a million-page crawl costs hundreds or thousands, and they are the same line items on every Apify alternative you might consider.

What are the main Apify alternatives for large-scale scraping?

Four kinds of alternative exist, and only one of them is priced in the same shape as Apify.

Most lists of Apify alternatives are really lists of other tools that scrape websites, which is a different question. If your crawl is expensive rather than infeasible, the relevant axis is not features — it is what each vendor charges you for. Price per compute hour, price per credit, price per gigabyte of proxy traffic, and price per operator hour all behave differently as volume grows.

1. Firecrawl. Credit-based scraping API with LLM-ready output. One scrape is one credit, and every optional output format costs more: JSON mode is +4, question or highlights extraction is +4 per format, PII redaction is +4. Tiers run $0, $20, $106, $424, and $762 for 1,000, 5,000, 100,000, 500,000, and 1,000,000 credits a month.
2. ZenRows. Managed fetching with JS rendering and proxy rotation. Credits run from 5,000 on the $16 tier to 5 million on the $456 tier, with concurrency from 5 to 200 and custom above that. JavaScript rendering multiplies credit cost by 10, and the combined JS-plus-proxy mode by 25. Failed and retried requests are not billed.
3. ScraperAPI. Credit-based fetching with a straightforward concurrency ladder, from 20 on the $49 tier up to 500-plus at the top end, spanning 100,000 to 10.5 million credits.
4. Bright Data and Oxylabs. Proxy-first vendors that also sell managed scrapers. Oxylabs lists SERP pricing at $0.50 to $1.35 per unit sequence and residential bandwidth from $3/GB down to $2/GB at volume. Bright Data's web-scraper page leans on an unlimited-concurrency claim without publishing prices, which is a claim we cannot verify and therefore do not repeat as fact.
5. Self-hosted crawlers. A Scrapy or Playwright pipeline on your own infrastructure. No per-page or per-CU fee at all, and your proxy bill stays your proxy bill. The trade is that queue management, retries, scheduling, and monitoring become your problem, permanently.
6. Agent-driven browsers. This is the route we work on, so read it with that in mind. ego (lite) is a Chromium browser that carries over your existing logins and extensions, and a coding agent drives it through the ego-browser skill inside its own isolated Space. It is where a workflow goes when static requests stop being enough: a sign-in step, a view that only renders under JavaScript, pagination hidden behind a click, a filter that is a form, a code you have to read from an inbox. It is free and runs on macOS today, with Windows and Linux on the roadmap. There is no per-page rate to line up against Apify's, so the question to ask is not which is cheaper per page. It is whether your workflow needs a session at all.

Why does proxy bandwidth dominate the bill?

Because it is the only line item that scales with the size of the pages you download rather than the amount of work you do.

For how that route compares on a like-for-like browser task, ego (lite) vs Playwright is the direct comparison.

Keeping a signed-in session alive across runs is covered in persistent browser sessions across agent runs.

Compute scales with CPU time. Concurrency scales with plan tier. Bandwidth scales with the internet. A megabyte of HTML is a megabyte whether it took a millisecond or a second to arrive, and residential proxy traffic is priced per gigabyte: $8/GB on Apify, $3/GB down to $2/GB at volume on Oxylabs.

At 50KB of compressed HTML per page, a million pages is about 50GB. That is $400 at Apify's listed rate before you count retries, and it gets worse the harder the target is: bot-protected pages are heavier, often need multiple requests to resolve, and can consume up to three times the resources of a friendly page.

Here is the uncomfortable part. Most of that bandwidth is wasted, and you already know it. A product-listing page might be 400KB of HTML, CSS, JavaScript, fonts, and tracking pixels so you can read a price and a stock status, when the actual signal is perhaps 2KB. You are paying residential rates to transport the packaging.

Three levers reduce it, and they are worth more than any vendor switch. Restrict the crawl to the pages that carry the data instead of following every link. Fetch structured endpoints where a site exposes them, since JSON responses are a fraction of the rendered page weight. And render only where rendering is required — which, in our model, is 100,000 pages out of a million.

How do the Apify alternatives compare?

On published rates, the spread between the cheapest and most expensive route is roughly tenfold, and none of it comes from the crawling itself.

Read each row for two things: what the vendor charges for and what it stops you doing. A price without a ceiling is marketing. A ceiling without a price is a sales call.

The Firecrawl row is the one people misread, so it is worth spelling out. A million credits sounds like a million pages, and for a bare scrape it is. Add JSON extraction and you have spent five credits a page, so the same $749 covers 200,000 pages. That is not a pricing trick. It is a real cost for real work, and it means the number you budget is not the number on the tier card.

Firecrawl's own documentation is unusually candid about the other constraint: your real bottleneck will be concurrent browsers. Free tier starts at two, Hobby at five, Standard at 25, Growth at 50, Scale at 100-plus. At 100 concurrent pages and a generous two seconds each, that is about 50 pages a second, or roughly 5.6 hours for a million. It works, but you cannot buy your way past it with credits alone.

Firecrawl's concurrency tiers come from its own pricing page.

For the scraping-library comparison behind that choice, see Playwright vs Puppeteer for web scraping.

ZenRows solves a different version of the same problem by making retries free. Not billed for failed or retried requests is a real advantage on hostile targets, and it is the kind of thing worth checking before you assume a cheaper headline rate wins.

Which criteria decide an Apify alternative?

Four numbers decide it, and three of them are properties of your workload rather than of any vendor.

Feature matrices are mostly noise at this scale, because every vendor does the same core job. What separates them is how their billing behaves when your crawl grows by an order of magnitude, and whether the ceiling they impose arrives before or after that growth does.

Notice how little of that table is about features. The vendors competing for the query Apify alternatives mostly compete on the same checklist: proxy pools, JS rendering, CAPTCHA handling, integrations. Those checklists tend to converge. Billing shape does not converge, and it is the part that decides your invoice.

How do you choose an Apify alternative?

Match the answer to your rendering ratio and your authorisation state, and the shortlist collapses to one row.

1. Static HTML, public, a million-plus pages. Stay on a CU-priced platform or self-host, and spend your energy on proxy rates and on cutting page weight. Moving between platforms at this rendering ratio saves single-digit percentages.
2. Mostly static, with a stubborn rendered minority. Split the crawl. Cheap fetches for the majority, a rendering path for the rest. This is the single highest-return decision available, and it costs nothing to make.
3. JavaScript-heavy throughout, public data. Credit-based APIs earn their premium here if they do the extraction for you, because you are buying tokens and structured output rather than just bytes. Budget for the format multipliers before you pick a tier.
4. Hostile targets and frequent blocks. Price retries, not requests. A vendor that does not bill failures can be cheaper at a higher headline rate.
5. Behind a login, or an internal dashboard. Stop comparing crawler prices. This is authorized, session-dependent work, and it runs in a browser that already holds the session, which is what ego (lite) is for. See our notes on scraping behind login walls for the mechanics, and be clear that no tool here bypasses a site's access controls.
6. A few hundred pages, refreshed often. None of this applies. You are not in the volume regime that makes cost models interesting, and a no-code tool is probably the right call. Our AI web scraping tools guide covers the no-code, API, and agent-browser routes with that scope in mind.

If you are weighing hosted browser infrastructure rather than crawlers specifically, the same logic applies one layer down: headless versus real browser is the decision that determines whether your per-page cost is dominated by rendering at all.

When is ego (lite) worth adding to the workflow?

A static request stops being enough at five specific moments: a sign-in, a view that only renders under JavaScript, pagination behind a click, a filter that is really a form, and a verification step.

Those five are worth naming because they are the ones a fetch cannot recover from on its own. curl returns the login page and reports 200. The HTML arrives and the rows are not in it. Page two exists, but only after a click that never happens inside a request. The data you want sits behind a form rather than a query string. The account asks for a code from an inbox you have open in another tab.

When that happens, the fix is not a bigger crawler. It is running the same workflow in a browser that already carries the session.

To keep this from being theory, we ran one task both ways and kept the evidence. The task: pull the first three non-sponsored fish oil listings from an Amazon search, with name, brand, price, rating, review count, and size. One pass through ego (lite), one pass over plain HTTP with a script. Both ran on the same machine on the same day, and both runs are shown below.

The browser route, step by step

The browser pass runs as four parts: the inputs you hand over, the steps the agent takes, what happens when it hits something it should not decide alone, and how you check the result. Here is each one against a real product search.

1. Inputs. The search URL, an Amazon account already signed in, and one sentence describing the fields to collect. ego (lite) is Chromium and carries over your existing logins, extensions, and history, so there is no cookie export step and no credential to paste anywhere. The ego-browser skill is bound to the browser and installed alongside ego (lite), and you turn it on by typing /ego-browser in the agent's chat box.

1. Key steps. The agent opens its own Space, a separate window marked with a blue glow so it is not driving the tabs you are using. Inside it the agent navigates, snapshots the page to read its structure, and captures what it finds. Those snapshots read nested iframes, which is where rendering-dependent extraction usually falls over.

3. Failure recovery. When the run reaches a sign-in, a verification code, or anything else it should not decide alone, it stops and pulls you into its Space instead of guessing. The documented behavior holds on the riskier end too: a job application waits for you to confirm before the final submit, and a booking flow runs up to the payment page and stops there. A crawler that hits a login wall logs a 200 and keeps going. This one asks.
4. Result validation. Check the collected rows against the page the agent was actually on, not against the URL you handed it. The snapshot the agent worked from is the evidence. This is where the run's one real anomaly surfaced, and it is the kind of thing that only exists on a rendered page.

The same task over plain HTTP instead

Here is the honest other half, because the interesting result is not that HTTP failed. It is where it struggled and what it cost to get it working.

The HTTP pass ran the same task with a script and no browser. It got there in the end. The log is a record of how much transport work stood between the two.

A challenged request is a hard stop. But most of the 503s here were not challenges, and telling the two apart took an experiment.

Rewriting the transport was the middle of the problem, not the end of it. The extractor then had to be corrected twice against the same page.

So both passes reached the three products. The difference is what each one spent to get there. The browser run opened a Space, read pages, and finished with a screenshot for every value it reported. The HTTP run wrote six scripts, swapped its HTTP client because of a fingerprint mismatch, added a retry layer, corrected a sponsored-detection bug, corrected a currency bug, and closed with a report admitting it had no output for the first product.

That is the part a per-page cost model misses. Neither approach is wrong, and the token and wall-clock counters in those screenshots are close enough that speed alone does not settle it. What separates them is where the time goes. One route spends it reading a page. The other spends it building and debugging the machinery that stands in for having read one.

For three products, that machinery is the whole cost and it does not pay for itself. At a hundred thousand pages the same code runs unattended for a day and the arithmetic flips completely.

When you do not need ego (lite) at all

Plain HTTP covers more of this work than the tooling conversation suggests. If the page returns its data in the HTML, fetch it. If the site publishes an API, use the API, because it is the access route the site intends and it survives redesigns that break selectors. And if a large public crawl is the whole job, with no step in it that needs a session, a hosted platform or a credit-based API will beat a local browser on throughput.

Read the Amazon case at the right size, too. It is not proof that HTTP is broken, because HTTP reached the data. And it is not proof that the browser was fast, because the browser run also took over twenty minutes and its own report records a failure. What it shows is where the work went on each side. The browser route's only real problem was the site's, not the tool's. The HTTP route's problems were all its own.

Adding a browser to a workflow that does not need one is the most common way to make a scraping job slower and more expensive than it had to be. Check the cheap route first, and note what it costs when the target needs a session.

Is self-hosting your own crawler still worth it?

Often yes, and the reason is arithmetic rather than craftsmanship: at volume you are paying for a queue and a scheduler, and both are commodities.

Scrapy and Playwright are free, well documented, and boring in the way production infrastructure should be. If your crawl is a recurring job against stable targets, a self-hosted pipeline removes the platform fee entirely and leaves you with server cost and the proxy bill you were going to pay regardless.

The honest counterweight is that the platform fee buys something real. Retry logic that handles five different failure modes. A scheduler that does not drift. Storage, logging, and the ability to re-run a dataset without touching production. Teams consistently underestimate what that costs in engineering hours, because the hours show up in a person's calendar rather than on an invoice.

A rough rule: self-host when the crawl is recurring and the targets are stable. Rent when the crawl is exploratory, or when the site you are crawling changes shape every few weeks. The platform fee is mostly a change-management fee.

Where do all of these options break down?

Three places, and they are the ones that turn a clean cost model into an overrun.

The first is that every vendor's marketing claims outrun its documentation, and the gap is widest exactly where you need precision. Bright Data's web-scraper page states unlimited concurrency and publishes no prices on the same page. That is not a lie we can disprove; it is a claim we cannot verify, which is a different and more useful thing to know. Treat any unlimited, enterprise-grade, or infinite wording as a prompt to go find the number, and if the number isn't published, assume the ceiling exists and is negotiated.

The second is bot protection, which is not a feature you buy so much as a condition you operate in. Efficiency gains from a faster fetch path evaporate when every request needs a residential IP, a warm session, and a retry budget. Our model hits $1,300 not because any vendor is gouging but because harder targets mean heavier pages and more wasted requests.

The third is the one nobody puts on a comparison table: this is public-data infrastructure, and using it that way is your responsibility. Terms of service, robots directives, personal data under GDPR or CCPA, and the difference between a site's public pages and its users' private information are all separate questions from which tool you pick. A cheap crawler does not make an impermissible crawl permissible. We are not lawyers and this is not legal advice, but the cheapest crawl is the one your organisation can actually defend.

FAQ

What is the cheapest Apify alternative for a million pages?

On published rates, a self-hosted Scrapy pipeline with cheap proxies, because there is no platform fee at all. Among managed options, a CU-priced platform running an HTTP pass beats credit-based APIs on raw volume. But the honest answer is that the platform choice is the smaller half of the bill: at $8/GB of residential traffic, a million 50KB pages costs about $400 in bandwidth whoever does the fetching.

Is Firecrawl a good Apify replacement?

For LLM-ready markdown and structured extraction, yes — it is simpler to forecast and the output saves you a parsing layer. For a bare million-page fetch, its concurrency ceiling becomes the constraint before its credit balance does. Free tier allows two concurrent browsers and the Scale tier 100-plus, so throughput is capped by plan rather than by spend.

Does Apify charge for failed requests?

Yes, indirectly. Compute units accrue for the runtime regardless of what the request returned, and proxy bandwidth is spent on the transfer. A retried request is billed twice with nothing to show for the first attempt.

Why are Apify alternatives often more expensive at scale?

They usually aren't, in platform terms. They look expensive because credit multipliers do not map cleanly onto pages: a scrape is one credit, but JSON mode is five, and JavaScript rendering on a proxy-based vendor can be ten or twenty-five times a plain fetch. Multiply the headline rate by the multiplier you actually need and the gap closes or reverses.

Do I need browser rendering for a million pages?

Almost never for all of them. A working assumption is that 90% of pages are readable as static HTML and 10% need JavaScript, which is why our model renders 100,000 pages and fetches the rest. Measure your ratio on a 1,000-page sample before you budget, because it moves the cost more than any vendor choice.

What about Bright Data and Oxylabs as Apify alternatives?

They are strongest when proxies are the problem. Oxylabs lists residential bandwidth from $3/GB down to $2/GB at volume, which is well below Apify's $8/GB, and SERP pricing at $0.50 to $1.35 per unit sequence. Bright Data publishes no prices on its web-scraper page and leans on an unlimited-concurrency claim we could not verify, so treat that as a quote to request rather than a fact to plan on.

The residential bandwidth rates quoted here are listed at Oxylabs' pricing page.

Bright Data's plans, which do not publish per-GB rates on the scraper page, start at its pricing page.

Can I replace Apify with ego (lite)?

They cover different steps, so it reads less like a replacement and more like a handoff. Apify runs hosted crawls at volume. ego (lite) is a free macOS browser that a coding agent drives through a session you are already signed into, and you reach for it when a step needs a login, a rendered view, a click through pagination, a form, or a verification code. If your job is a large public crawl and nothing in it needs a session, you do not need ego (lite) in the pipeline at all.

How do I estimate proxy bandwidth before a crawl?

Fetch a representative 1,000 URLs, measure the compressed response sizes, and multiply. Then add the retry factor: at an 85% first-pass success rate, a million targets is 1.15 to 1.3 million requests. Budget for the higher number, because page weight rises with bot protection.

Is Apify still the right choice at this volume?

Sometimes, yes. Its CU model is transparent, it charges for work rather than rows, and its actor ecosystem removes a lot of glue code. If your crawl is a few hundred thousand pages and your proxies come from a cheaper vendor, the platform is not your problem. The case for leaving is usually about credits expiring, plan-scoped concurrency arriving before you are ready, or bandwidth rates you can beat elsewhere.

What breaks first at a million pages?

The budget, not the technology. Concurrency ceilings and retry billing are documented and predictable. What surprises teams is the bandwidth bill, the assumption about how many pages need rendering, and how much of the crawl turns out to be pages nobody asked for. Fix the scope before you fix the vendor.

Most Apify alternative comparisons answer a question about features. The question that costs money at a million pages is about billing shape, and it has three parts: what does the vendor multiply for, what is its ceiling, and who pays for a failed request?

Apify answers those questions well. Its compute unit is a clean unit of work, and it charges you for browsing pages rather than counting rows. What it does not control is the part of the bill you will actually stare at: residential bandwidth at $8/GB, on pages where most of the bytes are scaffolding.

So the useful move is not to switch platforms. It is to cut the crawl down to the pages that carry the data, fetch the static majority with something 20 times cheaper than a browser, and render the 10% that needs it. Do that, and the vendor comparison becomes a rounding decision.

The parts a crawler cannot reach, the logged-in pages and internal tools are where the crawl stops and a session starts. ego (lite) runs that same workflow with the login already attached, and stops to ask you when a step needs a human. And when a plain fetch already returns the rows, that is the whole answer. Leave the browser out of it.

We went into that first million-page crawl expecting the platform to be the expensive part. Bandwidth answered instead.