Give every collection job its own environment instead of one IP carrying everything
The script runs fine locally, then starts returning challenge pages half an hour after deployment. You swap the IP, and a day later the whole range is blocked. Several jobs share one browser instance, cookies overwrite each other, the numbers stop matching and nothing tells you which step broke. NexBrowser gives each collection job its own browser environment and proxy exit, keeps login state around, and centralises scheduling, retries and exports so scripts can just collect.
Check the target site’s robots.txt and terms of service before you collect.
JOB-PRICE-US
203.0.113.22Running
JOB-PRICE-DE
198.51.100.51Running
JOB-SERP-UK
192.0.2.33Running
JOB-STOCK-JP
203.0.113.66Queued
JOB-NEWS-BR
198.51.100.90Retry 1/3
JOB-LIST-AU
192.0.2.17Done
6 collection jobs · running in parallel · exported to CSV / JSON
Writing the scraper is the easy part - keeping it running is not
Running once and running for three months are different problems, and the time goes almost entirely into the runtime layer.
It stalls shortly after deployment
Everything works in local debugging, then production starts returning challenges or blank pages. Usually the selectors are fine and the runtime is the giveaway: a default headless UA, missing fonts and plugin lists, perfectly regular request timing - stacked together, an unmistakably non-human profile.
Whole IP ranges get banned
Every job shares one datacenter exit, volume rises and the target blocks the entire range, until even ordinary browsing fails. You move to another line, get blocked again days later, and the budget quietly becomes an IP-replacement budget.
Jobs sharing a browser contaminate each other
Several jobs inside one instance overwrite each other’s cookies, localStorage and A/B bucket markers, so captured prices and rankings do not belong to one context. The run is neither reproducible nor safe to publish.
Every rerun starts by signing in again
Throwaway instances keep no cache or login state, so each round repeats the sign-in, the verification and the resource loading - slower, easier to break mid-run, and resumable only if you write the state management yourself.
What a site actually sees when collection gets flagged
Detection is stacked signals too. Knowing the layers tells you what the runtime can fix and what only pace and compliance can.
- 01
Exit IP and request rate
The dominant layer. Datacenter ranges, high concurrency from one exit, perfectly even intervals and uninterrupted access all draw throttling or bans. Assigning an exit per job and controlling concurrency and pacing beats rotating IPs endlessly.
- 02
Browser fingerprint and automation traits
Canvas, WebGL, audio, font list, screen parameters and hardware concurrency describe a device, while a default headless UA, missing plugin and language lists and injected automation markers announce a bot. Internal consistency matters more than uniqueness.
- 03
TLS and HTTP-level fingerprints
Cipher suite order, extension lists and HTTP/2 frame settings form a signature tied to a browser version. Hand-assembled headers from an HTTP client rarely line up with the browser they claim to be; requests from a real engine line up by construction.
- 04
Cookies and session continuity
Real users arrive with history: prior sessions, accumulated cookies, cached resources. Hitting the same site from a brand-new blank instance every time is behaviourally stranger, and it locks you out of anything needing a session.
- 05
Behavioural rhythm and page interaction
Dwell time, whether scrolling and mouse events exist, whether paging intervals vary, and whether resources load normally all feed the call. A session with no images, no interaction and millisecond paging looks nothing like a visit.
So: the first three layers are solved by the runtime itself - a real engine, coherent fingerprint parameters and a per-job exit - while the last two come down to keeping sessions and pacing sensibly. Note that NexBrowser provides a managed browser runtime and offers nothing for bypassing access controls or verification mechanisms. For how the isolation works, see multi-account isolation. For choosing and binding exits, see residential IP and proxy setup.
Four capabilities behind stable public data collection
Environments, proxies, scheduling and exports in one place, so collection scripts stop maintaining sessions and configuration themselves.
An environment and a proxy per job
Give every collection job its own browser environment and exit. Fingerprint parameters, login state and local storage persist, so a rerun needs no reconfiguration, and parallel jobs keep their sessions and caches separate.
Local API for your automation framework
Create, launch and recycle environments through the local API, and connect Selenium, Puppeteer, Playwright or any WebDriver client. Job parameters, proxy settings and results are readable and writable from your scripts, and the local API is free with no call limit.
Scheduling, retries and resumable runs
Orchestrate flows with no-code RPA and schedule them hourly or daily, with failed steps retried automatically. Sessions and login state carry across runs, so an interrupted long job resumes from its last state instead of restarting the batch.
Headless long runs and exports
A built-in headless mode suits jobs that run for hours on a server, while headed mode covers debugging and manual checks. Run logs and collected results are archived together and export to CSV or JSON for whatever comes next.
Four steps to move collection into a managed runtime
Get one real job working end to end, then copy the same pattern across the rest.
- 1
One environment per job
Name environments "job + region" - JOB-PRICE-US and so on - with a fingerprint template resembling the site’s ordinary visitors, and keep jobs that need a login in their own environments so the session persists.
- 2
Bind exits and cap concurrency
Attach a residential or static exit in the target site’s region and run the availability check. Decide each job’s concurrency and request interval up front instead of letting every job crowd onto one line.
- 3
Connect scripts or RPA
Build simple flows with drag-and-drop RPA. For complex logic, start the environment through the local API and take over the page with Selenium, Puppeteer or Playwright, writing results back into your own pipeline.
- 4
Schedule, monitor and export
Run jobs hourly or daily on a schedule with automatic retries and logging, export results to CSV or JSON, and alert only on genuinely abnormal runs so people handle the judgement calls.
For no-code orchestration see no-code RPA, and for the scripting interface see local API automation. Follow the target site’s robots.txt and terms of service, along with local law and data protection rules.
Three reasons collection teams pick it
A runtime that holds, an integration that takes an afternoon, costs that stay predictable.
A real engine, reproducible runs
A real Chromium core rather than assembled requests, with 20+ fingerprint parameters configured as a coherent template, so the same environment starts as the same device every time - results reproduce and failures can be traced.
No architecture changes to integrate
The local API speaks Selenium, Puppeteer, Playwright and WebDriver, so an existing script moves into a managed environment by changing how it connects. It is free with no call limit and is not priced per request.
A free tier you can actually work in
1 member and 10 windows free forever with no feature gating - enough to move your main jobs across. Extra capacity starts at 0.30 USDT per window per 30 days, so you scale with the concurrency you need instead of paying for idle capacity.
Move your least stable job into its own environment first
Download the Windows client, run one real job inside the 10 free windows, then decide whether to scale.
Data collection questions, answered
My scraper keeps getting detected - does changing the User-Agent help?
Barely. A UA is one string, while sites compare the whole environment for consistency: a UA claiming Chrome on Windows should come with matching fonts, plugins, WebGL renderer, screen parameters and TLS handshake traits. Running on a real Chromium core with those parameters configured as a coherent set does far more than editing the UA - and slowing the pace while keeping normal sessions and cache makes the visit look like a visit.
What should I do when collection IPs get banned?
Before rotating, work out which ban it is. Throttling on a single IP usually means concurrency and frequency are too high; losing a whole range usually means the datacenter block was recognised. Assign a dedicated exit per job, use residential IPs in the site’s own region, cap per-job concurrency and leave irregular gaps between requests. One line carrying every job will reach the same ending no matter how often you swap it.
Will Selenium or Playwright scripts be recognised?
Started with default settings, easily: automation markers, the headless default UA and missing language and plugin lists are all ready-made clues. Connecting those frameworks to a NexBrowser environment through the local API runs the page on a real Chromium core with a coherent fingerprint and persistent session, and the script itself barely changes - only how it connects.
What is the difference between headless and headed collection?
Headless suits long server-side batches: lighter, no graphical session needed. Headed suits debugging, manual verification and anything where you want to watch the page. Both use the same environment configuration, so the practical order is to get the flow right headed, then switch to headless for the batch. Headless defaults expose automation traits more readily, which is exactly why the real engine and a complete environment configuration matter.
Do parallel collection jobs interfere with each other?
Inside one browser instance, yes: cookies, localStorage and A/B bucket markers overwrite each other and the captured data no longer belongs to one context. With an environment per job, each has its own storage jar, exit IP and login state, so jobs run in parallel without interference and one failure does not stall the rest.
Is collecting public data legal?
Check the target site’s robots.txt and terms of service first, follow local law and data protection rules, be especially careful with personal data, logged-in content and copyrighted material, and keep the request rate low enough not to burden the server. NexBrowser provides a managed browser runtime - isolated environments, proxy configuration, scheduling and result export - and offers nothing for bypassing access controls or verification mechanisms.
Have other questions? We're here to help anytimeContact Support →
Related reading
- AI browser automationA persistent browser runtime for AI agents
- Market research by regionCompetitor price monitoring from a local view
- SEO rank trackingMulti-region SERPs and Google rank checks
- Local API automationFree, unlimited calls, standard framework support
- No-code RPADrag-and-drop flows and scheduled runs
- Residential IP and proxiesBind an exit per job and test it before the run
Ready to manage every account profile in one place?
Install in 3 minutes. 10 free windows on sign-up, no credit card.
Free Download for WindowsAlready have an account? Sign in to the client to sync all profiles.