Datacenter throughput for training corpus work at TB scale

Why SquadProxy treats datacenter as a first-class product

Most proxy vendors de-emphasise datacenter because their marketing centres on hostile-target scraping, where residential and mobile win. AI data collection is the opposite. The majority of training- corpus volume comes from sources that are:

• open by design (Common Crawl, Wikimedia dumps, arXiv bulk downloads, HuggingFace dataset hosting)
• tolerant of cloud ASN origins (GitHub, Stack Exchange dumps, Reddit public dumps, academic preprint servers)
• explicitly served from CDN edges optimised for large parallel reads

Common Crawl alone publishes roughly 2–3 billion pages per monthly crawl (around 250–450 TiB uncompressed), and for teams working off that baseline the bandwidth economics of residential don't close. Datacenter is the right answer on cost, latency, and — importantly — provenance: an AWS Ashburn exit has a clean, auditable identity. For corpus documented in a model card, that matters.

The AI-infrastructure-adjacent case

Beyond bulk collection, datacenter is the right tool when you are running AI workloads that talk to other AI infrastructure:

• Model API evaluation. Calling OpenAI, Anthropic, Bedrock, or Vertex at sustained QPS needs low, deterministic latency that only a peer-of-peer datacenter can deliver. Running that traffic through residential adds 100–300ms of jitter and poisons your eval timing.
• Embeddings and vector DB loads. When ingesting scraped corpus into Pinecone, Weaviate, Qdrant, or pgvector, the outbound side (embedding API) and inbound side (DB write) are both hyperscaler-hosted. Running the collection pipeline on a datacenter exit in the same region keeps the whole loop on fast paths.
• Internal API and partner allowlist. You may be scraping a partner's API with an IP allowlist. Datacenter is the only class of exit that works for that, because it's the only one with deterministic IPs.

How SquadProxy structures datacenter

• Shared pools — multi-tenant /24s across nine regions, HTTP/SOCKS5. Bills on bandwidth. Best for breadth across many small targets.
• Dedicated — /29 or larger allocations just for you, unmetered bandwidth, priced per IP per month. Best for TB-scale sustained collection on a short list of targets.
• Private ASN + BGP. For Lab and Enterprise customers running sustained >10 Gbps, we announce a dedicated BGP prefix with a custom ASN. This is the cleanest provenance chain available — any downstream audit of the data source ties back to an ASN exclusively under the customer's use.

Edges

• US East (Ashburn) — 40 Gbps uplink, AWS us-east-1 peer.
• US West (Oregon + San Jose) — 20 Gbps, AWS us-west-2 peer.
• EU (Frankfurt + Amsterdam + London) — 60 Gbps combined, AWS/GCP/Azure peers.
• APAC (Tokyo + Singapore + Seoul + Sydney) — 30 Gbps combined.

Honesty about subnet reputation

Datacenter IPs do not look residential to platforms that fingerprint subnet origin. For targets that care — a small fraction of AI-relevant collection — residential is the right answer and we'll tell you so. For the large majority of AI corpus volume that doesn't, paying residential rates is waste.

We run continuous reputation sweeps on shared pools; any /24 that takes a hit on Spamhaus or a major commercial list gets quarantined and returned to rotation only after the underlying issue is fixed.