Senior Web Scraping Engineer (Python/Large-Scale Crawling) - Full-time - Worldwide

About the Role

We operate a large-scale, sophisticated scraping network that powers our job-matching intelligence. We ingest and interact with thousands of sites, using autonomous agents and scrapers to discover, extract, and process data into our pipelines.

We’re looking for a Senior Web Scraping Engineer who has built and operated scraping systems at serious scale—not just small side projects.

Responsibilities

• Maintain, optimize, and evolve a complex Python-based scraping architecture.
• Work with Playwright, proxy networks, and headless browsers at scale.
• Optimize crawling strategies: timing, concurrency, proxy models, and site-specific tactics.
• Reduce latency and resource usage (e.g., shaving seconds off per crawl at 100k+ scale).
• Ensure reliability, resilience, and data quality across the scraping network.

Requirements

• Proven experience building and running large-scale scraping systems (not just simple scripts).
• Strong Python engineering skills and familiarity with Playwright or similar tools.
• Deep understanding of proxies, anti-bot defenses, and distributed crawling strategies.
• Experience with performance optimization across large volumes of pages.
• Comfortable joining an existing advanced setup and pushing it further.