Job Url: https://www.linkedin.com/jobs/search/?currentJobId=4363580814&f_AL=true&f_TPR=r86400&f_WT=2&keywords=software%20engineer&origin=JOB_SEARCH_PAGE_JOB_FILTER&start=100

Job Description: Senior Software Engineer
Xpertalent · United States (Remote)

Easy Apply

Save
Save Senior Software Engineer at Xpertalent
Show more options
Your profile is missing required qualifications


Show match details

Help me update my profile


BETA

Is this information helpful?



Get personalized tips to stand out to hirers
Find jobs where you’re a top applicant and tailor your resume with the help of AI.

Try Premium for PKR0
About the job
Senior Engineer: Building internet scale Web crawls

Engineering: Internet-Scale Web Crawling & Data Infrastructure

Location: Remote / Flexible
Type: Full-time 

Salary: Highly competitive: up to $250K DOE + Equity


Our client is building next-generation AI systems that rely on massive, high-quality web data. We’re looking for engineers who have designed, built, or operated large-scale web crawlers. Systems that operate reliably across millions or billions of URLs.


This is a hands-on role for people who have worked close to the metal on web-scale data acquisition, not just consumed third-party datasets.


Well funded exciting start up firm who are leading the way in Web Data and AI! 



Experience Required: 

Designing and building internet-scale web crawling pipelines
Handling real-world challenges like:
Rate-limiting, bot detection, CAPTCHAs, and dynamic content
Distributed crawling, scheduling, and prioritisation
Fault tolerance and crawl resumption at massive scale
Extracting, normalising, and validating large volumes of web data
Optimising for performance, cost, and data quality
Working closely with AI and research teams to support model training and evaluation


Must have: 

Built or operated large-scale crawlers (millions+ of pages)
Worked on search engines, data platforms, ad tech, AI datasets, or web archiving
Designed distributed systems for high-throughput data ingestion
Dealt with hostile or semi-hostile web environments


Strong experience with:

Python, Go, Rust, or Java
Distributed systems (queues, workers, schedulers)
HTTP, HTML parsing, JS rendering (Playwright, Puppeteer, Selenium, etc.)
Cloud infrastructure (AWS, GCP, or similar)


Package and benefits 

Work on hard, unsolved problems at true internet scale
Direct impact on cutting-edge AI systems
High-calibre engineering team
Competitive compensation and meaningful equity


If you are experience and passionate about building internet scale web crawls, APPLY ASAP!