Job Title: Software Engineer, Web Crawling

Company Name: Exa (exa.ai)

Job Details: Hiring,Remotely,in,San,Francisco,,CA,In-Office,or,Remote,150K-300K,Annually,Mid,level

Job Url: https://builtin.com/job/web-crawler-engineer/7085064

Job Description: Exa is building a search engine from scratch to serve every AI application. We build massive-scale infrastructure to crawl the web, train state-of-the-art embedding models to index it, and develop super high performant vector databases in Rust to search over it. We also own a $5M H200 GPU cluster that regularly lights up tens of thousands of machines.
As a Web Crawler engineer, you'd be responsible for crawling the entire web. Basically build Google-scale crawling!

Desired ExperienceYou have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly

You have experience with some high performance language (C++, Rust, etc.)

You are familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)

You’re comfortable optimizing a system to an exceptional degree

You care about the problem of finding high quality knowledge and recognize how important this is for the world

Example ProjectsBuild a distributed crawler that can handle 100M+ pages per day

Optimize crawl politeness and rate limiting across thousands of domains

Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures

Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency

This is an in-person opportunity in San Francisco. We're happy to sponsor international candidates (e.g., STEM OPT, OPT, H1B, O1, E3).