Job Title: Principal Agentic Engineer

Company Name: ZeroFox

Job Details: RemoteFull,Time

Job Url: https://hiring.cafe/viewjob/prr0myn4anyriqqy

Job Description: Posted 5d agoPrincipal Agentic Engineer@ ZeroFoxView All JobsWebsiteUnited StatesRemoteFull TimeResponsibilities:Design architectures, Build evaluation, Implement orchestrationRequirements Summary:7+ years software engineering experience with production systems; expertise in agent reliability, orchestration, and retrieval-augmented systems; backend development and observability.
ZeroFox protects organizations from external threats across the public attack surface, and we're building agentic AI into the core of how that work gets done.You'll define and build the production agentic systems that power ZeroFox's product: the architectures, tooling, and practices that make agents manage state across steps, handle failures without losing context, and know when to escalate to a human. This is deep agent systems work. You'll be solving the hardest problems in multi-step orchestration, reliability, and evaluation for non-deterministic systems operating on adversarial data.What you'll doDesign and implement production agent architectures: state management, error handling, retry logic, graceful degradation, human-in-the-loop escalationBuild evaluation and testing frameworks for non-deterministic agent workflows: offline tests, synthetic data generation, regression checks, and post-deploy monitoringImplement orchestration patterns: multi-agent coordination, tool-calling chains, memory management, context window optimizationDefine deployment and governance practices: agent versioning, rollback, behavioral telemetry, anomaly detectionInstrument agents with tracing, logging, and observability that make production behavior debuggableEstablish architectural standards and best practices for agentic development through design reviews, pairing, and mentorshipRequirementsRequired qualifications7+ years of software engineering experience building production systems, with hands-on experience designing and deploying LLM-based agentsDeep knowledge of agent reliability patterns: state management, error boundaries, escalation logic, context management, tool-calling failure modesExperience with agent orchestration frameworks and understanding of tradeoffs between existing frameworks and building customStrong backend engineering fundamentals: testing, monitoring, deployment, debugging, performance optimizationHands-on experience with retrieval-augmented systems and understanding of how retrieval quality affects agent behaviorExperience building systems that handle adversarial or noisy input (cybersecurity, fraud detection, content moderation, or similar domains)Familiarity with cloud-based AI deployment, including observability, reliability, and cost considerationsWhat success looks likeAgents ship to production with defined reliability contracts and clear operational ownershipAgent failures are detected and handled systematically through telemetry and evaluation pipelines, not customer reportsThe team has reusable patterns for state management, error handling, and escalation that new agent projects build on rather than reinventPreferred qualificationsExperience defining agent governance practices (versioning, behavioral telemetry, rollback) in a production environmentBackground in cybersecurity or adjacent domain with high-noise, adversarial dataExperience with tool-use architectures and integration patterns for connecting agents to external systemsTrack record of raising engineering capability across a team through mentoring, pairing, or design reviewBenefitsCompetitive compensationCommunity-driven culture with employee eventsRegular catered lunches for in-office work; snacks, drinks available dailyGenerous time off Comprehensive health benefits & 401(k) planFun, modern workspace Respectful and nourishing work environment, where every opinion is heard and everyone is encouraged to be an active part of the organizational cultureAbout usZeroFox is on a mission to make the internet safer for all. Innovation is at our core – we are relentless in the pursuit of finding new ways to disrupt external cyber threats on the surface, deep, and dark web. ZeroFox offers the only unified cybersecurity platform combining advanced AI analytics, digital risk and privacy protection, full-spectrum threat intelligence, and a robust portfolio of breach, incident and takedown response capabilities to protect customers from growing threats across the external attack surface. It’s a great time to join us in the Fox Den – with fresh private equity funding, expanding investments in AI, a people-first culture, and centers of excellence around the world, we’re growing like never before. If you’re looking for a mission-oriented, customer-focused, collaborative team and ready to take the fight to the adversary, apply to join us in the Den today.Equal OpportunityWe aim to build a team that represents a variety of backgrounds, perspectives, and skills. We embrace inclusion and ensure equal employment opportunity without discrimination or harassment based on race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity or expression, age, disability, national origin, marital or domestic/civil partnership status, genetic information, citizenship status, military or veteran status, or any other personal characteristic.