Company Name: Jump - Advisor AI

Job Details: Hiring,Remotely,in,USA,Remote,75K-90K,Annually,Mid,level

Job Url: https://builtin.com/job/qa-engineer-generative-ai/7874683

Job Description: Title: QA Engineer for Generative AIReports To: QA manager, who reports to Parker Wightman, Senior Director of EngineeringLocation: Draper, UT / Remote (USA)About The RoleJump is looking for a US–based QA engineer to help coordinate and run data labeling/annotation campaigns used to improve our AI/ML systems and evaluate/review  production system outputs,  such as meeting notes, recap emails, and tasks; answers in our Ask Anything feature; our pre-meeting prep product; and our AI agents.This role blends process design and hands‑on testing. You’ll use AI evaluation rubrics prepared by our product managers or data team to improve our products so our customers get accurate transcripts, summaries, and action items every time they interact with Jump. You’ll go deep into AI best practices and limitations.You’ll partner closely with Engineering, Product, and Customer teams to ship quickly and confidently. Familiarity with Jump (as a user, beta tester, or close to advisor workflows) is a big plus. If you already have AI evaluation experience, that’s great—but it’s not required. We’ll teach you our approach; candidates with AI evaluation experience will be compensated accordingly.What You’ll DoServe as the embedded QA engineer on two pods (Jump’s cross-functional teams), collaborating with product managers to evaluate AI outputs, run exploratory and regression testing, and unblock engineers and PMs.Learn and track AI/ML quality signals, including golden datasets, prompt/regression suites, and metrics such as WER, diarization accuracy, action-item precision/recall, summary faithfulness, hallucination rate, and PII handling.Build dashboards for quality KPIs (defect escape rate, flake rate, regression coverage, MTTD/MTTR, AI eval scores) and drive continuous improvement.Partner with Product and Engineering to ensure requirements are testable, edge cases are captured, and AI evaluation rubrics are clear and repeatable.Foster a no-drama, direct-and-kind culture that moves with high-quality velocity.About You3+ years in QA or Quality Engineering for SaaS productsStrong exploratory testing skills and clear, concise written communication for reproducing issuesCuriosity and aptitude to learn ML/AI evaluation (prompt testing, golden sets, offline evals, safety/guardrails)Familiarity with AI prompts, LLMs, and the Jump product (as a user or employee)You don’t need a traditional STEM background to excel here. You’ll thrive if youGet excited about spotting patternsHave a strong grasp of human language and thought processesYou might have a background inEditingTechnical writingNice-to-haves:Comfortable reading software system logs and finding patterns in messy dataFamiliarity with fintech or other regulated environmentsExperience with BigQuery or other data warehousesExperience with web API testingBasic familiarity with query languages, relational databases, and other data storage systemsWhat You’ve DoneBuilt or scaled a QA function (process, tooling, reporting), or partnered with product managers and engineers to identify and resolve AI-related bugsWritten great documentation, bug reports, or other clear technical writingInteracted meaningfully with LLMs and AI outputsNice-to-haves:Designed and executed AI evaluation workflows (golden datasets, human-in-the-loop scoring, clear rubrics)—a plus but not required; candidates with this experience may be considered for higher compensationCreated risk-based test plans and lightweight automation that caught regressions earlyAbout JumpJump is empowering financial advisors and their clients to thrive in the age of AI. We're growing incredibly quickly with a team that comes from Google, JP Morgan, BILL, Snowflake, Fidelity, Bain Capital, Harvard, Stanford and other top companies and schools. We can't wait to hear from you!Jump’s cultureHigh VelocityWorld ClassDirect + Kind + No DramaWe believe in building tight teams of extraordinarily capable people. Come join us to transform the advisor and enterprise experience with state-of-the-art technology.CompensationSalary: $75k to 90k (DOE)EquityHealth benefits