Company Name: RippleMatch

Job Details: Hiring,Remotely,in,USA,Remote,Mid,level

Job Url: https://builtin.com/job/mercor-data-scientist-application-ripplematch/7686457

Job Description: This role is with Mercor. Mercor uses RippleMatch to find top talent. Role OverviewMercor is seeking a data-driven analyst to conduct comprehensive failure analysis on AI agent performance across finance-sector tasks. You'll identify patterns, root causes, and systemic issues in our evaluation framework by analyzing task performance across multiple dimensions (task types, file types, criteria, etc.).Key ResponsibilitiesStatistical Failure Analysis: Identify patterns in AI agent failures across task components (prompts, rubrics, templates, file types, tags)Root Cause Analysis: Determine whether failures stem from task design, rubric clarity, file complexity, or agent limitationsDimension Analysis: Analyze performance variations across finance sub-domains, file types, and task categoriesReporting & Visualization: Create dashboards and reports highlighting failure clusters, edge cases, and improvement opportunitiesQuality Framework: Recommend improvements to task design, rubric structure, and evaluation criteria based on statistical findingsStakeholder Communication: Present insights to data labeling experts and technical teamsRequired QualificationsStatistical Expertise: Strong foundation in statistical analysis, hypothesis testing, and pattern recognitionProgramming: Proficiency in Python (pandas, scipy, matplotlib/seaborn) or R for data analysisData Analysis: Experience with exploratory data analysis and creating actionable insights from complex datasetsAI/ML Familiarity: Understanding of LLM evaluation methods and quality metricsTools: Comfortable working with Excel, data visualization tools (Tableau/Looker), and SQLPreferred QualificationsExperience with AI/ML model evaluation or quality assuranceBackground in finance or willingness to learn finance domain conceptsExperience with multi-dimensional failure analysisFamiliarity with benchmark datasets and evaluation frameworks2-4 years of relevant experienceWe consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.