Job Title: Senior Data Engineer

Company Name: Employers Holdings

Job Details: $90k-$135k/yrRemoteFull,Time

Job Url: https://hiring.cafe/viewjob/l7mac83yoku9f6fo

Job Description: Posted 3w agoSenior Data Engineer@ Employers HoldingsView All JobsWebsiteUnited States$90k-$135k/yrRemoteFull TimeResponsibilities:design pipelines, build pipelines, maintain pipelinesRequirements Summary:7+ years data engineering; Python, PySpark; Databricks stack; Delta Lake; Airflow; BI data layers; ML pipeline support; regulated industry experience a plus.Technical Tools Mentioned:Python, PySpark, Databricks, Delta Live Tables, Databricks Workflows, Unity Catalog, SQL Warehouses, Delta Lake, Airflow, Git, CI/CD, Infrastructure as Code
General SummarySenior Data Engineers are responsible for designing, developing, and maintaining scalable data platforms and pipelines that align with business objectives and drive operational efficiency. They analyze complex data challenges, architect reliable and secure data systems, and leverage advanced tools and technologies to meet evolving organizational needs.This role centers on the Databricks ecosystem and Python-based data engineering, with a strong emphasis on building production-grade data pipelines, optimizing data workflows, and enabling analytics and reporting capabilities across the organization.Key to success in this role is the ability to collaborate across cross-functional teams, apply critical thinking, and translate business requirements into actionable, high-impact data solutions. Familiarity with machine learning pipelines and feature engineering within a data engineering context is a plus.Analytics FocusResponsible for supporting the development of data pipelines and advanced queries leveraged by the business intelligence team as it develops and deploys dashboards and reports to support data-driven decision-making across the company.Essential Duties and ResponsibilitiesDesign, develop, and maintain scalable, performant data pipelines using Python, PySpark, and Databricks, following Medallion architecture patterns (Bronze, Silver, Gold).Build and optimize Delta Lake tables, manage schema evolution, and implement data quality checks across ingestion and transformation layers.Develop and maintain Airflow DAGs for pipeline orchestration, scheduling, and monitoring.Collaborate with product management and business stakeholders to interpret requirements and define data specifications.Create and maintain semantic data layers and curated datasets for BI platforms (e.g., Power BI, QuickSight).Analyze trade-offs between commercial and custom-built data solutions, offering technical guidance to decision-makers.Troubleshoot and resolve issues across data workflows, pipeline failures, and infrastructure.Support machine learning initiatives by building feature pipelines, data preparation workflows, and serving layers within the Databricks platform.Ensure compliance with legal, regulatory, and internal governance requirements across data solutions.Automate operational processes using CI/CD pipelines and infrastructure-as-code principles.Establish and promote best practices for data engineering, pipeline development, and monitoring.Partner with teams across the organization to translate business needs into technical data solutions.Create and maintain detailed technical documentation including data lineage diagrams, pipeline architecture, and workflow documentation.Serve as a subject matter expert in data engineering, pipeline optimization, Databricks platform capabilities, and data architecture.Provide mentorship to junior team members, fostering professional growth and collaboration.Communicate effectively with leadership and stakeholders regarding project status, risks, and milestones.Stay current on emerging data engineering technologies and industry trends to drive continuous improvement and innovation.Requirements — Technical Skills7+ years of experience in data engineering or related roles.Strong proficiency in Python and PySpark for data pipeline development and scripting.Deep hands-on experience with Databricks, including Delta Live Tables (DLT), Databricks Workflows, Unity Catalog, and SQL Warehouses.Strong knowledge of Medallion architecture (Bronze/Silver/Gold) within Databricks.Solid understanding of Lakehouse and Delta Lake concepts, including ACID transactions, time travel, and schema enforcement.Proficiency in Apache Airflow for workflow orchestration and scheduling.Strong knowledge in relational databases, SQL query optimization, and multiple database systems.Experience creating semantic data layers for BI platforms (e.g., Power BI, QuickSight).Experience combining internal and external data sources for in-depth industry and business analyses.Understanding of cloud computing platforms and integration with on-premise solutions.Familiarity with CI/CD pipelines, version control (Git), and infrastructure-as-code practices.Experience supporting ML pipelines and feature engineering within a data platform context is a plus.Strong understanding of testing methodologies, including data validation, pipeline testing, and automation.Experience working in regulated industries (e.g., insurance, banking, finance, healthcare).Databricks certification preferred (e.g., Databricks Certified Data Engineer Associate or Professional).Requirements — Professional SkillsOutstanding analytical and problem-solving skills with the ability to simplify complex tasks and absorb new information rapidly.Strong written and verbal communication skills, with the ability to convey technical information clearly to both technical and non-technical audiences.Highly self-motivated, able to take initiative and manage time effectively to meet deadlines and deliverables.Strong attention to detail with the ability to work independently with minimal supervision.Ability to manage multiple projects simultaneously in a fast-paced, evolving environment.Exceptional research skills with the ability to gather, analyze, and synthesize complex information quickly.Passion for delivering exceptional quality and continuous learning of new technologies and frameworks.Strong listening skills and the ability to build positive relationships with colleagues, stakeholders, and customers.Commitment to creating an inclusive, team-oriented environment that aligns with company culture and values diversity. Work EnvironmentRemote: This position is a fully remote (work‑from‑home) opportunity.Candidates must be currently located in the United States and authorized to work without sponsorship.A suitable home workspace that is private and free from distractions is required.Expected Work Hours: Work schedules are designed to meet both the needs of the role and the organization and may be adjusted as business needs evolve.Travel: Possible travel to off‑site locations may be required for meetings, as needed.Salary Range : $90,000 - $135,000 and a comprehensive benefits package, please follow the link to our benefits page for details! EMPLOYERS Benefits and PerksAbout EMPLOYERS:As a dynamic, fast-growing provider of workers' compensation insurance and services, we are seeking a goal-oriented individual willing to put their ideas to work!We offer a positive, challenging work environment, combined with an opportunity to build your career as you help us grow our business, in innovative and imaginative ways that are uniquely EMPLOYERS®!Headquartered in Nevada, EMPLOYERS attributes its long-standing success to its most valuable resource, our employees across the United States.  EMPLOYERS is known for the quality service and expertise we provide to our clients, and the exemplary work environment we provide for our employees.We live and breathe our core values: Integrity, Customer Focus, Collaboration, Initiative, Accountability, Innovation, and Personal Fulfillment.  These are the pillars that support how we do business with our clients as well as how we treat each other!At EMPLOYERS, you’ll discover an energetic environment that inspires top achievement.  As “America’s small business insurance specialist”, we have the resources, a solid reputation and an expanding nationwide identity to enrich your work/life and enhance your career. #LI-RemoteClick our link to learn more! About EMPLOYERS