Job Title: Senior Data Engineer

Company Name: Healthmap Solutions

Job Details: RemoteFull,Time

Job Url: https://hiring.cafe/viewjob/tgsxouc41mn4wamj

Job Description: Posted 1mo agoSenior Data Engineer@ Healthmap SolutionsView All JobsWebsiteUnited StatesRemoteFull TimeResponsibilities:design pipelines, optimize jobs, migrate dataRequirements Summary:5+ years data engineering with focus on data warehousing, ETL/ELT, distributed systems; 3+ years on Databricks; PySpark/SQL; Delta Lake; AWS; CI/CD; healthcare experience preferred.Technical Tools Mentioned:Python, Spark, Databricks, SQL, Delta Lake, AWS, S3, Unity Catalog, Photon, CI/CD
Position SummaryThe Senior Data Engineer drives the design, development, and operational excellence of our data platform built on Databricks and the Delta Lakehouse Architecture. This role requires deep expertise in scalable ETL/ELT, Spark optimization, and modern data governance using Unity Catalog. The Senior Data Engineer serves as a technical leader, establishing best practices for code quality, performance, and automated deployments across all environments.ResponsibilitiesDesign and implement scalable data pipelines using Delta Lake and manage enterprise-wide data access, security, and lineage using Unity CatalogOptimize large-scale Spark jobs (PySpark/SQL) and cluster configurations (Photon) to meet stringent SLA and cost performance targets across all workflowsBuild resilient data scheduling via Databricks Workflows (Jobs) and establish automated CI/CD pipelines for reliable code promotion across Dev, Staging, and Prod workspacesMigrate data and models from relational databases to DatabricksEnsure best practices for development using industry standard development patternsMonitor data pipelines performancePartner with Data Management and Full Stack development engineers to operationalize models with current applications and processesSupport existing data pipelines to ensure business continuityStay updated with the latest trends and technologies in data engineering and cloud computingPerform other related duties as assignedRequirementsBachelor’s degree is required5+ years of experience in Data Engineering, with a significant focus on data warehousing, ETL/ELT development, and distributed systems3+ years of hands-on experience developing enterprise solutions on the Databricks platformExpertise in PySpark and high-performance SQLDeep understanding practical knowledge of Delta Lake architecture and optimal maintenance best practicesExperience with cloud platforms (AWS preferred) and integrating Databricks with native cloud services (S3, Secret Manager, IAM)Solid experience implementing CI/CD for Databricks notebooks and associated librariesHealthcare experience is preferredSkillsStrong development experience with: Python, Spark, or similar technologiesExcellent verbal, communication, negotiation, and presentation skillsStrong analytical and problem-solving skillsAbility to work autonomously while collaborating across teams to deliver timely projectsAbility to explain complex concepts in simple termsDedicated, hardworking employee who achieves maximum efficiency and productivityStrong knowledge of domain-based design, data modeling and data structuresStrong knowledge of best practice in data managementNo Travel#LI-Remote