Company Name: CG Infinity

Job Details: Hiring,Remotely,in,United,States,Remote,Senior,level

Job Url: https://builtin.com/job/data-engineer/7810785

Job Description: We are seeking an experienced Data Engineer to design, build, and maintain scalable data processing pipelines using modern big data technologies. The ideal candidate will have strong development skills and experience with distributed computing frameworks.Required Skills & Qualifications:Bachelor's degree in Computer Science, Engineering, or related technical field5+ years of software development experienceStrong proficiency in Python, Java or Scala programming languagesExtensive hands-on experience with Apache Spark, including:  Spark SQL  Spark Streaming and its data sources KAFKA, CDC feed, etc  Performance optimization and tuning  Data transformation and processingHands on Experience working with cloud-based Spark platforms (Databricks, AWS EMR, AWS Glue) Strong understanding of Hive, Unity catalog, Glue catalogHands on Experience working with any data quality framework/tools Hands on Experience working with Observability and monitoring for data processing pipelines Strong understanding of distributed computing conceptsProficiency in version control systems (Github/GitLab/Bitbucket)Technical Requirements:Experience building and maintaining high-volume data processing pipelinesKnowledge of data modeling and ETL / ELT best practicesFamiliarity with SQL and NoSQL databasesUnderstanding of data warehouse concepts and dimensional modelingPreferred Qualifications:Experience with real-time data processing and streaming architecturesKnowledge of Delta Lake or similar data lakehouse technologiesExperience with CI/CD pipelinesCloud platform expertise (AWS/Azure/GCP)Contributions to open-source projectsKey Responsibilities:Design and implement scalable data pipelinesOptimize existing data workflows for performance and reliabilityCollaborate with data scientists and analysts to support their data needsImplement data quality checks and monitoringMaintain documentation for data processes and architecturesSoft Skills:Strong problem-solving abilitiesExcellent communication skillsTeam collaborationAbility to work independentlyStrong attention to detailThe ideal candidate should have a proven track record of developing and maintaining production-grade data pipelines and be passionate about working with large-scale data systems.Please note this position requires an active IRS MBI clearance.