Data Engineer

Design and develop scalable, high-performance data pipelines on the Databricks Lakehouse Platform to support Master Data Management functionalities including entity resolution, data quality, and analytics.

Build and optimize batch and real-time streaming data pipelines using Delta Live Tables and Change Data Capture/Feed patterns to ensure data reliability, consistency, and performance.

Collaborate with AI/ML Engineers, Product Managers, and Data Scientists to translate data requirements into efficient data models and maintain best practices in version control, automated testing, and monitoring.

5+ years of hands-on experience as a Data Engineer or equivalent role managing large-scale data platforms and pipelines.

Strong expertise with Databricks Lakehouse Platform components: Delta Lake, Databricks SQL, Unity Catalog, and Databricks Workflows.

Proficiency in building and optimizing Apache Spark data pipelines, specifically using PySpark and streaming technologies like Delta Live Tables.

Experience with data modeling including dimensional modeling (Kimball) and Slowly Changing Dimensions (SCDs), and knowledge of Change Data Capture (CDC) and Change Data Feed (CDF) patterns.

Experienced in architecting and optimizing complex batch and streaming data pipelines within Databricks environment for enterprise-grade Master Data Management systems.

Skilled in developing robust, testable, and idempotent data pipelines focused on data quality and operational reliability at scale.

Able to work cross-functionally with technical and product teams to design data infrastructure that supports analytics and intelligent features.