IN_Manager_Databricks Data Engineer_GCC_Advisory_Bangalore

Design, build, and maintain scalable, high-performance data pipelines on the Databricks Lakehouse Platform using Apache Spark, PySpark, and Delta Lake.

Collaborate with analytics, AI/ML, and business teams to support data consumption, ensure data quality, governance, and optimize performance and cost efficiency of Spark jobs.

Provide production support, troubleshoot data pipeline issues, document technical designs, and mentor junior engineers following enterprise security and compliance standards.

7+ years of experience as a Data Engineer with strong expertise in Databricks, Apache Spark, PySpark, and Spark SQL.

Bachelor's or Master's degree in Computer Science, Engineering, or related field with 60% or above; degrees preferred in Bachelor of Engineering or Bachelor of Technology.

Experience with Delta Lake, data warehousing, ETL/ELT patterns, and exposure to at least one cloud platform: Azure, AWS, or GCP.

Advanced SQL skills and understanding of distributed computing concepts.

Experienced in designing and operating end-to-end data pipelines on Databricks with proficiency in Delta Lake and Lakehouse architecture.

Familiarity with AI/ML workflows, data governance (Unity Catalog), and automation including CI/CD for data pipelines.

Capable of optimizing Spark jobs for performance and cost, mentoring junior staff, and managing production support in an Agile environment.