Lead Data Engineer - Databricks
ShyftLabs (company site, opens in a new tab)9+ yrsCoimbatoreHybridDirect career pageEst. 25–40 LPA
Posted on 22 Sept
Reach people at ShyftLabs
People at this company you can reach
Log in to see who works here
Create a free account or log in to see the people at ShyftLabs you could write to:
- Recruiters & HR
- People in this function
- Work addresses
Your match
Against your primary resume
Log in to see your match score
Create a free account or log in to unlock your CV match score across:
- Role fit
- Experience
- Industry relevance
Protocol Intelligence
How competitive this role is
Competition
Background fit
Strictness
Log in to see why each signal reads the way it does.
Job description
Structured overview of role & requirements
About This Role
- Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks, PySpark, and SQL, including data integration from multiple sources such as databases, Amazon S3, and REST APIs.
- Manage and optimize Databricks environments including Unity Catalog, Delta Lake tables with Medallion Architecture, Jobs, and Workflows to ensure data quality, performance, and scalability.
- Lead technical decisions and collaborate with cross-functional teams to deliver production-ready data engineering solutions, serving as a technical lead for the data engineering team.
Minimum Requirements
- 9+ years of experience in Data Engineering with at least 3 years hands-on experience on Databricks platform.
- Strong expertise in Python, PySpark, advanced SQL, and data modeling concepts including Star/Snowflake schemas and SCD.
- Experience integrating with REST APIs and managing structured and semi-structured data formats (CSV, JSON, Parquet, Delta).
- Prior experience in a Lead Data Engineer or Technical Lead role is mandatory.
Ideal Candidate Profile
- Experienced in managing and optimizing data pipelines in large-scale, production Databricks environments with strong focus on Spark performance and Medallion Architecture.
- Capable of technical leadership including guiding engineers, driving design decisions, and collaborating across teams for deliverables.
- Deep understanding of modern data warehousing, ETL/ELT development best practices, and hands-on experience with CI/CD, Git, and Databricks toolsets including Unity Catalog and Workflows.