Match Score
Against your primary resumeLogin to See Your Match Score
Create a free account or log in to unlock your CV match score across:
Protocol Intelligence
Data-driven signals on your job's competitivenessSenior, niche Databricks lead role at a smaller company reduces candidate density.
Databricks and Spark specialization significantly limits transferable candidate backgrounds.
Requires 9+ years and Databricks lead experience, making filters strict.
Job Description
Structured overview of role & requirementsAbout This Role
Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks, PySpark, and SQL, including data integration from multiple sources such as databases, Amazon S3, and REST APIs.
Manage and optimize Databricks environments including Unity Catalog, Delta Lake tables with Medallion Architecture, Jobs, and Workflows to ensure data quality, performance, and scalability.
Lead technical decisions and collaborate with cross-functional teams to deliver production-ready data engineering solutions, serving as a technical lead for the data engineering team.
Minimum Requirements
9+ years of experience in Data Engineering with at least 3 years hands-on experience on Databricks platform.
Strong expertise in Python, PySpark, advanced SQL, and data modeling concepts including Star/Snowflake schemas and SCD.
Experience integrating with REST APIs and managing structured and semi-structured data formats (CSV, JSON, Parquet, Delta).
Prior experience in a Lead Data Engineer or Technical Lead role is mandatory.
Ideal Candidate Profile
Experienced in managing and optimizing data pipelines in large-scale, production Databricks environments with strong focus on Spark performance and Medallion Architecture.
Capable of technical leadership including guiding engineers, driving design decisions, and collaborating across teams for deliverables.
Deep understanding of modern data warehousing, ETL/ELT development best practices, and hands-on experience with CI/CD, Git, and Databricks toolsets including Unity Catalog and Workflows.