Match Score
Against your primary resumeLogin to See Your Match Score
Create a free account or log in to unlock your CV match score across:
Protocol Intelligence
Data-driven signals on your job's competitivenessSenior Databricks specialization and 8+ years requirement reduce candidate competition.
Databricks/Spark specialization reduces transferability, though core data engineering skills remain fairly transferable.
Explicit 8+ years, required Databricks/Spark/Python expertise, and leadership demands raise screening strictness.
Job Description
Structured overview of role & requirementsAbout This Role
Lead the design, development, and implementation of scalable data pipelines on Databricks for an enterprise transportation client.
Provide technical leadership and mentorship to a team of data engineers, enforce best practices, and conduct code reviews to ensure high-quality data solutions.
Optimize Spark workloads, manage Databricks features (Delta Lake, Unity Catalog, Delta Live Tables, Photon, SQL Analytics), and ensure operational excellence through monitoring and troubleshooting.
Minimum Requirements
8+ years of data engineering experience with at least 2-3 years in a lead or senior capacity.
Extensive hands-on experience with Databricks platform including Spark, Delta Lake, Unity Catalog, and related features such as DLT and Photon.
Strong proficiency in Python (PySpark) and SQL; experience with cloud platforms (AWS, Azure, or GCP) and their data services.
Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field.
Ideal Candidate Profile
Proven ability to architect and optimize large-scale data platforms specifically using Databricks and Spark technologies.
Experienced in leading technical teams with a focus on coding standards, architectural patterns, and quality assurance.
Familiarity with modern data engineering tools including workflow orchestration (Apache Airflow, Databricks Workflows), CI/CD pipelines, and version control (Git).