Data Engineering Lead - Python, Pyspark, Azure Databricks
Optum (company site, opens in a new tab)6+ yrsGurugramOn-siteDirect career pageEst. 25–40 LPA
Posted 1d ago
Reach people at Optum
People at this company you can reach
Log in to see who works here
Create a free account or log in to see the people at Optum you could write to:
- Recruiters & HR
- People in this function
- Work addresses
Your match
Against your primary resume
Log in to see your match score
Create a free account or log in to unlock your CV match score across:
- Role fit
- Experience
- Industry relevance
Protocol Intelligence
How competitive this role is
Competition
Background fit
Strictness
Log in to see why each signal reads the way it does.
Job description
Structured overview of role & requirements
About This Role
- Design, build, and maintain scalable batch and streaming data pipelines on Azure Databricks using Medallion Architecture with Delta Lake.
- Develop complex data transformations in PySpark/Spark SQL and implement data governance, lineage, and security including PHI/PII protection using Unity Catalog.
- Drive CI/CD and DataOps practices, troubleshoot production issues, optimize Spark jobs and Delta tables for performance/cost, and mentor junior engineers.
Minimum Requirements
- Bachelor's or Master's degree in Computer Science, IT or Engineering (B.E./B.Tech./MCA or equivalent).
- 6+ years of data engineering experience, including 3+ years hands-on with Azure Databricks.
- Hands-on experience with Delta Lake, Lakehouse architecture, Unity Catalog, and Azure services such as ADF, ADLS Gen2, Key Vault, Azure SQL, Power BI.
- Proficiency in Python, PySpark, SQL, Spark internals, performance tuning; experience with file formats like CSV, JSON, XML, Parquet, Avro, and schema evolution.
Ideal Candidate Profile
- Experienced in building and optimizing large-scale data pipelines using Azure Databricks and Delta Lake in a cloud environment with strong focus on data governance and security.
- Skilled in employing Medallion Architecture and data modeling to create consumption objects, with experience tuning Spark jobs and Delta tables for cost/performance efficiency.
- Able to lead and mentor junior engineers, collaborate across teams, and drive CI/CD and DataOps in an Agile delivery setup.