Lead Data Engineer with CICD & AI
Enable Data (company site, opens in a new tab)BengaluruRemoteDirect career pageEst. 20–35 LPA
Posted on 17 Sept
Reach people at Enable Data
People at this company you can reach
Log in to see who works here
Create a free account or log in to see the people at Enable Data you could write to:
- Recruiters & HR
- People in this function
- Work addresses
Your match
Against your primary resume
Log in to see your match score
Create a free account or log in to unlock your CV match score across:
- Role fit
- Experience
- Industry relevance
Protocol Intelligence
How competitive this role is
Competition
Background fit
Strictness
Log in to see why each signal reads the way it does.
Job description
Structured overview of role & requirements
About This Role
- Design, build, and maintain data ingestion pipelines into Databricks using PySpark and PySQL integrating sources like SAP and website scraping.
- Lead development and deployment of Databricks Apps, configure Unity Catalog permissions, and implement AI features including prompt engineering, Retrieval Augmented Generation, and Claude API integrations.
- Create and automate CI/CD pipelines for Teams Bot, web applications, and Databricks components; ensure security compliance with scans and validations.
Minimum Requirements
- Experience with Databricks including Unity Catalog, Service Principals, and Databricks Apps development.
- Proficiency in PySpark, PySQL, FastAPI, React, and prompt engineering for AI models (e.g., Claude).
- Hands-on experience in CI/CD pipelines automation using Azure DevOps including security validation steps (Snyk scans, linting).
- Work Experience Required: Not explicitly mentioned in the JD. Notice Period: Immediate joiners only.
Ideal Candidate Profile
- Has a strong technical foundation spanning data architecture, full stack development, and AI integration within cloud environments (Azure/Databricks).
- Operates effectively as an individual contributor in agile sprint cycles, able to deliver with limited documentation and close collaboration with cross-functional teams.
- Demonstrates expertise in debugging, code reengineering, and implementing AI safety guard rails tailored to Databricks and advanced NLP applications.