Lead Data Engineer
Firmable6–10 yrsRemoteDirect career pageEst. 28–45 LPA
Posted 21d ago
Your match
Against your primary resume
Log in to see your match score
Create a free account or log in to unlock your CV match score across:
- Role fit
- Experience
- Industry relevance
Reach people at Firmable
People at this company you can reach
Log in to see who works here
Create a free account or log in to see the people at Firmable you could write to:
- Recruiters & HR
- People in this function
- Work addresses
Protocol Intelligence
How competitive this role is
Competition
Background fit
Strictness
Log in to see why each signal reads the way it does.
Job description
Structured overview of role & requirements
About This Role
- Architect and own the entire end-to-end data extraction and ETL pipeline that converts unstructured web data into high-accuracy B2B datasets across 13 markets.
- Design extractor frameworks and agentic extraction pipelines that handle complex challenges like anti-bot defences, JS rendering, schema drift, and perform cost and performance optimization of the sourcing layer.
- Operate and maintain LLM infrastructure in production with eval harnesses, prompt versioning, drift detection, and lead orchestration using Airflow or equivalent to scale data workflows reliably.
Minimum Requirements
- 6-10 years of experience shipping production extraction, ETL, or data pipeline systems; specifically with large-scale web extraction considering anti-bot, proxy, JS rendering, and schema drift.
- Experience shipping LLMs as production systems within extraction pipelines, including structured outputs, prompt versioning, evaluation sets, and monitoring model drift.
- Expertise in Python (production-grade, concurrency, performance-aware) and advanced SQL for complex transformations and optimization.
- Extensive experience with Airflow or similar orchestration platforms for dependency management, recovery, and cost optimization at scale.
Ideal Candidate Profile
- Proven ability to architect greenfield AI-native data extraction platforms that integrate LLMs, rule-based logic, and agentic workflows from first principles.
- Strong product mindset focused on coverage, accuracy, and cost metrics influencing end-user outcomes rather than only operational uptime.
- Comfortable working in lean, fast-moving, senior-level engineering teams with minimal process and autonomy over architecture decisions.