Lead Data Engineer

Architect and own the entire end-to-end data extraction and ETL pipeline that converts unstructured web data into high-accuracy B2B datasets across 13 markets.

Design extractor frameworks and agentic extraction pipelines that handle complex challenges like anti-bot defences, JS rendering, schema drift, and perform cost and performance optimization of the sourcing layer.

Operate and maintain LLM infrastructure in production with eval harnesses, prompt versioning, drift detection, and lead orchestration using Airflow or equivalent to scale data workflows reliably.

6-10 years of experience shipping production extraction, ETL, or data pipeline systems; specifically with large-scale web extraction considering anti-bot, proxy, JS rendering, and schema drift.

Experience shipping LLMs as production systems within extraction pipelines, including structured outputs, prompt versioning, evaluation sets, and monitoring model drift.

Expertise in Python (production-grade, concurrency, performance-aware) and advanced SQL for complex transformations and optimization.

Extensive experience with Airflow or similar orchestration platforms for dependency management, recovery, and cost optimization at scale.

Proven ability to architect greenfield AI-native data extraction platforms that integrate LLMs, rule-based logic, and agentic workflows from first principles.

Strong product mindset focused on coverage, accuracy, and cost metrics influencing end-user outcomes rather than only operational uptime.

Comfortable working in lean, fast-moving, senior-level engineering teams with minimal process and autonomy over architecture decisions.