Senior Developer, Data Engineer
Intercontinental ExchangeMatch Score
Against your primary resumeLogin to See Your Match Score
Create a free account or log in to unlock your CV match score across:
Protocol Intelligence
Data-driven signals on your job's competitivenessLog in to see why each signal reads the way it does.
Job Description
Structured overview of role & requirementsAbout This Role
Own and maintain the end-to-end Apache Airflow orchestration layer, including DAG development, testing, and operational excellence on Kubernetes.
Design, build, and support robust ETL/ELT pipelines and a lakehouse architecture using Apache Iceberg and Databricks ensuring data quality, freshness, and governance.
Collaborate with data scientists and ML engineers to deliver reproducible feature pipelines and contribute reusable components and platform improvements.
Minimum Requirements
Minimum 3 years of production experience running Apache Airflow, including scheduler internals and executor types.
Proven experience managing Airflow deployments on Kubernetes (KubernetesExecutor or CeleryKubernetesExecutor) and Helm-based upgrades.
Strong Python skills applied to DAG authoring, with emphasis on testability and dynamic task management.
Work Experience Required: Not explicitly mentioned in the JD
Ideal Candidate Profile
Deep expertise in data engineering at scale with hands-on knowledge of lakehouse architectures and Apache Iceberg table management.
Experience working in collaborative, cross-functional teams combining data engineering with ML and data science workflows.
Operator and strategic mindset capable of owning infrastructure tool choices, automation, runbooks, and mentoring others in engineering standards.