Lead Software Engineer - Data Engineer + Python + Spark/Pyspark+ AWS

Lead design, build, and operation of scalable, reliable batch and streaming data platforms and pipelines powering critical analytics and AI-assisted workflows.

Accountable for building curated domain datasets with clear contracts, lineage, measurable SLAs/SLOs, and strong controls for security, privacy, and auditability.

Hands-on engineering leadership setting technical standards across Java and Python implementations, orchestration, transformation frameworks, and delivering APIs and search/indexing pipelines with operational responsibility in a regulated enterprise.

5+ years applied experience with formal software engineering training or certification.

Proven hands-on experience as lead Data Engineer delivering production-grade curated datasets and pipelines end-to-end.

Strong proficiency in Java and Python for production-grade data systems, advanced SQL skills, experience with pipeline orchestration tools like Airflow, and transformation frameworks like dbt.

Experience with secure engineering practices including authentication/authorization, secrets management, least privilege, and policy enforcement; demonstrated leadership in AI-assisted software development tooling adoption.

Engineering leader with recent exposure to designing and operating large-scale batch and streaming data platforms in regulated environments with strong operational rigor.

Technically versatile in Java and Python with advanced SQL and expertise in orchestration (Airflow), transformation frameworks (dbt), and search/indexing technologies (Elasticsearch).

Experienced in integrating AI-assisted development tools responsibly and enabling AI/ML workflows securely, with cross-functional collaboration across product, UX, platform, and controls teams.