Lead Software Engineer - Pyspark, AWS, Python

Lead design, development, and maintenance of scalable cloud-based data processing pipelines and infrastructure adhering to engineering standards and governance.

Architect and optimize data models for large-scale datasets ensuring data quality, efficient storage, and high-performance analytics.

Drive data engineering strategy and collaboration with cross-functional teams to deliver technically sound and strategically relevant data solutions.

5+ years of applied software engineering experience with formal training or certification.

Expertise in distributed data processing frameworks like Spark and cloud data lakehouse platforms (AWS Data Lake services or Databricks).

Proficiency in Python, SQL, and at least one additional programming language (Java or Scala) with hands-on Apache Spark experience.

Expertise in orchestration tools (Airflow or AWS Step Functions), relational and NoSQL databases, data modeling techniques, plus experience with microservices, containerization (Docker, Kubernetes), and test-driven development (TDD/BDD).

Track record of leading design workshops and coding sessions fostering innovation in data engineering.

Experienced in architecting reusable, future-ready design patterns and working with streaming platforms like Kafka.

Proficient in leveraging AI-assisted software development tools with sound understanding of responsible AI use, secure data handling, and engineering resiliency.