Lead Software Engineer - Pyspark, AWS, Python

Lead design, development, and maintenance of scalable cloud-based data processing pipelines and infrastructure, ensuring industry best practices and data governance compliance.

Architect and optimize large-scale data models and infrastructure for efficient analytics and high data quality.

Drive data strategy execution, process re-engineering, and adoption of AI-assisted engineering tools to improve code quality, delivery speed, and operational outcomes.

5+ years of applied experience with formal training or certification in software engineering concepts.

Expertise in distributed data processing frameworks (Spark), cloud data lake platforms (AWS, Databricks), and scheduling/orchestration tools (Airflow or similar).

Proficiency in Python, SQL, and at least one additional programming language such as Java or Scala.

Hands-on experience with microservices, serverless computing, distributed cluster tools (Docker, Kubernetes), data modeling techniques, CI/CD, test-driven/behavior-driven development, and streaming platforms like Kafka.

Experienced technical leader capable of architecting reusable, future-ready data engineering solutions across diverse organizational use cases.

Demonstrated ability to lead cross-functional teams, organize design workshops, and promote a culture of excellence and innovation in data engineering.

Proficient in responsible AI-assisted development workflows, balancing automation with security, compliance, and team coaching.