Lead Software Engineer - Java/Python, AWS, Spark

Lead design, development, and maintenance of scalable, cloud-based data processing pipelines and infrastructure, ensuring compliance with engineering standards and governance.

Architect data models for large-scale datasets to optimize storage, retrieval, and advanced analytics while maintaining data integrity and quality.

Define and execute enterprise data strategy including end-to-end data infrastructure management, driving data quality initiatives and compliance with governance and regulatory requirements.

5+ years of applied experience with formal training or certification in software engineering concepts.

Expertise in Spark or equivalent distributed data processing framework and AWS Data Lake services or Databricks (cloud data Lakehouse platform).

Expertise with scheduling/orchestration tools (Airflow preferred), relational and NoSQL databases, and programming in Python and at least one other language (Java or Scala).

Experience with microservices, serverless computing, containerization tools (Docker, Kubernetes), data modeling techniques, and CI/CD with test-driven or behavior-driven development.

Experienced technical leader capable of driving architecture and design workshops, fostering innovation and team adoption of AI-assisted software development tools.

Strong data engineering background in large-scale batch and real-time data processing, including expertise in streaming platforms (Kafka, MQ) and reusable design patterns.

Demonstrated ability to align data engineering solutions with business objectives and compliance requirements, with knowledge of responsible AI use and security in engineering workflows.