Big Data Engineer - Python and Spark

Responsible for redesigning and optimizing large-scale data processing and compute workflows using Python, Spark, and related Big Data technologies.

Handle complex data engineering tasks including working with unstructured, undocumented code to produce best-in-class scalable solutions.

Participate in both real-time and batch processing solutions, ensuring quality and performance in data management and computational systems.

Master’s or Engineering Degree with 0-2 years experience in Big Data systems including Hive, Hadoop, Spark (Python/Scala).

Hands-on experience with Unix scripting, Python, Scala programming, and strong SQL skills.

At least 3 years experience designing software systems requiring intensive computation across real-time and batch processing domains.

Experience working in onsite-offsite delivery models; knowledge of data governance, security, and regulatory practices.

Experienced in managing and engineering large datasets and data warehouses with strong data preprocessing and application engineering skills.

Capable of analyzing and solving complex business problems with clear articulation to management.

Familiar with supervised and unsupervised machine learning techniques and exposure to related ETL and performance management tools (e.g., Talend, Pepperdata, Cloudera).