Data Engineer

Our Data Engineering role will be responsible for building and maintaining data processing pipelines and also the transformation and optimisation of data for analytical use. As Data Engineer, you'll be part of our experienced software dev function, working in a cross-functional Agile team.

We have opportunities for Data Engineers at every level within a team, so upon reviewing your application we will discuss the great opportunities for development or challenges we offer based off your professional profile.

Due to the interesting work we do and the sector this team is working in, we require all candidates to hold current eDV clearance.

Responsibilities

• Build data pipelines that clean, transform, and aggregate data from disparate sources
• Collaborate with stakeholders and other engineers
• Contribute to the completion of milestones associated with your project
• Contribute to continuous improvement within your team
• Collaborate with your peers on technical direction within your team

• Strong analytic skills related to working with unstructured datasets
• Python (PySpark, Pandas, PyArrow)
• Distributed data processing (Apache Spark)
• Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi)
• Cloud services (AWS, Azure or GCP)
• Messaging / Streaming (Kafka, AWS SQS, Other Cloud Queuing Native services)
• SQL and NoSQL databases and storage (HDFS, Iceberg, Elastic, S3, Data Lake)
• Containerisation and orchestration (Docker / Kubernetes / Openshift)
• Testing frameworks and best practices