Key Responsibilities
As a Data Engineer, you apply broad knowledge and technical know-how in the Data Engineering domain to deliver the following responsibilities:
Validate design, build, test and deploy software solutions into production
Support your team to develop innovative software that maintains data/network security management, leveraging set processes to reduce the time between commission and production
Collaborate effectively with your team to translate customer needs into leading-edge software application products, services or systems that meet company’s time, cost and quality standards
Seek diverse viewpoints from colleagues in your team, and apply analytical thinking to resolve complex problems that don’t have an immediate or ‘tried-and-tested’ solution
Contribute to continuous improvement initiatives of our systems and processes, apply ‘best practice’ corporate testing standards and our quality assurance approach to drive improved outcomes (e.g. productivity, customer experience and/or profitability)
Apply a continuous learning and development mindset to your work, using tried and tested methodologies but also finding opportunities to experiment to incrementally improve or resolve challenges
Take ownership of your own actions and seek guidance from more experienced engineers in order to support your development and contribute to the delivery of team/mission outcomes

Essential
Hands-on experience in the following:
Spark Core, Spark SQL, SQL/Hive/Impala, Git/SVN/Any other VCS and Data warehousing (AWS Redshift)
Skilled in the following:
Hadoop Ecosystem(HDP/Cloudera/MapR/EMR etc)
Azure data factory/Airflow/control-M/Luigi
PL/SQL
Exposure to NOSQL(Hbase/Cassandra/GraphDB(Neo4J)/MongoDB)
File formats (Parquet/ORC/AVRO/Delta/Hudi etc.)
Kafka/Kinesis/Eventhub

Highly Desirable
Experience and knowledgeable on the following:
Spark Streaming
Cloud exposure (Azure/AWS/GCP)
Azure data offerings – ADF, ADLS2, Azure Databricks, Azure Synapse, Eventhubs, CosmosDB etc.
Presto/Athena
Azure DevOps
Jenkins/ Bamboo/Any similar build tools
Power BI
Prior experience in building or working in team building reusable frameworks,
Data modelling.
Data Architecture and design principles. (Delta/Kappa/Lambda architecture)
Exposure to CI/CD
Code Quality – Static and Dynamic code scans
Agile SDLC