Big Data Engineer

Head Resourcing ,
Glasgow, City of Glasgow
More Details
Contract Type: Permanent

Overview

I am currently looking for a Big Data Engineer to join a Software Services Company in Glasgow. Working within the Data Science team you will help design, implement and optimise processes to ingest data from a variety of sources including IoT devices using big data technologies This role will be working with high velocity data from some of the world's largest retailers, supporting ground-breaking BI and Artificial Intelligence use cases Key Responsibilities include Design and implement pipelines to ingest data into and consume data from the data lake Source and transform data from historical and streaming data sources Implement automated monitoring and alerting on pipelines to ensure consistent processing of data Design systems to allow real time analytics on high velocity data Proactively ensure optimum performance by revisiting design choices to check they are still fit for purpose Collaborate with Data Scientists and BI Developers to ensure the wider team's objectives are being met by data lake design Support wider team in consuming data from the data lake Be able to provide technical leadership in the use and design of big data technologies In order to be successful in the role candidates need to have Experience utilising the following technologies, preferably in the Azure platform: Azure Data Lake Azure Data Factory Databricks Python/PySpark or Scala Data pipeline design, and optimisation Optimising data lake file storage Optimising data lakes for real time advanced analytics Time management, planning and organisation If you would like to find out more, then please send across an up-to-date CV at your earliest convenience

People also viewed