Job Purpose

Responsible for developing, maintaining and optimising MPM’s data infrastructure. Provide technical expertise to ensure efficient data handling, ensuring availability, scalability and security of the centralised data asset. Ensure that all ETL pipeline and scheduled jobs run efficiently and effectively. To build a centralised data asset in which data is stored logically. Train colleagues in the use and capability of Azure Databricks.

Key Responsibilities

This role deals with a wide variety of tasks, including but not limited to the following:

• Lead, design and maintain scalable ETL pipelines that ingest, clean and aggregate raw data from various sources such as APIs, storing data appropriately
• Develop and implement medallion (Bronze/Silver/Gold) architecture across Azure Databricks
• Monitor and control costs related to data storage, compute, and other services to ensure efficient and cost-effective use of resources
• Automate manual tasks by leading data-driven solutions, transforming processes into efficient, system-driven operations that improve productivity and accuracy
• Create and update technical documentation, such as data dictionaries and catalogues, through Unity Catalog to support the data infrastructure through large language model (LLM) functionalities such as Genie
• Establish strong data governance, ensuring colleagues have the level of data access that they require
• Build capability across the data function by coaching and upskilling colleagues (pairing, knowledge-sharing, and clear documentation), helping reduce single points of failure and improving consistency.
• Coordinate delivery of assigned data workstreams end-to-end, working closely with stakeholders and the wider team to agree scope, manage actions and dependencies, and deliver high-quality outcomes.
• Be an advocate for a “data-first” approach across the business, championing the use of Databricks to create a single source of truth for data and metrics across the business
• Contribute to best practices and continuous improvement of the Databricks function

Knowledge, Skills and Experience

Essential

Background in Data Engineering recognised through certification (Databricks Data Engineer Associate or similar) or through demonstratable experience

Experience building and maintaining data pipelines in Databricks

Experience working with cloud environments, preferably Azure

Experience in Data Warehousing, Data Lakes, Blob Storage, Lakehouses and Data Modelling concepts (Kimball, star schemas)

Strong ability to write Python code, including experience with data manipulation libraries (PySpark, Pandas etc)

Strong ability to write SQL queries for data retrieval and manipulation

Strong problem-solving skills to address and resolve data-related issues.

Effective communicator at a technical & non-technical level.

IT literate use of Microsoft applications.

High attention to detail.

Desirable

Ability to lead and manage projects, including scoping, planning, and executing tasks to meet deadlines.

Experience with source control and CI/CD (GitHub, Azure DevOps, DABs)

Data architecting, solutions architecting and experience optimising pipelines and Delta Lake

Databricks Data Engineer Professional certified