Senior Data Platform Engineer CR 12/11/26

We are looking for a Senior Data Platform Engineer to help drive the evolution of our Azure-based Data Platform and

Lakehouse architecture.

This role is focused on the platform itself

. The professional will own and evolve the foundations that data product teams build on:

Apache Airflow orchestration (migrating from ADF), Azure resource management

(Key Vault, identity and access, networking, storage), Azure DevOps CI/CD and Infrastructure as Code, Databricks and Unity Catalog governance, observability and cost efficiency

, while remaining hands-on and close to the teams that use the platform.

Key Responsibilities: Orchestration Platform (Apache Airflow & ADF)

• Own the Apache Airflow platform: environments, deployment, upgrades, configuration, connections, variables, pools and
• DAG standards.
• Work on the migration from
• Azure Data Factory to Airflow and the evolution of the Metadata-Driven Orchestration Framework
• (DAG generation, reusable operators, dependency and scheduling patterns).
• Ensure Airflow is reliable and observable
• : SLAs, alerting, retries, backfills, health monitoring and clear failure ownership.
• Azure Resource Management
• Manage Azure resources with Infrastructure as Code (Terraform)
• : Databricks workspaces, Data Lake Storage
• , Data Factory, Key Vault
• , networking and supporting services.
• Own secrets and identity management
• : Key Vault, service principals and managed identities, secret scopes and rotation.
• Own access management
• : Azure RBAC, Entra ID groups, Unity Catalog grants and ownership, following least privilege
• , plus access requests and periodic reviews.
• Own
• Azure DevOps
• : repositories, CI/CD pipelines
• , branch policies and multi-environment promotion
• , including deployment validation and Terraform state.
• Track cost and governance of Azure resources: tagging, budgets, right-sizing and cost attribution (e.g. query tagging).
• Hands-on Data Platform evolution
• Contribute to the implementation of the Data Catalog and
• Data Quality Framework and it’s lineage across the platform.
• Drive observability
• : pipelines that are observable, debuggable and alert-driven, with clear ownership of failures.
• Operate the Databricks platform: Unity Catalog structure, cluster and job policies, Auto Loader ingestion patterns, and performance and cost optimization of workloads.
• Technical Leadership
• Participate in architecture reviews and technical decision-making, stating trade-offs
• (performance, cost, reliability, complexity) with data.
• Define and promote engineering standards
• : naming conventions, patterns and guidelines, through code reviews and design reviews.
• Collaborate with Architects, Product Owners and data engineers in technical maturation of data products.
• Build and maintain developer tooling and automation
• , including
• AI-assisted engineering workflows (shared agent instructions, skills, automated PR review).
• Support & Operations (Run Team Collaboration)
• Provide second-line support to the Run/Operations team for complex production topics.
• Troubleshoot and resolve platform incidents
• , failures and performance issues; lead root cause analysis and post-mortems.
• Ensure proper handover, documentation and operational readiness of platform changes.
• Continuously improve runbooks, monitoring and operational processes.