Data Platform Engineer (Azure Databricks)

Fully Remote from Spain

€25–30/hour

12-month contract (Autonomo setup)

We are looking for a Data Platform Engineer to support and evolve multiple Unity Catalog–enabled Azure Databricks environments .

This role is focused on platform engineering and operations , NOT on building data pipelines or ML models. You will work on improving, automating, and scaling the platform itself.

Key Responsibilities:

Platform Engineering & Operations

• Improve implementation patterns across jobs and platform components
• Identify and resolve scalability, reliability, and maintainability bottlenecks
• Standardise workspace configurations and environment structures
• Monitor platform health and performance

Unity Catalog Management

• Maintain consistent and secure Unity Catalog configurations
• Ensure governance structures (catalogs, schemas, grants, service principals) are efficient and developer-friendly

Performance & Cost Optimisation

• Optimise compute configurations and autoscaling behaviour
• Analyse Spark jobs, Delta Lake usage, and storage layout for performance and cost efficiency

Automation & Infrastructure-as-Code

• Automate environments using Databricks Asset Bundles (DABs) and Terraform
• Ensure reproducible, versioned environments aligned with CI/CD principles
• Minimise manual configuration

Documentation & Guidance

• Document platform patterns and operational runbooks
• Support engineers and analysts working with Databricks

Required Experience:

• 3–5+ years in Platform (Data) Engineering
• Strong hands-on experience with Azure Databricks
• Strong knowledge of Unity Catalog (access models, governance, service principals, workspace configs)
• Solid PySpark and SQL skills
• Understanding of performance optimisation (partitioning, Z-ordering, caching, data layout)
• Experience with CI/CD, Git workflows, and Infrastructure-as-Code
• Bonus: MLOps experience

If you’re based in Spain and available for a 12-month Autonomo contract, feel free to apply or message me directly.