What you will do
- Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and Spark SQL.
- Build and maintain robust ETL/ELT pipelines for data ingestion, transformation, processing, and loading.
- Orchestrate and schedule data workflows using Databricks Lakeflow Jobs, notebooks, and Python wheel packages.
- Work with Unity Catalog, managed tables, and streaming jobs to manage and process enterprise data.
- Implement scalable data engineering solutions using the Databricks Lakehouse platform.
- Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.
- Ensure data quality, integrity, consistency, availability, and reliability across data pipelines.
- Implement appropriate data modelling approaches to support Data Lake, Data Warehouse, and Lakehouse environments.
- Use Git for source control, version management, and collaborative development.
- Troubleshoot pipeline failures, performance bottlenecks, and data-related issues.
- Continuously improve data engineering processes, architecture, and operational efficiency.
- Collaborate with business stakeholders and technical teams to deliver reliable data solutions.
What you bring
- 10+ years of overall experience in the data domain, particularly Data Warehousing and ETL.
- 4+ years of hands-on Databricks experience.
- Solid understanding of Databricks architecture and Lakehouse concepts.
- Hands-on experience with Databricks Lakeflow Jobs, notebooks, and Python wheel packages.
- Practical knowledge of Unity Catalog, managed tables, and streaming jobs.
- Strong experience developing and maintaining ETL/ELT data pipelines.
- Good understanding of Data Lakes, Data Warehousing, and Lakehouse architecture.
- Experience with data modelling and data transformation techniques.
- Hands-on experience with Git and version control practices.
- Ability to optimize data pipelines and Databricks workloads for performance and cost efficiency.
- Databricks Certified Data Engineer Associate or Professional certification is an advantage.
How you work
- Design and implement scalable, reliable, and high-performance data engineering solutions.
- Analyse complex data processing requirements and translate them into effective technical solutions.
- Troubleshoot and resolve data pipeline failures and performance issues.
- Maintain high standards of data quality, integrity, and reliability.
- Optimize data processing workflows and cloud resource utilization.
- Communicate effectively with technical teams, business stakeholders, and project leadership.
- Manage stakeholder expectations and collaborate across cross-functional teams.
- Work independently while contributing effectively to team objectives.
What you can expect
- An opportunity to work on enterprise-level Databricks and Lakehouse data engineering initiatives.
- Exposure to modern data engineering technologies and scalable data platforms.
- Opportunities to work with PySpark, Python, Spark SQL, Lakeflow Jobs, and Unity Catalog.
- Collaboration with technical teams and business stakeholders.
- Opportunities to build and optimize high-performance data pipelines and enterprise data solutions.