Senior Data Engineer (Databricks)
- London
- Permanent
- Full-time
We are seeking a highly skilled Data Engineer with strong Databricks expertise to join our dynamic data engineering team. Your primary focus will be to design, develop, and optimize data pipelines and workflows leveraging Databricks on AWS infrastructure.Why Join Us?
- Be part of a skilled team driving innovative solutions using Databricks and AWS.
- Opportunities to directly impact and shape the strategic direction of our data platform.
- Competitive compensation with significant professional growth opportunities.
- Develop and maintain high-performance data pipelines using Databricks and Spark
- Collaborate with architects and analysts to build data models and transformations.
- Ensure data quality, security, and compliance within Databricks environments.
- Implement monitoring and logging strategies to ensure data pipeline reliability and performance.
- Deploy and manage code via CI/CD workflows.
- Troubleshoot and resolve complex data processing challenges efficiently.
- A Bachelor's degree in Computer Science or a related technical field or equivalent experience.
- 5-10 years of commercial experience with modern data stack i.e. Databricks, Snowflake or similar.
- Expert-level proficiency in Databricks, including PySpark, Spark SQL, and Delta Lake.
- Strong experience with data modeling, optimization, and performance tuning within Databricks.
- Hands-on experience with AWS infrastructure, specifically for Databricks deployments.
- Solid understanding of data governance, security best practices, and compliance.
- Familiarity with CI/CD principles and practices.
- Excellent problem-solving skills and ability to collaborate effectively in agile teams.
- Experience with programming languages, ideally Python
- Prior experience working within financial services or highly regulated environments