Lead Data Engineer

Job Locations: UK-London

Job ID: 2023-1563

Position Type: Permanent Full-Time

Overview

The Lead Data Engineer will lead agile data engineering activities as a technical leader, delivering data and insights products to the rail industry.

You will understand the business and IT strategy to align technical solutions with outcomes, develop APIs and integrations to transfer data between systems and deliver aggregated data solutions.

You will develop both quick prototypes and high-quality products, judging which is needed. The role will also coach data engineers and undertake code reviews to ensure the team can continue to deliver sustainable solutions.

Responsibilities

• Lead ambiguous and complex data engineering projects as a technical leader.
• Design, code, test, and document data applications, applying software engineering best practices.
• Initiate the development and management of scalable APIs, machine learning endpoints, data integration pipelines and data storage solutions.
• Influence the information strategy and platform architecture to align technical solutions with outcomes.
• Influence new Data Engineering tools, frameworks, standards, and processes to improve the productivity and quality of outputs for Data Engineers across the team.
• Ensure that agreed standards and tools are applied to achieve well-engineered outcomes, and that own work and that of colleagues meets coding standards, through code review processes.
• Ensure technical solutions support the accessibility, retrievability, security, quality, retention, and ethical handling of specific sets of data.

Qualifications

• Experience of strategies and design approaches for APIs and data pipelines.
• Experience designing and implementing data-driven tools and solutions.
• Experience developing scalable APIs in Python using frameworks such as Fast API or Flask.
• Experience developing scalable data pipelines in Spark/Python using orchestrators such as Airflow.
• Experience using SQL, Python, and other scripting/programming to solve a wide array of data challenges.
• Experience working with a diverse range of data infrastructure and systems, such as PostgreSQL; SQL Server; Azure data lakes; Data Bricks; Kubernetes.
• Communicate effectively and share thoughts and ideas through methods appropriate to the audience.
• Adapt and respond effectively when embracing new opportunities, change and in navigating uncertainty.
• Actively contribute as part of a team and work towards achieving team goals and outcomes.
• Take responsibility and demonstrate accountability in completing tasks and achieving objectives, actively seeking to resolve problems and identify opportunities.
• Committed to customer service and placing customer satisfaction at the heart of our success to ensure we deliver against our shared goals.
• Build effective professional relationships with stakeholders, understanding their unique goals and motivations, gaining credibility and effectively influencing to achieve desired outcomes.
• Can work collaboratively within an evolving industry, gaining stakeholder confidence through understanding their goals and motivations and demonstrating credibility as an expert.
• Make timely, informed decisions taking account of the benefits and constraints involved.

We are committed to investing in our staff and offer formal and informal learning opportunities for development.

We value diversity and equal opportunities in employment and are committed to creating a workplace which is inclusive to everyone. As a member of the Disability Confident Scheme, we encourage candidates with disabilities who meet the minimum criteria, to apply for our jobs. If you have applied under the Disability Confident Scheme, please let us know in advance by emailing.

If you require any reasonable adjustments with respect to our selection process including information in an alternative format, please contact us.

We understand the importance of work-life balance and we offer our staff the flexibility to work within our core hours and the option to vary their location between both the office and home. If you are looking for further flexibility, speak to us at interview stage so that we can consider your request.

We value our staff and we offer a competitive benefits package to ensure our staff can achieve their best throughout their journey with us. This includes 30 days annual leave (plus bank holidays); a holiday buy and sell scheme; private medical and dental cover; a season ticket loan and travel subsidy; access to a cycle to work scheme; volunteer leave; a performance related bonus and pension.