Data engineer with commodities experience

Job Title: Data Engineer
Location: London, hybrid 
Salary: 450-550 Inside ir35 

The Company:
A forward-thinking, innovative firm embarking on a multi-year Digitalisation Program to revolutionize data management and utilization across its business. The company is focused on building cutting-edge data infrastructure powered by cloud technologies to support AI/ML models and optimize trading and physical assets.

The Role:
As a Data Engineer, you will be a key player in the development of next-generation data infrastructure. You will work closely with the Data Engineering Team to build, maintain, and optimize data pipelines, assist with data integration challenges, and support the development of scalable AI/ML models. This is a pivotal role within an ambitious digital transformation program.

Key Responsibilities:

• Develop, test, and maintain data pipelines
• Solve complex data integration challenges using ETL/ELT techniques
• Cleanse, preprocess, and deliver data for prescriptive and predictive modeling
• Work with data scientists to build scalable AI/ML models
• Develop dashboards and Business Intelligence tools
• Ensure high data quality, governance, and security
• Collaborate on CI/CD pipeline development and code quality standards

Technical Stack:

• Cloud platforms: Data Factory, Synapse, Databricks, Redshift, Glue, Athena, BigQuery
• Distributed computing tools: Spark
• Data pipeline development (ETL/ELT)
• Programming languages: Python, SQL (TSQL, ANSI SQL, Spark SQL)
• Experience with data lakes and data warehousing

Desirable Skills:

• Knowledge of commodity markets
• Experience in hybrid cloud environments
• Familiarity with authorization frameworks such as OPA or Ranger
• Experience with analytical models/tools

Soft Skills:

• Strong communication skills to explain data concepts to stakeholders
• Team player with a proactive, problem-solving attitude
• Strong organizational and time management skills

How to Apply:
Interested? Get in touch directly or apply through the provided link to take the next step in your career!

''