Senior Data Engineer
Role: Senior Data Engineer
Job summary
Our client, a leader in the RegTech space, is seeking a Senior Data Engineer to join their fast-growing team. The successful candidate will join the testing team to work on ETL and development tasks. This is an exciting and challenging opportunity to build out new pipelines combining and processing large amounts of structured data from a variety of sources.
Key activities and responsibilities
● Architect and build pipelines using AWS cloud computing solutions that make data available with robustness, maintainability, efficiency, scalability, availability and security.
● Develop Python and/or Spark (Preferably PySpark, but Spark-Scala is also good to have) code that implements complex data transformations.
● Design and maintain databases and APIs for storage and transmission of data between applications.
● Monitor pipelines in production (and develop tools to facilitate this).
● Work collaboratively with other team members (brainstorming, troubleshooting, and code review).
● Liaise with other development teams to ensure the integrity of data pipelines.
Knowledge, skills and experience required
Skills:
● Excellent Python and PySpark programming (including: Pandas/PySpark dataframes, web and database connections)
● Excellent understanding of ETL processes within Amazon Web Services (AWS)
● Apache Spark, AWS Glue, Athena, S3, Step Functions, Lake Formation
● Software development life-cycle best practices
● Test-driven development
● Serverless computing (AWS Lambda, API Gateway, SQS, SNS, EventBridge, S3, etc.)
● SQL and NoSQL database design and management (DynamoDB, MySQL)
● Strong SQL coding skills (Spark-SQL, Presto SQL, MySQL, etc.)
● Infrastructure as code (CloudFormation)
● Experience in Shell Scripting (preferably Linux)
● Version control with Git/Github
● Agile principles, processes and tools
● Excellent written and verbal communication skills.
Experience:
● Designing, deploying and managing complex production data pipelines that interact with a range of data sources (file systems, web, database, users).
● Strong experience with Amazon Web Services (AWS)
● 5 years’ work in a similar role.
Knowledge:
● Data modeling, data pipeline architecture, Big Data implementation.
● Software development lifecycle best practices.
● Financial knowledge would be an asset.
Qualifications/Training:
● Bachelor’s degree or equivalent in Computer Science or a related subject.
If you feel you’re the right candidate for this role, please apply! (only applicants with appropriate experience and skill-sets will be contacted).
Similar Vacancies:
Senior Escaltions Engineer / 3rd Line Support
- Bristol, UK
- 10% Bonus
Senior Escaltions Engineer , 3rd Line Support , Microsoft 365 , Active Directory , Hyper V , Back Up Replication , Hybrid Working Our grwoing MSP...
SOC Analyst ( SC Cleared )- Perm - Hybrid
- London, UK
- £50000 - £60000 per annum
Senior SOC AnalystThis is a chance for an individual to be in ‘at the ground up’ at the start of the development of the SOC and will play a...
Head of Sales Operations / CRM Manager ( FTC)
- Central London, UK
- £50000 per annum
Head of Sales Operations / CRM Manager ( FTC) ( London Hybrid 1/2 days on site) Our Award winning Reg Tech client is looking to recruit an...