Role: Senior Data Engineer 

Job summary

Our client, a leader in the RegTech space, is seeking a Senior Data Engineer to join their fast-growing team. The successful candidate will join the testing team to work on ETL and development tasks. This is an exciting and challenging opportunity to build out new pipelines combining and processing large amounts of structured data from a variety of sources.

Key activities and responsibilities

● Architect and build pipelines using AWS cloud computing solutions that make data available with robustness, maintainability, efficiency, scalability, availability and security.
● Develop Python and/or Spark (Preferably PySpark, but Spark-Scala is also good to have) code that implements complex data transformations.
● Design and maintain databases and APIs for storage and transmission of data between applications.
● Monitor pipelines in production (and develop tools to facilitate this).
● Work collaboratively with other team members (brainstorming, troubleshooting, and code review).
● Liaise with other development teams to ensure the integrity of data pipelines.

Knowledge, skills and experience required

Skills:

● Excellent Python and PySpark programming (including: Pandas/PySpark dataframes, web and database connections)
● Excellent understanding of ETL processes within Amazon Web Services (AWS)
● Apache Spark, AWS Glue, Athena, S3, Step Functions, Lake Formation
● Software development life-cycle best practices
● Test-driven development
● Serverless computing (AWS Lambda, API Gateway, SQS, SNS, EventBridge, S3, etc.)
● SQL and NoSQL database design and management (DynamoDB, MySQL)
● Strong SQL coding skills (Spark-SQL, Presto SQL, MySQL, etc.)
● Infrastructure as code (CloudFormation)
● Experience in Shell Scripting (preferably Linux)
● Version control with Git/Github
● Agile principles, processes and tools
● Excellent written and verbal communication skills.

Experience:

● Designing, deploying and managing complex production data pipelines that interact with a range of data sources (file systems, web, database, users).
● Strong experience with Amazon Web Services (AWS)
● 5 years’ work in a similar role.

Knowledge:

● Data modeling, data pipeline architecture, Big Data implementation.
● Software development lifecycle best practices.
● Financial knowledge would be an asset.

Qualifications/Training:
● Bachelor’s degree or equivalent in Computer Science or a related subject.

If you feel you’re the right candidate for this role, please apply! (only applicants with appropriate experience and skill-sets will be contacted).