Job Title: Data Engineer

Job Summary:
We are seeking a skilled Data Engineer to create parser scripts to process large volumes of data, using Python, PySpark and Databricks.
The ideal candidate will possess a strong foundation in data engineering practices, and is used to work on cloud data platforms, to use DevOps/version control tooling for code deployment, and is well versed with data tooling. This role requires hands-on experience with Python, PySpark and Databricks.

Key Responsibilities: Data engineering

• Develop, update and maintain data processing scripts in Python and PySpark
• Query data with Databricks and ensure jobs are running without issues
• Manage and update the Unity catalogue in Databricks
• Check compute usage to scale back for cost savings

DevOps:

• Leverage Azure DevOps for automated deployment pipelines, and CI/CD.
• Use GitHub/Bitbucket to check in/check out code, perform peer reviews, manage pull requests from Dev to Prod

Qualifications:

• Minimal of 4 years experience as Data engineer.
• Strong experience with Python and PySpark and ingesting and processing large data volumes.
• Experience with Databricks and Unity catalog.
• Experience with Azure DevOps and GitHub / Bitbucket for version control and CI/CD.

Preferred Qualifications:

• Prior experience in a Data engineering role.
• Knowledge of Python and PySpark
• Certification in Databricks

Working hours: PST