Unosquare Logo
7018 - Data Engineer Software Development Professional
Seniority
Location
Required skills
Published 13 days ago

Job Title: Data Engineer

Job Summary:
We are seeking a skilled Data Engineer to create parser scripts to process large volumes of data, using Python, PySpark and Databricks.
The ideal candidate will possess a strong foundation in data engineering practices, and is used to work on cloud data platforms, to use DevOps/version control tooling for code deployment, and is well versed with data tooling. This role requires hands-on experience with Python, PySpark and Databricks.

Key Responsibilities: Data engineering

  • Develop, update and maintain data processing scripts in Python and PySpark
  • Query data with Databricks and ensure jobs are running without issues
  • Manage and update the Unity catalogue in Databricks
  • Check compute usage to scale back for cost savings

DevOps:

  • Leverage Azure DevOps for automated deployment pipelines, and CI/CD.
  • Use GitHub/Bitbucket to check in/check out code, perform peer reviews, manage pull requests from Dev to Prod

Qualifications:

  • Minimal of 4 years experience as Data engineer.
  • Strong experience with Python and PySpark and ingesting and processing large data volumes.
  • Experience with Databricks and Unity catalog.
  • Experience with Azure DevOps and GitHub / Bitbucket for version control and CI/CD.

Preferred Qualifications:

  • Prior experience in a Data engineering role.
  • Knowledge of Python and PySpark
  • Certification in Databricks

Working hours: PST