Senior Data Engineer (Python, Airflow, Bigquery, Scraping) - Full-time - Overlap with CEST

Project Description

We are seeking a full-time Senior Data Engineer with expertise in Python and Apache Airflow. The role involves automating operations on online delivery platforms by enhancing and managing existing data scraping systems. The successful candidate will help improve current logic and manage credentials to automate actions and generate reports.

General Information

Our company is focused on automating online delivery operations for restaurants by scraping data from private APIs. With over a year of experience, we have an existing system that uses Apache Airflow, MongoDB, and BigQuery. The current architecture is functional, but we are transitioning from public to private API scraping, requiring improvements to the existing system. We offer a flexible work schedule with a minimum of five hours overlap with Central European working hours.

Tasks and Deliverables:

• Automate operations on online delivery platforms.
• Retrieve and analyze data from delivery platform hubs to generate reports.
• Reverse engineer and improve existing logic for data scraping.
• Manage and enhance credential management processes.
• Utilize existing codebase and architecture with Apache Airflow.
• Communicate effectively with the team to ensure seamless project execution.

Required Experience:

• Proficiency in Python.
• Experience with Apache Airflow.
• Strong communication skills.
• Ability to work with large codebases.

Nice to Have:

• General knowledge of data pipelines.
• Experience with Google Cloud.
• Back-end data scraping experience.
• Experience with private API scraping.
• Familiarity with MongoDB and BigQuery.
• Experience in automating operations for delivery platforms.

Engagement Highlights:

• Full-time engagement expected to last a few months.
• Opportunity to work on a project that enhances automated delivery operations.
• Immediate start possibility if a suitable candidate is found.
• Potential for skill growth and role expansion.