Senior Data Engineer ( python , Java and SQL , Data pipelines )

Job Description

Senior Data Engineer

Mission of the Role

You are a passionate and skilled data engineer who enjoys building scalable, production-grade data pipelines that enable machine learning and AI at scale. You thrive on solving complex data challenges and understand how critical a robust data pipeline is to the success of any predictive model.

As a Senior Data Engineer on the Forecasting team, you will be responsible for designing, implementing, and maintaining the data pipelines that power deep neural network models used for global sales and revenue forecasting. You will work closely with ML engineers, data scientists, and product stakeholders to build automated, secure, and cost-efficient pipelines using GCP-native services.

You take pride in writing clean, testable, and production-ready code, and are comfortable operating in cloud environments. You know how to make data pipelines observable, recoverable, and performant. You are also comfortable taking architectural ownership and contributing to long-term platform improvements.

You will:

• Design and build robust, scalable, and maintainable data pipelines for training and inference of deep learning models used in forecasting and other Machine Learning based algorithms
• Develop and orchestrate end-to-end workflows using GCP-native tools such as Cloud Workflows, Cloud Run, BigQuery, Cloud Functions, and Pub/Sub
• Automate and maintain the pipeline lifecycle, including data extraction, transformation, loading (ETL/ELT), and triggering model runs
• Monitor and Optimise performance, cost-efficiency, and data freshness across all stages of the pipeline
• Collaborate with data scientists to productionise AI/ML models and integrate them seamlessly into business workflows
• Own the data ingestion, validation, and transformation logic powering the forecasting models, ensuring high-quality, consistent input data
• Handle deployment, versioning, and configuration of pipeline components using Docker, Git, and CI/CD practices
• Implement logging, monitoring, and alerting to ensure reliability and traceability in a high-scale environment
• Review code, mentor junior engineers, and help define best practices in our evolving data engineering stack

Qualifications

You have:

• 7+ years of experience in data engineering or backend engineering roles
• Strong expertise in Python and SQL, with experience building production-grade data pipelines
• Solid understanding of Docker, Git, and shell scripting in Linux environments
• Hands-on experience with GCP services
• Experience in building, deploying, and maintaining data workflows that feed AI/ML models
• Familiarity with model lifecycle management and infrastructure challenges in ML pipelines
• Proficiency in setting up CI/CD pipelines for data and ML systems using GitLab CI, Cloud Build, or similar tools
• Exposure to Java for backend services or pipeline components (even if not primary language)
• A proactive, collaborative mindset and strong communication skills across engineering and data science teams

Nice to have:

• Exposure to forecasting or time series modelling pipelines
• Experience with event-driven architectures
• Familiarity with infrastructure-as-code tools like Terraform
• Understanding of data quality frameworks and observability tools
• Knowledge of model versioning tools and experiment tracking systems

Additional Information

Why Join us?

• You’ll be working on a high-impact platform that forecasts consumer demand and drives business-critical decisions globally
• You'll be part of a tight-knit, collaborative, and innovative Forecasting & AI team within a leading global data science organisation
• You’ll have access to top-tier infrastructure, GCP services, and challenging real-world problems in predictive modelling
• Flexible working hours, remote-friendly culture, and strong focus on personal and professional growth
• Competitive compensation and performance-based bonuses

Our Benefits

• Flexible working environment
• Volunteer time off
• LinkedIn Learning
• Employee-Assistance-Program (EAP)

About NIQ

NIQ is the world’s leading consumer intelligence company, delivering the most complete understanding of consumer buying behavior and revealing new pathways to growth. In 2023, NIQ combined with GfK, bringing together the two industry leaders with unparalleled global reach. With a holistic retail read and the most comprehensive consumer insights—delivered with advanced analytics through state-of-the-art platforms—NIQ delivers the Full View™. NIQ is an Advent International portfolio company with operations in 100+ markets, covering more than 90% of the world’s population.

For more information, visit NIQ.com

Want to keep up with our latest updates?

Follow us on: LinkedIn | Instagram | Twitter | Facebook

Our commitment to Diversity, Equity, and Inclusion

NIQ is committed to reflecting the diversity of the clients, communities, and markets we measure within our own workforce. We exist to count everyone and are on a mission to systematically embed inclusion and diversity into all aspects of our workforce, measurement, and products. We enthusiastically invite candidates who share that mission to join us. We are proud to be an Equal Opportunity/Affirmative Action-Employer, making decisions without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability status, age, marital status, protected veteran status or any other protected class. Our global non-discrimination policy covers these protected classes in every market in which we do business worldwide. Learn more about how we are driving diversity and inclusion in everything we do by visiting the NIQ News Center: https://nielseniq.com/global/en/news-center/diversity-inclusion