Data Engineer

Decskill, founded in 2014 as an IT Consulting Company, places paramount importance on its greatest asset: its people. Our main mission is to deliver value through knowledge and talent, and we achieve this by fostering a culture of excellence and investing in the development and well-being of our people. With over 600 dedicated professionals and offices in Lisbon, Porto, Madrid, and Luxembourg, Decskill operates across three core areas:

Decskill Talent: We believe that our people are key to our success.

Through Decskill Talent, we empower our team to embrace the digital transformation challenges of our clients. We collaborate with clients to drive innovation, ensuring project success and business growth.

Decskill Boost: Equipping our team with the latest tools and methodologies, we optimize Time-to-Market and deliver innovative solutions exceeding client expectations.

Decskill Connect: Our team collaborates closely with clients to implement and manage IT infrastructures that generate long-term value.

At Decskill, we believe that by nurturing and empowering our people to confront the challenges of digital transformation, we create value not only for our clients but also for our entire ecosystem, fostering a digital community dedicated to growth and progress.

We are looking for a Data Engineer for a remote project.

Main Tasks and Responsibilities:

• Projects understanding and Communication

• Understand problems from a user perspective and communicate to clearly understand the issue

• Ensure the architecture provided by the Data Architect is clearly understood by yourself

• Communicate with the Data Architect and your peers on the technical solution you are developing and communicate with the Project Manager in charge of the project you are working on

• Write and communicate on new or updated interface contracts

• Strong understanding of data warehousing concepts, data lakes, ETL/ELT processes, and data modeling

• Develop data pipelines based on the defined architecture

• Ensure the regular good practices are applied

• Deploy requested infrastructure, particularly using Terraform

• Make peer reviews and ask to your peers to review your code when merging a new version of the codebase

• Define tests with your project manager, based on the functional and technical requirements of the pipeline you´re developing

• Perform those tests and communicate regularly on the results

• Regularly summarize the results of your tests in a dedicated document

• Deployments

• Present to the Data Architect in charge of the architecture, and the Lead DataOps, the development that was performed through our Deployment Reviews

• Track and communicate on any potential errors in the entire period of active monitoring following a deployment

• Ensure diligent application of deployment process, logging, and monitoring strategy

Requirements:

• Proficiency with PySpark and Spark SQL for data processing

• Experience with Databricks using Unit Catalog

• Knowledge of Delta Live Tables (DLT) for automated ETL and workflow orchestration in Databricks

• Familiarity with Azure Data Lake Storage

• Experience with orchestration tools (e.g., Apache Airflow or similar) for building and scheduling ETL/ELT pipelines

• Knowledge of data partitioning and data lifecycle management on cloudbased storage

• Familiarity with implementing data security and data privacy practices in a cloud environment

• Terraform: At least one year of experience with Terraform and know good practices of GitOps

• English B2+ (mandatory);

• The candidate must be living in Portugal (mandatory).

Nice to Have / Plus:

• Databricks Asset Bundles

• Kubernetes

• Apache Kafka

• Data Engineer - DataBricks 3

• Vault

If you’re interested in this job, please send your CV to camila.cesario@decskill.com with reference “CC/DE”

Decskill is committed to equality and non-discrimination with all our talents. We recruit and promote talent, based on diversity and inclusion, regardless of age, gender, ethnicity, race, nationality or any other form of discrimination incompatible with the dignity of the human being.