Software Engineer-(AWS Databricks +Mongo DB) (Full-time at a Fortune 500 tech MNC)

Job Description

Profile Summary

• Data Engineer with 5 years of experience in Big Data technologies.
• Proven expertise in designing and developing scalable data pipelines using Apache Spark and PySpark.
• Proficient in Python programming.
• Hands-on experience with Databricks, MongoDB, Docker, and Kubernetes.
• Solid understanding of DevOps practices and tools.

Primary Skills

• Design, develop, and maintain scalable and reliable data pipelines.
• Translate business requirements and data models into efficient ETL code.
• Work independently on Spark-based data processing workflows.
• Skilled in writing complex business logic using PySpark and Spark SQL.
• Strong understanding of Spark clusters, distributed computing, and parallel processing.
• Capable of writing unit test cases to validate ETL logic.
• Experience in building reusable functions and applying modular programming approaches.
• Worked with AWS Athena for table creation, indexing, and writing complex SQL queries.
• Experience working with both relational databases (e.g., Oracle, SQL Server) and NoSQL databases (e.g., MongoDB) for data extraction and loading.

Secondary Skills

• Experience in creating AWS IAM roles and defining access policies.
• Familiar with Docker image creation and customization.
• Experience with Camunda for workflow orchestration and automation.

Additional Expectations

• Strong communication skills.
• Proficient in troubleshooting and problem-solving.
• Demonstrates a continuous learning mindset.
• Focused on automating manual tasks through coding wherever feasible.

Technical Skills

Mandatory Skills:

Python for Data, Apache Spark, Java, Python, Scala, Spark SQL, Databricks