Over 6+ years of experience in Data Engineering, Data Pipeline Design, Development and Implementation as a Sr. Data Engineer and Data Developer.
• Strong experience in Software Development Life Cycle (SDLC) including Requirements Analysis, Design Specification and Testing as per Cycle in both Waterfall and Agile methodologies.
• Strong experience in writing scripts using Python API, PySpark API and Spark API for analyzing the data.
• Experience with Azure Cloud, Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Azure Analytical Service, Big Data Technologies (Apache Spark)
• Experience with GCP Cloud Storage, Big Query, Composer, Cloud Dataproc, Cloud SQL, Cloud Functions, Cloud Pub/Sub
• Worked on ETL Migration services by creating and deploying AWS Lambda functions to provide a serverless data pipeline that can be written to glue catalog and queried from Athena.
• Extensively used Python Libraries PySpark, Pytest, Pymongo, PyExcel, Psycopg, embedPy, NumPy and Beautiful Soup.
• Migrated an existing on-premises application to GCP. Used GCP services like Cloud Dataflow and Dataproc for small data sets processing and storage.
• Hands On experience on Spark Core, Spark SQL, Spark Streaming and creating the Data Frames handle in SPARK with Scala.
• Experience in NoSQL databases and worked on table row key design and to load and retrieve data for real time data processing and performance improvements based on data access patterns.
• Experience with Unix/Linux systems with scripting experience and building data pipelines.
• Extensive experience in Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, and Map Reduce concepts.
• Performed complex data analysis and provided critical reports to support various departments.
• Expertise in development of various reports, dashboards using various PowerBI and Tableau Visualizations
• Experience in building large scale highly available Web Applications. Working knowledge of web services and other integration patterns.
• Experienced with version control systems like Git, GitHub, CVS, and SVN to keep the versions and configurations of the code organized. Efficiently facilitating task tracking and issue management using JIRA.
• Good communication skills, work ethics and the ability to work in a team efficiently with good leadership skills.
• Experience with continuous integration and automation using Jenkins.
• Experience building docker image using accelerator and scanning images using various scanning techniques
• Over 6+ years of experience in Data Engineering, Data Pipeline Design, Development and Implementation as a Sr. Data Engineer and Data Developer.
• Strong experience in Software Development Life Cycle (SDLC) including Requirements Analysis, Design Specification and Testing as per Cycle in both Waterfall and Agile methodologies.
• Strong experience in writing scripts using Python API, PySpark API and Spark API for analyzing the data.
• Experience with Azure Cloud, Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Azure Analytical Service, Big Data Technologies (Apache Spark) • Experience with GCP Cloud Storage, Big Query, Composer, Cloud Dataproc, Cloud SQL, Cloud Functions, Cloud Pub/Sub
• Worked on ETL Migration services by creating and deploying AWS Lambda functions to provide a serverless data pipeline that can be written to glue catalog and queried from Athena.
• Extensively used Python Libraries PySpark, Pytest, Pymongo, PyExcel, Psycopg, embedPy, NumPy and Beautiful Soup.
• Migrated an existing on-premises application to GCP. Used GCP services like Cloud Dataflow and Dataproc for small data sets processing and storage.
• Hands On experience on Spark Core, Spark SQL, Spark Streaming and creating the Data Frames handle in SPARK with Scala.
• Experience in NoSQL databases and worked on table row key design and to load and retrieve data for real time data processing and performance improvements based on data access patterns.
• Experience with Unix/Linux systems with scripting experience and building data pipelines.
• Extensive experience in Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, and Map Reduce concepts.
• Performed complex data analysis and provided critical reports to support various departments. • Expertise in development of various reports, dashboards using various PowerBI and Tableau Visualizations • Experience in building large scale highly available Web Applications. Working knowledge of web services and other integration patterns.
• Experienced with version control systems like Git, GitHub, CVS, and SVN to keep the versions and configurations of the code organized. Efficiently facilitating task tracking and issue management using JIRA.
• Good communication skills, work ethics and the ability to work in a team efficiently with good leadership skills.
• Experience with continuous integration and automation using Jenkins. • Experience building docker image using accelerator and scanning images using various scanning techniques