Rachit M.

Worked as a data engineer on a wide array of big data technologies to bring volumes of data to a common platform and analyze it. Roles and responsibilities : 1. Build distributed, reliable and scalable data pipelines using Sqoop and Spark to acquire data from multiple OLTP databases and ingest them into a Data Lake instead of traditional data warehouses. 2. Develop new extracts and migrate the existing one’s running on the traditional warehouses to a more cost effective and scalable platform built on Scala and Spark. 3. Develop Oozie workflows to orchestrate and schedule the entire process. 4. Worked extensively on tuning Spark applications to reduce the application run time and resource utilization. 5. Migrated Spark applications running on YARN to Kubernetes effectively cutting down the runtime cost by thousands of dollars. 6. Create Containerized applications using Docker to be run on Kubernetes. 7. Designed and implemented a continuous build-test-deployment (CI/CD) system with multiple component pipelines using Jenkins to support scheduled and unscheduled releases. 8. Responsible for the quality of the code written by the team ensuring proper design reviews, code reviews, unit testing and integration testing.

Worked as a data engineer collaborating closely with the team of Hadoop administrators, data analysts and scientists. Roles and responsibilities : 1. Develop innovative solutions using newer set of big data technologies which are more efficient and cost effective in handling large volumes of data. 2. Worked closely with a team of data scientists working on merchant transaction data to identify anomalies. 3. Developed an extract, transform and load process using Spring Batch to read the data for anomalous merchants and push the data to Elastic Search where it could be effectively indexed for better search capabilities like text search. 4. Developed rest APIs using Spring MVC framework to expose the data indexed in Elastic Search. 5. Developed a centralized logging framework wherein the logs would be written to Kafka topics and later stored to HDFS location. 6. Used Sonar to ensure good unit test code coverage and the code is written up to set quality standards.