Shubham S.

Developed a dynamic Databricks workflow as a sole contributor, enabling trigger-based execution by backend microservices. This workflow autonomously generates delta tables and views, crucial for backend processes that create Excel workbooks containing fund information. The data source originates from transactional tables with fund audit financial data. The solution was built from the ground up, offering parallel processing capabilities and extensive configurability via JSON-controlled settings. Engineered agile pipelines with the ability to execute intricate calculations driven by microservices parameters. Optimized SQL queries, enhancing system performance by eliminating redundancies.

Designed and executed ETL production pipelines using PySpark and HIVE for data extraction, decryption, reconciliation, and transformation. Developed Databricks framework for downstream predictive analytics, enabling easy access to data as HIVE tables. Enhanced PySpark notebooks to achieve 40% reduction in runtime by parallelizing data loads. Delivered adhoc notebooks for real-time data manipulation needs. Established historical data ingestion workflow and incremental orchestration for daily loads.