MODULES

ETS (Educational Testing Service)

Worked on various projects during my contract role at ETS.

Part of the DII (Data Innovation and Insights) Team, joined as a Sr.Cloud Engineer and eventually became a Lead Cloud Engineer and Architect. Owned the complete SDLC lifecycle for the DII project while being a key-contributor as well. Onboarded various solutions on cloud, able to re-design the existing solution and able to achieve below results:


ANALYTICS PIPELINE

Worked on building analytics pipeline which streams realtime events across all internal applications and onboard the applications to the DII platform. AWS Services used: API Gateway, Kinesis Data Forehose (converts events to parquet format for analytics), S3, Glue Catalog, Glue Crawlers, Athena, Power BI (for KPI reporting).

TRANSFORMATION PIPELINE

To enrich data coming from various sources and streamline batch workloads, create packages and upload to datalake for data scientists/data teams to checkout the DII datasets based on their requirements. AWS Services used: AWS Congito, AWS S3, AWS Step Functions,  ECS Tasks (Fargate), ECR, Data Lake.

MiGRATION FROM POWERBI TO TABLEAU

Based on the business decision, I have led and successfully migrated our analytics pipeline to point to the Tableau Cloud from PowerBI. All the existing dashboards are migrated, and able to create new KPI graphs using Tableau, and share with the Senior Leadership (including C Level).

RE-DESIGN AND PIVOT

To reduce the glue cost usage (crawlers and catalog) and Athena query costs - I have suggested design improvements to the Architecture Review, worked on the design, requirements and onboard new infrastructure, development, and deployment of new components. 

Onboarded data teams to our platform using AWS SageMaker, where direct access to the cleaned datasets are provided to run both analytics and transformation pipelines. 

This delivered some key-wins from both business and technical team perspectives. 

Tech-team wins

Business-team wins