Architecting a Unified, Automated Data Ecosystem on AWS

Client Overview

Our client is a popular name for making computer peripherals and software. They are also one of the world’s leading input and interface device makers. Some of their popular product lines include making keyboards, mice, tablet peripherals, headphones & headsets.

Their long term business goal was to build user friendly products that allowed users to perform their best. Since they relied heavily on data to make best products in the market, they needed data analytics solutions that met their business vision.

Challenge

Since the client dealt with so much data, they needed a central platform which would be one source of truth for their sales, supply chain, manufacturing, ordering systems, & customer feedback query.

They also wanted the platform to provide high-speed data extraction and retrieval for best performance levels. Beyond that they also wanted the data processing to be asynchronous for maximum efficiency.

Our Solution

After proper analysis of their current set up, we utilized Redshift for database management, AWS Glue for ETL, Fivetran for replication, MySql for metadata database, and Tableau for reporting. To make our process efficient, all data extracted from different sources were uploaded to Amazon S3 staging area. Continuous integration was enabled using RedShift . This made incoming files automatically trigger data loading into corresponding staging tables in Redshift. Transformations were made in Amazon Glue, with the logic scripted in SQL. Rules and logic were stored in a MySQL database for easy management. Tableau was integrated for easy reporting and precise analytics.

Business Outcome

Post our solution implementation, all data was centrally accessible, that too in real-time! With a single source of truth, the company could efficiently plan manufacturing operations based on live insights from its data pipelines and sales trends. This helped them significantly improve decision-making and operational performance.