Rearchitecting Data Platform on AWS
KUGU Home is a market leader in digital building management. They strive to provide excellent service to their fastly growing customer base.
To that end, KUGU is transforming their data platform into a scalable and reliable system. Here, we explore the transition challenges, strategies for smooth migration, and the resulting performance improvements, setting a new standard for scalability and reliability.
​
Technologies: AWS, ECS, Lambda, Glue, Spark, S3, Athena, Kafka, ECR
What we did
Rearchitect the whole infrastructure based on AWS Managed Services and custom ECS container based application. Our primary objective was to enhance efficiency through automation, thereby reallocating resources and bandwidth to accelerate the development and deployment of new features.
Challenge
The initial setup consisted of a complex network of self-hosted and internally managed services, encompassing vital elements such as Data Platform, Backend Systems, Data Pipelines, Authentication Processes, User Management, Frontend Interfaces and monitoring via Grafana. Although this framework initially met the needs of KUGU, it quickly became apparent that it was consuming an unsustainable amount of the team's resources. The intricate maintenance required for these interconnected services demanded considerable time and effort, detracting from the primary mission of creating value.
Solution
Our approach to enhancing the platform's architecture was centered around scalability, reliability, and efficiency. The key strategies we adopted include:
-
ECS and Container-Based Services: By transitioning to smaller, container-based services, we established clearer boundaries around each service. This shift not only simplified management but also enabled scalable solutions tailored to demand.
-
Automated Deployment: We streamlined our deployment processes, freeing our team from the complexities associated with continuous integration and continuous deployment (CI/CD), thereby accelerating our development cycle.
-
Infrastructure-as-Code (IaC): Our infrastructure is now well-documented, transparent, and easily replicable, thanks to the adoption of IaC practices. This ensures consistency and efficiency in infrastructure management.
-
Separation of Environments via Accounts: We implemented a clear separation between development stages through the use of distinct accounts. This provides our engineers and product managers with dedicated environments for development and testing, enhancing productivity and focus.
​
-
Performance Testing Setup: We established a robust performance testing framework for the system's critical components. This setup allows for continuous monitoring and ensures the reliability of our services.
Result
The implementation of the new platform architecture has yielded remarkable outcomes, demonstrating success across several key metrics:
Enhanced Development Productivity: The development process has become significantly more focused and efficient. By minimizing distractions related to infrastructure management, the team can now concentrate more on feature development, leading to a faster rate of innovation and implementation.
Shorter Feedback Loops: With the improved system, we've managed to shorten the feedback loop considerably. When issues arise, the team is now able to receive and respond to feedback more rapidly, enabling quicker resolutions and continuous improvement.
Reliable Development and Production Environments: The reliability of both development and production environments has been markedly improved. This reliability ensures that feature development and testing are conducted in a stable and consistent setting, thereby enhancing the overall quality of the product.
Scalable Environment: Perhaps most importantly, the scalability of the environments has been greatly enhanced. This scalability is critical not only for meeting current customer demands but also for accommodating future growth. As KUGU's customer base expands, the platform is well-prepared to adapt and scale accordingly, supporting the business's growth trajectory and ensuring customer satisfaction.
Egor Sokhanev, Head of Technology
Data Max solves problems. They have deep know-how and are able to solve complex challenges. They look at the problem, propose a solution and are able to follow through with implementing it at high quality, and bring results.
Data Max engineers platforms. Not only software, but also considers all interaction, and glue pieces together.