Releem has been utilizing MySQL to store all metrics gathered from agents, aimed at recommending configurations and showcasing metrics on the Releem Dashboard. With Releem's growth, driven by monthly additions of hundreds of servers across various operating systems and AWS RDS, our database swelled to 300 GB. This rapid expansion accelerated data growth and slowed Releem Dashboard and internal operations. It also escalated the costs required to maintain high-performance levels.
To address these challenges, we embarked on a few months of testing with ClickHouse for storing metrics. The outcomes were highly positive, marked by:
- Reduction in database size
- Shorter times for aggregation operations
- Improved performance of Releem Dashboard.
Following this successful trial, we have now fully transitioned all metrics from MySQL to ClickHouse. This strategic move allows us to utilize MySQL for application logic while leveraging ClickHouse's robust capabilities for metrics handling.
For those interested in the technicalities and strategic insights behind this migration, we've documented
the process in detail on our blog. This includes the rationale behind the shift, the benefits observed, and the impact on our operations.