Enhancing infrastructure observability: Machine learning for proactive monitoring and anomaly detection

Abstract

This study addresses the critical challenge of proactive anomaly detection and efficient resource man-agement in infrastructure observability. Introducing an innovative approach to infrastructure monitoring, this workintegrates machine learning models into observability platforms to enhance real-time monitoring precision. Employ-ing a microservices architecture, the proposed system facilitates swift and proactive anomaly detection, addressingthe limitations of traditional monitoring methods that often fail to predict potential issues before they escalate. Thecore of this system lies in its predictive models that utilize Random Forest, Gradient Boosting, and Support VectorMachine algorithms to forecast crucial metric behaviors, such as CPU usage and memory allocation. The empiri-cal results underscore the system’s efficacy, with the GradientBoostingRegressor model achieving an R² score of0.86 for predicting request rates, and the RandomForestRegressor model significantly reducing the Mean SquaredError by 2.06% for memory usage predictions compared to traditional monitoring methods. These findings not onlydemonstrate the potential of machine learning in enhancing observability but also pave the way for more resilientand adaptive infrastructure management.