A framework for monitoring and measuring a large-scale distributed system in real time

Abstract

Increasingly, Internet services are supported by a large-scale and distributed infrastructure on top of the Internet. This includes Content Delivery Networks (CDNs), Peer-to-peer (P2P) networks for content distribution, and more recently Cloud Computing built out of distributed data centers. Applications include Social networks (e.g. Facebook), content distribution and sharing (e.g. YouTube) and various location-based services and applications (e.g. Google Maps). To ensure proper operation and good QoE, these services need centralized monitoring and control of its operation. In this paper, we report our experience in building such a monitoring system. The key components include a reporting mechanism for capturing the system's state, and a visualization library to help administrator to keep track of how well the system is operating. We applied our monitoring system to support a P2P-assisted video streaming network for broadcasting the 2012 London Olympic Games in Hong Kong. We explain how our system keep track (in real time) of key performance numbers interested by the administrators, such as number of concurrent users, the average rate of contribution from different peers, and the QoE observed at different peer nodes, and illustrate various ways such information is visualized.