The Measurement Manager: Modular and Efficient End-to-End Measurement Services

Abstract

End-to-end network measurement is used to improve the precision, efficiency, and fairness for a variety of Internet protocols and applications. Measurement is typically performed in one of three ways: (1) actively, by injecting specially crafted probe packets into the network, (2) passively, by observing existing data traffic, and (3) customized, where applications use their own traffic to perform customized measurements. All current approaches suffer from drawbacks. Passive techniques are efficient but are constrained by the shape of the existing traffic. Active techniques are faster, more accurate and more flexible but impose a significantly higher overhead. And finally, custom techniques combine flexibility with efficiency, but are so tightly coupled with each application that they are not reusable. To address these shortcomings, we present the Measurement Manager, a practical, modular, and efficient service for performing end-to-end network measurements between hosts. Our architecture introduces a new hybrid approach to network measurement, where applications can pool together their data packets to be reused as padding inside network probes in a transparent and systematic way. We achieve this through the Measurement Manager Protocol (MGRP), a new transport protocol for sending probes that combines data packets and probes on the fly. In MGRP, active measurement algorithms specify the probes they wish to send using a Probe API and applications allow MGRP to use data from their own packets to fill the otherwise wasted probe padding. We have implemented the Measurement Manager inside the Linux kernel and have adapted existing applications and active measurement tools to use our system. Through experimentation we provide detailed empirical evidence that piggybacking data packets on measurement probes is not only feasible but improves source and cross traffic as well as the performance of measurement algorithms while not affecting their accuracy. We show that the Measurement Manager is an architecture with broad applications that can be used to build a generic measurement overlay network as well as expanding the solution space for estimation algorithms, since every application packet can now act as a potential probe.