An Approach to Improve Existing Measurement Frameworks in Software Development Organizations

Abstract

Measurement is a key mechanism to characterize, evaluate, and improve software development, management, and maintenance processes. Nowadays, software organizations use metrics for very different purposes. Data is collected to describe, monitor, understand, assess, compare, validate, and appraise very diverse attributes related to software processes or products. Improving data collection and better using the existing data are important problems for software organizations.

This dissertation proposes an approach for improving measurement and data use when a large number of diverse metrics are already being collected by a software organization. The approach combines two methods. One looks at an organization's measurement framework in a top-down fashion and the other looks at it in a bottom-up fashion.

The top-down method, based on the Goal-Question-Metric (GQM) Paradigm, is used to identify the measurement goals of data users and map them to the metrics being used by the organization. This allows the measurement practitioners to: (1)~identify which metrics are and are not useful to the organization; and (2)~check if the goals of data user groups can be satisfied by the data that is being collected by the organization.

The bottom-up method is based on a data mining technique called Attribute Focusing (AF). It is used to identify useful information in the existing data that the data users were not aware of.

To validate the approach and to assess its usefulness, a case study was performed in a real industrial environment. The top-down and bottom-up methods were applied in the customer satisfaction measurement framework at the IBM Toronto Laboratory. The top-down method was applied to improve the customer satisfaction (CUSTSAT) measurement from the point of view of three data user groups. The bottom-up method was used to gain new insights into the existing CUSTSAT data.

The top-down method identified several new metrics for the interviewed user groups. It also contributed to better understanding the data user needs and led to modification of some of the data analyses and presentations done for those groups. The bottom-up method produced important insights on both the customer satisfaction domain and the measurement framework itself. Unexpected associations between key variables prompted new insights on their importance for the organization. Some of these associations have also revealed problems with the metrics being used to collect the data. (Also cross-referenced as UMIACS-TR-97-82)