Effect of Testing Techniques on Software Reliability Estimates Obtained Using Time-Domain Models

Mei-Hwa Chen, Aditya P. Mathur and Vernon J. Rego

Software Engineering Research Center
Department of Computer Sciences
Purdue University
W. Lafayette, In 47907


Over the past two decades of research in software reliability, researchers have proposed several models to estimate reliability of software as testing progresses. Among these models, the time-domain models, i.e. those which use the notion of ``time'' between failures or the number of faults within a certain time duration, are the most common. Almost invariably these models assume that (a)~testing is carried out using an {\em operational} profile and (b)~functional testing, also known as black-box testing, is used during the system test phase. The failure data is obtained during the system test phase and used to estimate various model parameters and predict reliability. In this paper we present empirical evidence to show that the testing method used does affect the reliability estimates obtained using one of these models, namely the Musa execution time model. The evidence presented suggests that reliability models need to take into account additional data, generated during testing, such as some form of code coverage, to obtain accurate reliability estimates. It also suggests that further research is necessary to determine which testing method, or combination thereof, leads to higher reliability. We also present arguments against the notion that the use of coverage measures for large programs is not desirable in reliability estimation due to high cost.