Efficacy Measurement of Early Intervention Techniques - Tom Crick (Proceedings of 14th International Workshop on Automated Verification of Critical Systems, 2014)
Compiler technology has, for some considerable time, been sufficiently advanced that individual programmers are able to produce, in reasonably short periods of time, tools that might aid with the development process in novel ways: for example, one can easily produce a C compiler tool that will detect uncommon uses of integer arithmetic (such as the rare multiplication of values that are commonly only added) and flag such uses as potential errors.However, there is currently no convenient way to measure the efficacy of such techniques: where one might assume that uncommon uses of integer arithmetic might be erroneous, we do not have a way of measuring the cost saving associated with the potential early detection of occurrences of such things.We present a method of measuring the efficacy of a single early intervention, based on the replaying of previous executions of a compile-build-test cycle. This measurement process allows us to identify the software errors that were introduced during an original development and subsequently fixed; additionally, it allows us to identify the subset of such errors that would have been identified by the early intervention. By these means, we can take an existing historical record of a development, and extract from it meaningful information about the value of a proposed new early intervention technique.