Software that has not been tested cannot be regarded as reliable, yet there is a mass of bioinformatics software available whose testing is unclear, or even absent.  It is common to see journal articles that "test" performance of an algorithm or an implementation by pitting programs against each other and using secondary measures of quality: score of a function, memory footprint, time taken, number of contigs --- and so on.

Software testing can be broadly divided into verification --- that a program is doing what we intended it to do, and validation , that our intention was correct in the first place.  It is more common to see validation of software, by such means as direct comparison across implementations, yet comparatively rare to see verification, which must be based on tests of output correctness.  But perhaps that's not so surprising: computationally complex bioinformatics problems must often be solved with heuristics, which may even include a stochastic element.  Testing programs that can have varying output given the same input clearly, therefore, presents significant challenges.

This talk outlines some practices that we can use to test correct execution of bioinformatics software, both treating programs as "black boxes" in the case where source code is not available, or as more transparent ones, when we can inspect the code directly.  I will present some work we have completed on some published and very commonly used phylogenetics software