Test oracle

For other uses, see Oracle (disambiguation).

In software testing, a test oracle (or just oracle) is a provider of information that describes correct output based on the input of a test case. Testing with an oracle involves comparing actual results of the system under test (SUT) with the expected results as provided by the oracle.[1]

The term "test oracle" was first introduced in a paper by William E. Howden.[2] Additional work on different kinds of oracles was explored by Elaine Weyuker.[3]

An oracle can operate separately from the SUT; accessed at test runtime, or it can be used before a test is run with expected results encoded into the test logic.[4]

However, method postconditions are part of the SUT, as automated oracles in design by contract models.[5]

Determining the correct output for a given input (and a set of program or system states) is known as the oracle problem or test oracle problem,[6]: 507  which some consider a relatively hard problem, and involves working with problems related to controllability and observability.[7]

  1. ^ Earl T. Barr et al; The Oracle Problem in Software Testing: A Survey, 2015
  2. ^ Howden, W.E. (July 1978). "Theoretical and Empirical Studies of Program Testing". IEEE Transactions on Software Engineering. 4 (4): 293–298. doi:10.1109/TSE.1978.231514.
  3. ^ Weyuker, Elaine J.; "The Oracle Assumption of Program Testing", in Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, January 1980, pp. 44-49
  4. ^ Jalote, Pankaj; An Integrated Approach to Software Engineering, Springer/Birkhäuser, 2005, ISBN 0-387-20881-X
  5. ^ Meyer, Bertrand; Fiva, Arno; Ciupa, Ilinca; Leitner, Andreas; Wei, Yi; Stapf, Emmanuel (September 2009). "Programs That Test Themselves". Computer. 42 (9): 46–55. doi:10.1109/MC.2009.296.
  6. ^ Cite error: The named reference Oracle survey was invoked but never defined (see the help page).
  7. ^ Ammann, Paul; and Offutt, Jeff; "Introduction to Software Testing, 2nd edition", Cambridge University Press, 2016, ISBN 978-1107172012