From 553120a2bc6d0465084f452b831c6afd4c34bd3c Mon Sep 17 00:00:00 2001 From: Karl Berry Date: Mon, 14 Jul 2025 09:13:13 -0700 Subject: [PATCH] doc: xfail doesn't mean testing a failure condition. * doc/automake.texi (Generalities about Testing): mention that xfail is not about testing a failure condition. See thread around: https://lists.gnu.org/archive/html/automake/2025-07/msg00008.html --- doc/automake.texi | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/doc/automake.texi b/doc/automake.texi index 83599a722..27b520349 100644 --- a/doc/automake.texi +++ b/doc/automake.texi @@ -9600,14 +9600,21 @@ all of the testsuite has been run. @cindex xpass @cindex unexpected pass @cindex unexpected test pass -It's not uncommon, especially during early development stages, that some -tests fail for known reasons, and that the developer doesn't want to -tackle these failures immediately (this is especially true when the +It's not uncommon, especially during early development stages, that +some tests fail for known reasons, and that the developer doesn't want +to tackle these failures immediately (this is especially true when the failing tests deal with corner cases). In this situation, the better policy is to declare that each of those failures is an @emph{expected -failure} (or @emph{xfail}). In case a test that is expected to fail ends -up passing instead, many testing environments will flag the result as a -special kind of failure called @emph{unexpected pass} (or @emph{xpass}). +failure} (or @emph{xfail}). When writing such a test, it is highly +desirable to note in the test what has to happen to make the test +succeed. In case a test that is expected to fail ends up passing +instead, many testing environments will flag the result as a special +kind of failure called @emph{unexpected pass} (or @emph{xpass}). + +An expected failure result is not related to the contents of the test +itself. For example, a test might check whether a given system call +correctly returns failure given an invalid argument. If so, then the +@emph{test} passes, even though it was testing a failure condition. @cindex hard error @cindex Distinction between errors and failures in testsuites -- 2.47.2