conformtest: Support system-specific XFAILs.

conformtest has an internal XFAIL mechanism to allow failures of
individual expectations to be ignored, so that known hard-to-fix
failures (e.g. those affecting ABIs or requiring kernel changes) do
not cause the overall tests to FAIL and so hide other failures from
the same (header, standard) pair.

Various such bugs are system-specific, so this patch adds a mechanism
to allow system-specific XFAILs.  A system-independent XFAIL is
achieved by putting "xfail-" at the start of the relevant expectation
in the *-data files.  A system-specific XFAIL instead uses
"xfail[cond]-", where "cond" is a condition listed in
conformtest-xfail-conds in a sysdeps makefile (so one for x32 might
set conformtest-xfail-conds = x86_64-x32-linux, for example, and then
an expectation for tv_nsec's type could use
xfail[x86_64-x32-linux]-).  The actual names are arbitrary, just
needing to match between the makefiles and the expectations, and if
necessary you can use "xfail[cond1|cond2]-" for a test that is
expected to fail under multiple conditions.  As with
system-independent XFAILs, I think system-specific ones should have a
bug filed in Bugzilla and a comment referencing that bug.

Tested for x86_64, including with test expectations and makefiles
changed to use the new facility.

	* conform/conformtest.pl ($xfail_str): New variable.
	(--xfail=): New command-line option.
	(top level): Handle expectations starting xfail[cond]-.
	* conform/Makefile (conformtest-xfail): New variable.
	($(conformtest-header-tests)): Pass $(conformtest-xfail) to
	conformtest.pl.

diff --git a/ChangeLog b/ChangeLog
index bcfa26e2bb8634e2dca9ca2cf3349cc040762f67..1f8760f571b599072d4f7205798a70acafdbcf18 100644 (file)
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,12 @@
+2017-03-17  Joseph Myers  <joseph@codesourcery.com>
+
+       * conform/conformtest.pl ($xfail_str): New variable.
+       (--xfail=): New command-line option.
+       (top level): Handle expectations starting xfail[cond]-.
+       * conform/Makefile (conformtest-xfail): New variable.
+       ($(conformtest-header-tests)): Pass $(conformtest-xfail) to
+       conformtest.pl.
+
 2017-03-16  Joseph Myers  <joseph@codesourcery.com>
 
        * conform/conformtest.pl: Use compilation instead of execution

diff --git a/conform/Makefile b/conform/Makefile
index b393641e5d3b2e0f99abe44cb0c761e4ea982dbd..c07cc1d8d964dbf18a786f8401342f572cc3d782 100644 (file)
--- a/conform/Makefile
+++ b/conform/Makefile
@@ -192,13 +192,18 @@ test-xfail-XOPEN2K8/signal.h/conform = yes
 test-xfail-XOPEN2K8/sys/wait.h/conform = yes
 
 conformtest-cc-flags = -I../include $(+sysdep-includes) $(sysincludes) -I..
+# conformtest-xfail-conds may be set by a sysdeps Makefile fragment to
+# a list of conditions that are considered to be true when encountered
+# in xfail[cond]- lines in test expectations.
+conformtest-xfail = $(if $(conformtest-xfail-conds),\
+                        --xfail='$(conformtest-xfail-conds)')
 $(conformtest-header-tests): $(objpfx)%/conform.out: \
                             conformtest.pl $(conformtest-headers-data)
        (set -e; std_hdr=$*; std=$${std_hdr%%/*}; hdr=$${std_hdr#*/}; \
         mkdir -p $(@D)/scratch; \
         $(PERL) -I. conformtest.pl --tmpdir=$(@D)/scratch --cc='$(CC)' \
                 --flags='$(conformtest-cc-flags)' --standard=$$std \
-                --headers=$$hdr > $@); \
+                --headers=$$hdr $(conformtest-xfail) > $@); \
        $(evaluate-test)
 
 $(linknamespace-symlists-tests): $(objpfx)symlist-%: list-header-symbols.pl

diff --git a/conform/conformtest.pl b/conform/conformtest.pl
index 54b38f85e91021af6fb7dbfcd65e4084d30fa093..41682e9b86f5d0b9d3f79cd4e6b8e0b2ba785a05 100644 (file)
--- a/conform/conformtest.pl
+++ b/conform/conformtest.pl
@@ -7,8 +7,10 @@ use POSIX;
 $standard = "XOPEN2K8";
 $CC = "gcc";
 $tmpdir = "/tmp";
+$xfail_str = "";
 GetOptions ('headers=s' => \@headers, 'standard=s' => \$standard,
-           'flags=s' => \$flags, 'cc=s' => \$CC, 'tmpdir=s' => \$tmpdir);
+           'flags=s' => \$flags, 'cc=s' => \$CC, 'tmpdir=s' => \$tmpdir,
+           'xfail=s' => \$xfail_str);
 @headers = split(/,/,join(',',@headers));
 
 # List of the headers we are testing.
@@ -347,6 +349,15 @@ while ($#headers >= 0) {
     if (/^xfail-/) {
       s/^xfail-//;
       $xfail = 1;
+    } elsif (/^xfail\[([^\]]*)\]-/) {
+      my($xfail_cond) = $1;
+      s/^xfail\[([^\]]*)\]-//;
+      # "xfail[cond]-" or "xfail[cond1|cond2|...]-" means a failure of
+      # the test is allowed if any of the listed conditions are in the
+      # --xfail command-line option argument.
+      if ($xfail_str =~ /\b($xfail_cond)\b/) {
+       $xfail = 1;
+      }
     }
     my($optional) = 0;
     if (/^optional-/) {