Combine more conformtest tests into single execution of the compiler.

In <https://sourceware.org/ml/libc-alpha/2018-11/msg00225.html>,
Florian reported that the change from conformtest.pl to conformtest.py
had increased conform/ test time, possibly because of increased
startup overhead for Python scripts.

This patch improves conformtest.py performance by arranging for as
many tests of a (header, standard) pair as possible to use a single
execution of the compiler, so it does not need to initialize and parse
the whole header under test separately for every test assertion.
Specifically, compilation tests that are not marked as "optional" or
"xfail" are combined into a single source file, and are only then run
separately if compilation of that combined file fails.  For me, this
reduces the wall clock time for the conformtest.py tests (not the
whole of the conform/ directory) from two minutes to 15 seconds.

Tested for x86_64, and with build-many-glibcs.py.

	* conform/conformtest.py (CompileSubTest.__init__): Set
	self.run_early to False.
	(ExecuteSubTest.__init__): Likewise.
	(HeaderTests.run): Try running all non-optional, non-XFAILed
	compilation tests in a single execution of the compiler.

diff --git a/ChangeLog b/ChangeLog
index 29b42207714f4e17372c4e0ccbefe7ac3fc0130b..29f649e20a58a574457df46a08f2bd208b232c6c 100644 (file)
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,11 @@
 2018-11-22  Joseph Myers  <joseph@codesourcery.com>
 
+       * conform/conformtest.py (CompileSubTest.__init__): Set
+       self.run_early to False.
+       (ExecuteSubTest.__init__): Likewise.
+       (HeaderTests.run): Try running all non-optional, non-XFAILed
+       compilation tests in a single execution of the compiler.
+
        * conform/conformtest.py (CompileSubTest): New class.
        (ExecuteSubTest): Likewise.
        (ElementTest.run): Rename to gen_subtests.  Append tests to

diff --git a/conform/conformtest.py b/conform/conformtest.py
index e86f247b9a88fe6513c88aa15e7609d7111fd7a5..31642c169e3c35eced6646eb874dfbd3a24f3f1e 100644 (file)
--- a/conform/conformtest.py
+++ b/conform/conformtest.py
@@ -33,6 +33,7 @@ class CompileSubTest(object):
 
     def __init__(self, name, text):
         """Initialize a CompileSubTest object."""
+        self.run_early = False
         self.name = name
         self.text = text
 
@@ -46,6 +47,7 @@ class ExecuteSubTest(object):
 
     def __init__(self, name, text):
         """Initialize an ExecuteSubTest object."""
+        self.run_early = False
         self.name = name
         self.text = text
 
@@ -660,6 +662,21 @@ class HeaderTests(object):
             available = self.compile_test('Availability of <%s>' % self.header,
                                           '')
             if available:
+                # As an optimization, try running all non-optional,
+                # non-XFAILed compilation tests in a single execution
+                # of the compiler.
+                combined_list = []
+                for test in self.tests:
+                    if not test.optional and not test.xfail:
+                        for subtest in test.subtests:
+                            if isinstance(subtest, CompileSubTest):
+                                combined_list.append(subtest.text)
+                                subtest.run_early = True
+                combined_ok = self.compile_test('Combined <%s> test'
+                                                % self.header,
+                                                '\n'.join(combined_list))
+                # Now run the other tests, or all tests if the
+                # combined test failed.
                 for test in self.tests:
                     # A test may run more than one subtest.  If the
                     # initial subtest for an optional symbol fails,
@@ -673,7 +690,12 @@ class HeaderTests(object):
                     self.group_ignore = False
                     self.group_skip = False
                     for subtest in test.subtests:
-                        subtest.run(self)
+                        if combined_ok and subtest.run_early:
+                            self.total += 1
+                            print('PASSCOMBINED: %s' % subtest.name)
+                            sys.stdout.flush()
+                        else:
+                            subtest.run(self)
             namespace_name = 'Namespace of <%s>' % self.header
             if available:
                 self.check_namespace(namespace_name)