From: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Date: Sun, 12 Feb 2023 06:33:13 +0000 (-0800)
Subject: gh-89792: Limit test_tools freeze test build parallelism based on the number of cores... 
X-Git-Tag: v3.11.3~148
X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=cec99ed1a7f76dda36816be0656e6f64e1d3ae1d;p=thirdparty%2FPython%2Fcpython.git

gh-89792: Limit test_tools freeze test build parallelism based on the number of cores (GH-101841)


unhardcode freeze test build parallelism. base it on the number of cpus, don't use more than max(2, os.cpu_count()/3).
(cherry picked from commit dfc2e065a2e71011017077e549cd2f9bf4944c54)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
---

diff --git a/Misc/NEWS.d/next/Tests/2023-02-11-20-28-08.gh-issue-89792.S-Y5BZ.rst b/Misc/NEWS.d/next/Tests/2023-02-11-20-28-08.gh-issue-89792.S-Y5BZ.rst
index a3a3070d7f37..9de278919ef2 100644
--- a/Misc/NEWS.d/next/Tests/2023-02-11-20-28-08.gh-issue-89792.S-Y5BZ.rst
+++ b/Misc/NEWS.d/next/Tests/2023-02-11-20-28-08.gh-issue-89792.S-Y5BZ.rst
@@ -1,3 +1,4 @@
-``test_tools`` now copies up to 10x less source data to a temporary
-directory during the ``freeze`` test by ignoring git metadata and other
-artifacts.
+``test_tools`` now copies up to 10x less source data to a temporary directory
+during the ``freeze`` test by ignoring git metadata and other artifacts.  It
+also limits its python build parallelism based on os.cpu_count instead of hard
+coding it as 8 cores.
diff --git a/Tools/freeze/test/freeze.py b/Tools/freeze/test/freeze.py
index 0ae983b15c98..b4c76ff36a87 100644
--- a/Tools/freeze/test/freeze.py
+++ b/Tools/freeze/test/freeze.py
@@ -163,16 +163,25 @@ def prepare(script=None, outdir=None):
     if not MAKE:
         raise UnsupportedError('make')
 
+    cores = os.cpu_count()
+    if cores and cores >= 3:
+        # this test is most often run as part of the whole suite with a lot
+        # of other tests running in parallel, from 1-2 vCPU systems up to
+        # people's NNN core beasts. Don't attempt to use it all.
+        parallel = f'-j{cores*2//3}'
+    else:
+        parallel = '-j2'
+
     # Build python.
-    print(f'building python in {builddir}...')
+    print(f'building python {parallel=} in {builddir}...')
     if os.path.exists(os.path.join(srcdir, 'Makefile')):
         # Out-of-tree builds require a clean srcdir.
         _run_quiet([MAKE, '-C', srcdir, 'clean'])
-    _run_quiet([MAKE, '-C', builddir, '-j8'])
+    _run_quiet([MAKE, '-C', builddir, parallel])
 
     # Install the build.
     print(f'installing python into {prefix}...')
-    _run_quiet([MAKE, '-C', builddir, '-j8', 'install'])
+    _run_quiet([MAKE, '-C', builddir, 'install'])
     python = os.path.join(prefix, 'bin', 'python3')
 
     return outdir, scriptfile, python