Alexey Izbyshev [Mon, 26 Oct 2020 00:09:32 +0000 (03:09 +0300)]
bpo-42146: Fix memory leak in subprocess.Popen() in case of uid/gid overflow (GH-22966)
Fix memory leak in subprocess.Popen() in case of uid/gid overflow
Also add a test that would catch this leak with `--huntrleaks`.
Alas, the test for `extra_groups` also exposes an inconsistency
in our error reporting: we use a custom ValueError for `extra_groups`,
but propagate OverflowError for `user` and `group`.
Gregory P. Smith [Sat, 24 Oct 2020 19:07:35 +0000 (12:07 -0700)]
bpo-35823: Allow setsid() after vfork() on Linux. (GH-22945)
It should just be a syscall updating a couple of fields in the kernel side
process info. Confirming, in glibc is appears to be a shim for the setsid
syscall (based on not finding any code implementing anything special for it)
and in uclibc (*much* easier to read) it is clearly just a setsid syscall shim.
A breadcrumb _suggesting_ that it is not allowed on Darwin/macOS comes from
a commit in emacs: https://lists.gnu.org/archive/html/bug-gnu-emacs/2017-04/msg00297.html
but I don't have a way to verify if that is true or not.
As we are not supporting vfork on macOS today I just left a note in a comment.
Alexey Izbyshev [Sat, 24 Oct 2020 17:47:38 +0000 (20:47 +0300)]
bpo-35823: subprocess: Fix handling of pthread_sigmask() errors (GH-22944)
Using POSIX_CALL() is incorrect since pthread_sigmask() returns
the error number instead of setting errno.
Also handle failure of the first call to pthread_sigmask()
in the parent process, and explain why we don't handle failure
of the second call in a comment.
Alexey Izbyshev [Sat, 24 Oct 2020 00:47:01 +0000 (03:47 +0300)]
bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe (GH-11671)
* bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe
When used to run a new executable image, fork() is not a good choice
for process creation, especially if the parent has a large working set:
fork() needs to copy page tables, which is slow, and may fail on systems
where overcommit is disabled, despite that the child is not going to
touch most of its address space.
Currently, subprocess is capable of using posix_spawn() instead, which
normally provides much better performance. However, posix_spawn() does not
support many of child setup operations exposed by subprocess.Popen().
Most notably, it's not possible to express `close_fds=True`, which
happens to be the default, via posix_spawn(). As a result, most users
can't benefit from faster process creation, at least not without
changing their code.
However, Linux provides vfork() system call, which creates a new process
without copying the address space of the parent, and which is actually
used by C libraries to efficiently implement posix_spawn(). Due to sharing
of the address space and even the stack with the parent, extreme care
is required to use vfork(). At least the following restrictions must hold:
* No signal handlers must execute in the child process. Otherwise, they
might clobber memory shared with the parent, potentially confusing it.
* Any library function called after vfork() in the child must be
async-signal-safe (as for fork()), but it must also not interact with any
library state in a way that might break due to address space sharing
and/or lack of any preparations performed by libraries on normal fork().
POSIX.1 permits to call only execve() and _exit(), and later revisions
remove vfork() specification entirely. In practice, however, almost all
operations needed by subprocess.Popen() can be safely implemented on
Linux.
* Due to sharing of the stack with the parent, the child must be careful
not to clobber local variables that are alive across vfork() call.
Compilers are normally aware of this and take extra care with vfork()
(and setjmp(), which has a similar problem).
* In case the parent is privileged, special attention must be paid to vfork()
use, because sharing an address space across different privilege domains
is insecure[1].
This patch adds support for using vfork() instead of fork() on Linux
when it's possible to do safely given the above. In particular:
* vfork() is not used if credential switch is requested. The reverse case
(simple subprocess.Popen() but another application thread switches
credentials concurrently) is not possible for pure-Python apps because
subprocess.Popen() and functions like os.setuid() are mutually excluded
via GIL. We might also consider to add a way to opt-out of vfork() (and
posix_spawn() on platforms where it might be implemented via vfork()) in
a future PR.
* vfork() is not used if `preexec_fn != None`.
With this change, subprocess will still use posix_spawn() if possible, but
will fallback to vfork() on Linux in most cases, and, failing that,
to fork().
[1] https://ewontfix.com/7
Co-authored-by: Gregory P. Smith [Google LLC] <gps@google.com>
dependabot[bot] [Fri, 23 Oct 2020 21:32:12 +0000 (14:32 -0700)]
build(deps): bump actions/upload-artifact from v1 to v2.2.0 (GH-22920)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from v1 to v2.2.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/actions/upload-artifact/releases">actions/upload-artifact's releases</a>.</em></p>
<blockquote>
<h2>v2.2.0</h2>
<ul>
<li>Support for artifact retention</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/actions/upload-artifact/commit/27bce4eee761b5bc643f46a8dfb41b430c8d05f6"><code>27bce4e</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/upload-artifact/issues/112">#112</a> from thboop/main</li>
<li><a href="https://github.com/actions/upload-artifact/commit/f8b42f7ab442a66b3d51a5ca02855b194a36ae2d"><code>f8b42f7</code></a> update licensed files</li>
<li><a href="https://github.com/actions/upload-artifact/commit/2106e8cf10e032ca9d5724c4c676543febe74f0b"><code>2106e8c</code></a> update contributing.md</li>
<li><a href="https://github.com/actions/upload-artifact/commit/db66798ebcfbaa7f3f8ff66bce013213265c30d1"><code>db66798</code></a> Ignore Generated Files in Git PR's</li>
<li><a href="https://github.com/actions/upload-artifact/commit/d359fd0772ed6802a84728dd6b09ec99f41a67b7"><code>d359fd0</code></a> Manual Verification of licenses</li>
<li><a href="https://github.com/actions/upload-artifact/commit/350822c32f871c559dbe1667c24424e06c4f03e3"><code>350822c</code></a> Add Licensed Workflow and config</li>
<li><a href="https://github.com/actions/upload-artifact/commit/abecf4abf4b70bc636949d61150be883b87416c2"><code>abecf4a</code></a> Updated README.md (<a href="https://github-redirect.dependabot.com/actions/upload-artifact/issues/118">#118</a>)</li>
<li><a href="https://github.com/actions/upload-artifact/commit/604e071d21906545dedcfaf210deae74f8c5276a"><code>604e071</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/upload-artifact/issues/126">#126</a> from yacaovsnc/main</li>
<li><a href="https://github.com/actions/upload-artifact/commit/4560c23b396d494f0cb7066e1d6e258e8feb8051"><code>4560c23</code></a> Check for invalid retention-days input</li>
<li><a href="https://github.com/actions/upload-artifact/commit/59018c2f85dd0e101b75544aa87f13bb0c94e0b7"><code>59018c2</code></a> Add an option to specify retention period</li>
<li>Additional commits viewable in <a href="https://github.com/actions/upload-artifact/compare/v1...27bce4eee761b5bc643f46a8dfb41b430c8d05f6">compare view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
dependabot[bot] [Fri, 23 Oct 2020 21:29:41 +0000 (14:29 -0700)]
build(deps): bump actions/cache from v1 to v2.1.2 (GH-22919)
Bumps [actions/cache](https://github.com/actions/cache) from v1 to v2.1.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/actions/cache/releases">actions/cache's releases</a>.</em></p>
<blockquote>
<h2>v2.1.2</h2>
<ul>
<li>Adds input to limit the chunk upload size, useful for self-hosted runners with slower upload speeds</li>
<li>No-op when executing on GHES</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/actions/cache/commit/d1255ad9362389eac595a9ae406b8e8cb3331f16"><code>d1255ad</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/cache/issues/424">#424</a> from actions/dhadka/upload-chunk-size</li>
<li><a href="https://github.com/actions/cache/commit/68cfb2ccb73b1982be3fa55e3d7c842697d7f1ed"><code>68cfb2c</code></a> Add units to description</li>
<li><a href="https://github.com/actions/cache/commit/cce3c03a74623545a53c433d301f3f7725c72454"><code>cce3c03</code></a> Add new input to action.yml</li>
<li><a href="https://github.com/actions/cache/commit/4bceb75b5b7743784c63c94b81c50a485cbdcda0"><code>4bceb75</code></a> Use parseInt instead of Number to handle empty strings</li>
<li><a href="https://github.com/actions/cache/commit/a6f1f4b32eec85780fedc5b354a583e9b2999100"><code>a6f1f4b</code></a> Adds input for upload chunk size</li>
<li><a href="https://github.com/actions/cache/commit/d606e039ae32f64a8593bf4a37b0bf205c695237"><code>d606e03</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/cache/issues/421">#421</a> from actions/dhadka/ghes</li>
<li><a href="https://github.com/actions/cache/commit/d3e4f218f30bd71a2c29e2b2a1e4f811f4327162"><code>d3e4f21</code></a> Use warning instead of info</li>
<li><a href="https://github.com/actions/cache/commit/55a58944386e69f7c5bad52ef43a61c578b6c1c6"><code>55a5894</code></a> Update dist</li>
<li><a href="https://github.com/actions/cache/commit/3f6dfcbcc44a8e2fd9e539c1dd15af6559e74ced"><code>3f6dfcb</code></a> Merge branch 'main' of <a href="http://github.com/actions/cache">http://github.com/actions/cache</a> into dhadka/ghes</li>
<li><a href="https://github.com/actions/cache/commit/0f71d4ac9a7f4c36aba5ac3cfc4567d2d4eae813"><code>0f71d4a</code></a> Add tests for isGhes</li>
<li>Additional commits viewable in <a href="https://github.com/actions/cache/compare/v1...d1255ad9362389eac595a9ae406b8e8cb3331f16">compare view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Eric Snow [Fri, 23 Oct 2020 00:42:51 +0000 (18:42 -0600)]
bpo-36876: Fix the C analyzer tool. (GH-22841)
The original tool wasn't working right and it was simpler to create a new one, partially re-using some of the old code. At this point the tool runs properly on the master. (Try: ./python Tools/c-analyzer/c-analyzer.py analyze.) It take ~40 seconds on my machine to analyze the full CPython code base.
Note that we'll need to iron out some OS-specific stuff (e.g. preprocessor). We're okay though since this tool isn't used yet in our workflow. We will also need to verify the analysis results in detail before activating the check in CI, though I'm pretty sure it's close.
Add documentation to help diagnose CDLL dependent DLL loading errors
on windows for OSError with message:
"[WinError 126] The specified module could not be found"
This error is otherwise difficult to diagnose.
Dong-hee Na [Thu, 22 Oct 2020 09:44:18 +0000 (18:44 +0900)]
_testmultiphase: Fix possible ref leak (GH-22881)
This is just test code, but sometimes external contributors reference the code snippets from test code.
`PyModule_AddObject` should be handled in the proper way.
Géry Ogam [Wed, 21 Oct 2020 21:17:35 +0000 (23:17 +0200)]
bpo-35181: Correct importlib documentation for some module attributes (GH-15190)
@ericsnowcurrently This PR will change the following:
In the library documentation importlib.rst:
- `module.__package__` can be `module.__name__` for packages;
- `spec.parent` can be `spec.__name__` for packages;
- `spec.loader` is not `None` for namespaces packages.
In the language documentation import.rst:
- `spec.loader` is not `None` for namespace packages.
kpinc [Wed, 21 Oct 2020 20:34:15 +0000 (15:34 -0500)]
Fix bpo-39416: Change "Numeric" to lower case; an english word, not a class name (GH-22867)
This is a trivial fix to [bpo-39416](), which didn't come up until it was already committed
```
Change "Numeric" to "numeric".
I believe this is trivial enough to not need an issue or a NEWS entry, although
I'm unclear on what branches the original pull request received backports.
```
kpinc [Wed, 21 Oct 2020 17:13:50 +0000 (12:13 -0500)]
bpo-39416: Document some restrictions on the default string representations of numeric classes (GH-18111)
[bpo-39416](): Document string representations of the Numeric classes
This is a change to the specification of the Python language.
The idea here is to put sane minimal limits on the Python language's default
representations of its Numeric classes. That way "Marty's Robotic Massage Parlor
and Python Interpreter" implementation of Python won't do anything too
crazy.
Some discussion in the email thread:
Subject: Documenting Python's float.__str__()
https://mail.python.org/archives/list/python-dev@python.org/thread/FV22TKT3S2Q3P7PNN6MCXI6IX3HRRNAL/
Максим [Wed, 21 Oct 2020 02:08:19 +0000 (05:08 +0300)]
bpo-23706: Add newline parameter to pathlib.Path.write_text (GH-22420) (GH-22420)
* Add _newline_ parameter to `pathlib.Path.write_text()`
* Update documentation of `pathlib.Path.write_text()`
* Add test case for `pathlib.Path.write_text()` calls with _newline_ parameter passed
Paul Moore [Tue, 20 Oct 2020 20:02:24 +0000 (21:02 +0100)]
bpo-42041: Clarify how subprocess searches for the executable (GH-22715)
Clarify in the subprocess documentation how searching for the executable to run works, noting that ``sys.executable`` is the recommended way to find the current interpreter.
Ruben Vorderman [Mon, 19 Oct 2020 23:30:02 +0000 (01:30 +0200)]
bpo-41586: Add pipesize parameter to subprocess & F_GETPIPE_SZ and F_SETPIPE_SZ to fcntl. (GH-21921)
* Add F_SETPIPE_SZ and F_GETPIPE_SZ to fcntl module
* Add pipesize parameter for subprocess.Popen class
This will allow the user to control the size of the pipes.
On linux the default is 64K. When a pipe is full it blocks for writing.
When a pipe is empty it blocks for reading. On processes that are
very fast this can lead to a lot of wasted CPU cycles. On a typical
Linux system the max pipe size is 1024K which is much better.
For high performance-oriented libraries such as xopen it is nice to
be able to set the pipe size.
The workaround without this feature is to use my_popen_process.stdout.fileno() in
conjuction with fcntl and 1031 (value of F_SETPIPE_SZ) to acquire this behavior.
Mark Sapiro [Mon, 19 Oct 2020 22:49:19 +0000 (15:49 -0700)]
bpo-27321 Fix email.generator.py to not replace a non-existent header. (GH-18074)
This PR replaces #1977. The reason for the replacement is two-fold.
The fix itself is different is that if the CTE header doesn't exist in the original message, it is inserted. This is important because the new CTE could be quoted-printable whereas the original is implicit 8bit.
Also the tests are different. The test_nonascii_as_string_without_cte test in #1977 doesn't actually test the issue in that it passes without the fix. The test_nonascii_as_string_without_content_type_and_cte test is improved here, and even though it doesn't fail without the fix, it is included for completeness.
Zackery Spytz [Mon, 19 Oct 2020 22:47:37 +0000 (16:47 -0600)]
bpo-41845: Move PyObject_GenericGetDict() back into the limited API (GH22646)
It was moved out of the limited API in 7d95e4072169911b228c9e42367afb5f17fd3db0.
This change re-enables it from 3.10, to avoid generating invalid extension modules for earlier versions.