Victor Stinner [Fri, 30 Oct 2020 21:51:02 +0000 (22:51 +0100)]
bpo-42208: Call GC collect earlier in PyInterpreterState_Clear() (GH-23044)
The last GC collection is now done before clearing builtins and sys
dictionaries. Add also assertions to ensure that gc.collect() is no
longer called after _PyGC_Fini().
Pass also the tstate to PyInterpreterState_Clear() to pass the
correct tstate to _PyGC_CollectNoFail() and _PyGC_Fini().
Eric W [Fri, 30 Oct 2020 04:56:28 +0000 (05:56 +0100)]
bpo-42160: tempfile: Reduce overhead of pid check. (GH-22997)
The _RandomSequence class in tempfile used to check the current pid every time its rng property was used.
This commit replaces this code with `os.register_at_fork` to reduce the overhead.
Victor Stinner [Tue, 27 Oct 2020 20:34:33 +0000 (21:34 +0100)]
bpo-42161: Remove private _PyLong_Zero and _PyLong_One (GH-23003)
Use PyLong_FromLong(0) and PyLong_FromLong(1) of the public C API
instead. For Python internals, _PyLong_GetZero() and _PyLong_GetOne()
of pycore_long.h can be used.
Removed the unicodedata.ucnhash_CAPI attribute which was an internal
PyCapsule object. The related private _PyUnicode_Name_CAPI structure
was moved to the internal C API.
Rename unicodedata.ucnhash_CAPI as unicodedata._ucnhash_CAPI.
Georges Toth [Tue, 27 Oct 2020 00:31:06 +0000 (01:31 +0100)]
bpo-30681: Support invalid date format or value in email Date header (GH-22090)
I am re-submitting an older PR which was abandoned but is still relevant, #10783 by @timb07.
The issue being solved () is still relevant. The original PR #10783 was closed as
the final request changes were not applied and since abandoned.
In this new PR I have re-used the original patch plus applied both comments from the review, by @maxking and @pganssle.
For reference, here is the original PR description:
In email.utils.parsedate_to_datetime(), a failure to parse the date, or invalid date components (such as hour outside 0..23) raises an exception. Document this behaviour, and add tests to test_email/test_utils.py to confirm this behaviour.
In email.headerregistry.DateHeader.parse(), check when parsedate_to_datetime() raises an exception and add a new defect InvalidDateDefect; preserve the invalid value as the string value of the header, but set the datetime attribute to None.
Add tests to test_email/test_headerregistry.py to confirm this behaviour; also added test to test_email/test_inversion.py to confirm emails with such defective date headers round trip successfully.
This pull request incorporates feedback gratefully received from @bitdancer, @brettcannon, @Mariatta and @warsaw, and replaces the earlier PR #2254.
bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111)
* Implement running the parser a second time for the errors messages
The first parser run is only responsible for detecting whether
there is a `SyntaxError` or not. If there isn't the AST gets returned.
Otherwise, the parser is run a second time with all the `invalid_*`
rules enabled so that all the customized error messages get produced.
Victor Stinner [Mon, 26 Oct 2020 15:43:47 +0000 (16:43 +0100)]
bpo-1635741: _PyUnicode_Name_CAPI moves to internal C API (GH-22713)
The private _PyUnicode_Name_CAPI structure of the PyCapsule API
unicodedata.ucnhash_CAPI moves to the internal C API. Moreover, the
structure gets a new state member which must be passed to the
getcode() and getname() functions.
* Move Include/ucnhash.h to Include/internal/pycore_ucnhash.h
* unicodedata module is now built with Py_BUILD_CORE_MODULE.
* unicodedata: move hashAPI variable into unicodedata_module_state.
Serhiy Storchaka [Mon, 26 Oct 2020 06:43:39 +0000 (08:43 +0200)]
bpo-42006: Stop using PyDict_GetItem, PyDict_GetItemString and _PyDict_GetItemId. (GH-22648)
These functions are considered not safe because they suppress all internal errors
and can return wrong result. PyDict_GetItemString and _PyDict_GetItemId can
also silence current exception in rare cases.
Remove no longer used _PyDict_GetItemId.
Add _PyDict_ContainsId and rename _PyDict_Contains into
_PyDict_Contains_KnownHash.
Alexey Izbyshev [Mon, 26 Oct 2020 00:09:32 +0000 (03:09 +0300)]
bpo-42146: Fix memory leak in subprocess.Popen() in case of uid/gid overflow (GH-22966)
Fix memory leak in subprocess.Popen() in case of uid/gid overflow
Also add a test that would catch this leak with `--huntrleaks`.
Alas, the test for `extra_groups` also exposes an inconsistency
in our error reporting: we use a custom ValueError for `extra_groups`,
but propagate OverflowError for `user` and `group`.
Gregory P. Smith [Sat, 24 Oct 2020 19:07:35 +0000 (12:07 -0700)]
bpo-35823: Allow setsid() after vfork() on Linux. (GH-22945)
It should just be a syscall updating a couple of fields in the kernel side
process info. Confirming, in glibc is appears to be a shim for the setsid
syscall (based on not finding any code implementing anything special for it)
and in uclibc (*much* easier to read) it is clearly just a setsid syscall shim.
A breadcrumb _suggesting_ that it is not allowed on Darwin/macOS comes from
a commit in emacs: https://lists.gnu.org/archive/html/bug-gnu-emacs/2017-04/msg00297.html
but I don't have a way to verify if that is true or not.
As we are not supporting vfork on macOS today I just left a note in a comment.
Alexey Izbyshev [Sat, 24 Oct 2020 17:47:38 +0000 (20:47 +0300)]
bpo-35823: subprocess: Fix handling of pthread_sigmask() errors (GH-22944)
Using POSIX_CALL() is incorrect since pthread_sigmask() returns
the error number instead of setting errno.
Also handle failure of the first call to pthread_sigmask()
in the parent process, and explain why we don't handle failure
of the second call in a comment.
Alexey Izbyshev [Sat, 24 Oct 2020 00:47:01 +0000 (03:47 +0300)]
bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe (GH-11671)
* bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe
When used to run a new executable image, fork() is not a good choice
for process creation, especially if the parent has a large working set:
fork() needs to copy page tables, which is slow, and may fail on systems
where overcommit is disabled, despite that the child is not going to
touch most of its address space.
Currently, subprocess is capable of using posix_spawn() instead, which
normally provides much better performance. However, posix_spawn() does not
support many of child setup operations exposed by subprocess.Popen().
Most notably, it's not possible to express `close_fds=True`, which
happens to be the default, via posix_spawn(). As a result, most users
can't benefit from faster process creation, at least not without
changing their code.
However, Linux provides vfork() system call, which creates a new process
without copying the address space of the parent, and which is actually
used by C libraries to efficiently implement posix_spawn(). Due to sharing
of the address space and even the stack with the parent, extreme care
is required to use vfork(). At least the following restrictions must hold:
* No signal handlers must execute in the child process. Otherwise, they
might clobber memory shared with the parent, potentially confusing it.
* Any library function called after vfork() in the child must be
async-signal-safe (as for fork()), but it must also not interact with any
library state in a way that might break due to address space sharing
and/or lack of any preparations performed by libraries on normal fork().
POSIX.1 permits to call only execve() and _exit(), and later revisions
remove vfork() specification entirely. In practice, however, almost all
operations needed by subprocess.Popen() can be safely implemented on
Linux.
* Due to sharing of the stack with the parent, the child must be careful
not to clobber local variables that are alive across vfork() call.
Compilers are normally aware of this and take extra care with vfork()
(and setjmp(), which has a similar problem).
* In case the parent is privileged, special attention must be paid to vfork()
use, because sharing an address space across different privilege domains
is insecure[1].
This patch adds support for using vfork() instead of fork() on Linux
when it's possible to do safely given the above. In particular:
* vfork() is not used if credential switch is requested. The reverse case
(simple subprocess.Popen() but another application thread switches
credentials concurrently) is not possible for pure-Python apps because
subprocess.Popen() and functions like os.setuid() are mutually excluded
via GIL. We might also consider to add a way to opt-out of vfork() (and
posix_spawn() on platforms where it might be implemented via vfork()) in
a future PR.
* vfork() is not used if `preexec_fn != None`.
With this change, subprocess will still use posix_spawn() if possible, but
will fallback to vfork() on Linux in most cases, and, failing that,
to fork().
[1] https://ewontfix.com/7
Co-authored-by: Gregory P. Smith [Google LLC] <gps@google.com>
dependabot[bot] [Fri, 23 Oct 2020 21:32:12 +0000 (14:32 -0700)]
build(deps): bump actions/upload-artifact from v1 to v2.2.0 (GH-22920)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from v1 to v2.2.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/actions/upload-artifact/releases">actions/upload-artifact's releases</a>.</em></p>
<blockquote>
<h2>v2.2.0</h2>
<ul>
<li>Support for artifact retention</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/actions/upload-artifact/commit/27bce4eee761b5bc643f46a8dfb41b430c8d05f6"><code>27bce4e</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/upload-artifact/issues/112">#112</a> from thboop/main</li>
<li><a href="https://github.com/actions/upload-artifact/commit/f8b42f7ab442a66b3d51a5ca02855b194a36ae2d"><code>f8b42f7</code></a> update licensed files</li>
<li><a href="https://github.com/actions/upload-artifact/commit/2106e8cf10e032ca9d5724c4c676543febe74f0b"><code>2106e8c</code></a> update contributing.md</li>
<li><a href="https://github.com/actions/upload-artifact/commit/db66798ebcfbaa7f3f8ff66bce013213265c30d1"><code>db66798</code></a> Ignore Generated Files in Git PR's</li>
<li><a href="https://github.com/actions/upload-artifact/commit/d359fd0772ed6802a84728dd6b09ec99f41a67b7"><code>d359fd0</code></a> Manual Verification of licenses</li>
<li><a href="https://github.com/actions/upload-artifact/commit/350822c32f871c559dbe1667c24424e06c4f03e3"><code>350822c</code></a> Add Licensed Workflow and config</li>
<li><a href="https://github.com/actions/upload-artifact/commit/abecf4abf4b70bc636949d61150be883b87416c2"><code>abecf4a</code></a> Updated README.md (<a href="https://github-redirect.dependabot.com/actions/upload-artifact/issues/118">#118</a>)</li>
<li><a href="https://github.com/actions/upload-artifact/commit/604e071d21906545dedcfaf210deae74f8c5276a"><code>604e071</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/upload-artifact/issues/126">#126</a> from yacaovsnc/main</li>
<li><a href="https://github.com/actions/upload-artifact/commit/4560c23b396d494f0cb7066e1d6e258e8feb8051"><code>4560c23</code></a> Check for invalid retention-days input</li>
<li><a href="https://github.com/actions/upload-artifact/commit/59018c2f85dd0e101b75544aa87f13bb0c94e0b7"><code>59018c2</code></a> Add an option to specify retention period</li>
<li>Additional commits viewable in <a href="https://github.com/actions/upload-artifact/compare/v1...27bce4eee761b5bc643f46a8dfb41b430c8d05f6">compare view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
dependabot[bot] [Fri, 23 Oct 2020 21:29:41 +0000 (14:29 -0700)]
build(deps): bump actions/cache from v1 to v2.1.2 (GH-22919)
Bumps [actions/cache](https://github.com/actions/cache) from v1 to v2.1.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/actions/cache/releases">actions/cache's releases</a>.</em></p>
<blockquote>
<h2>v2.1.2</h2>
<ul>
<li>Adds input to limit the chunk upload size, useful for self-hosted runners with slower upload speeds</li>
<li>No-op when executing on GHES</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/actions/cache/commit/d1255ad9362389eac595a9ae406b8e8cb3331f16"><code>d1255ad</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/cache/issues/424">#424</a> from actions/dhadka/upload-chunk-size</li>
<li><a href="https://github.com/actions/cache/commit/68cfb2ccb73b1982be3fa55e3d7c842697d7f1ed"><code>68cfb2c</code></a> Add units to description</li>
<li><a href="https://github.com/actions/cache/commit/cce3c03a74623545a53c433d301f3f7725c72454"><code>cce3c03</code></a> Add new input to action.yml</li>
<li><a href="https://github.com/actions/cache/commit/4bceb75b5b7743784c63c94b81c50a485cbdcda0"><code>4bceb75</code></a> Use parseInt instead of Number to handle empty strings</li>
<li><a href="https://github.com/actions/cache/commit/a6f1f4b32eec85780fedc5b354a583e9b2999100"><code>a6f1f4b</code></a> Adds input for upload chunk size</li>
<li><a href="https://github.com/actions/cache/commit/d606e039ae32f64a8593bf4a37b0bf205c695237"><code>d606e03</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/cache/issues/421">#421</a> from actions/dhadka/ghes</li>
<li><a href="https://github.com/actions/cache/commit/d3e4f218f30bd71a2c29e2b2a1e4f811f4327162"><code>d3e4f21</code></a> Use warning instead of info</li>
<li><a href="https://github.com/actions/cache/commit/55a58944386e69f7c5bad52ef43a61c578b6c1c6"><code>55a5894</code></a> Update dist</li>
<li><a href="https://github.com/actions/cache/commit/3f6dfcbcc44a8e2fd9e539c1dd15af6559e74ced"><code>3f6dfcb</code></a> Merge branch 'main' of <a href="http://github.com/actions/cache">http://github.com/actions/cache</a> into dhadka/ghes</li>
<li><a href="https://github.com/actions/cache/commit/0f71d4ac9a7f4c36aba5ac3cfc4567d2d4eae813"><code>0f71d4a</code></a> Add tests for isGhes</li>
<li>Additional commits viewable in <a href="https://github.com/actions/cache/compare/v1...d1255ad9362389eac595a9ae406b8e8cb3331f16">compare view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Eric Snow [Fri, 23 Oct 2020 00:42:51 +0000 (18:42 -0600)]
bpo-36876: Fix the C analyzer tool. (GH-22841)
The original tool wasn't working right and it was simpler to create a new one, partially re-using some of the old code. At this point the tool runs properly on the master. (Try: ./python Tools/c-analyzer/c-analyzer.py analyze.) It take ~40 seconds on my machine to analyze the full CPython code base.
Note that we'll need to iron out some OS-specific stuff (e.g. preprocessor). We're okay though since this tool isn't used yet in our workflow. We will also need to verify the analysis results in detail before activating the check in CI, though I'm pretty sure it's close.
Add documentation to help diagnose CDLL dependent DLL loading errors
on windows for OSError with message:
"[WinError 126] The specified module could not be found"
This error is otherwise difficult to diagnose.
Dong-hee Na [Thu, 22 Oct 2020 09:44:18 +0000 (18:44 +0900)]
_testmultiphase: Fix possible ref leak (GH-22881)
This is just test code, but sometimes external contributors reference the code snippets from test code.
`PyModule_AddObject` should be handled in the proper way.
Géry Ogam [Wed, 21 Oct 2020 21:17:35 +0000 (23:17 +0200)]
bpo-35181: Correct importlib documentation for some module attributes (GH-15190)
@ericsnowcurrently This PR will change the following:
In the library documentation importlib.rst:
- `module.__package__` can be `module.__name__` for packages;
- `spec.parent` can be `spec.__name__` for packages;
- `spec.loader` is not `None` for namespaces packages.
In the language documentation import.rst:
- `spec.loader` is not `None` for namespace packages.
kpinc [Wed, 21 Oct 2020 20:34:15 +0000 (15:34 -0500)]
Fix bpo-39416: Change "Numeric" to lower case; an english word, not a class name (GH-22867)
This is a trivial fix to [bpo-39416](), which didn't come up until it was already committed
```
Change "Numeric" to "numeric".
I believe this is trivial enough to not need an issue or a NEWS entry, although
I'm unclear on what branches the original pull request received backports.
```
kpinc [Wed, 21 Oct 2020 17:13:50 +0000 (12:13 -0500)]
bpo-39416: Document some restrictions on the default string representations of numeric classes (GH-18111)
[bpo-39416](): Document string representations of the Numeric classes
This is a change to the specification of the Python language.
The idea here is to put sane minimal limits on the Python language's default
representations of its Numeric classes. That way "Marty's Robotic Massage Parlor
and Python Interpreter" implementation of Python won't do anything too
crazy.
Some discussion in the email thread:
Subject: Documenting Python's float.__str__()
https://mail.python.org/archives/list/python-dev@python.org/thread/FV22TKT3S2Q3P7PNN6MCXI6IX3HRRNAL/
Максим [Wed, 21 Oct 2020 02:08:19 +0000 (05:08 +0300)]
bpo-23706: Add newline parameter to pathlib.Path.write_text (GH-22420) (GH-22420)
* Add _newline_ parameter to `pathlib.Path.write_text()`
* Update documentation of `pathlib.Path.write_text()`
* Add test case for `pathlib.Path.write_text()` calls with _newline_ parameter passed