Ben Darnell [Wed, 11 Oct 2023 00:39:25 +0000 (20:39 -0400)]
test: Close the thread pool in run_on_executor test
If this executor was left around it would be GC'd at an unpredictable
time and would often be reported as a failure in other circlerefs tests.
(For unknown reasons this would occur most often in i686 (i.e. 32-bit)
linux builds).
Ben Darnell [Sat, 14 Oct 2023 02:39:41 +0000 (22:39 -0400)]
*: Lint on the newest version of python too.
We previously only typechecked on the oldest version of python we
supported, incorrectly assuming nothing we depended on would be
removed. Now we typecheck on the latest version of python.
Assume support for modern version of ssl and remove some pre-SNI
code paths which rely on functions that are now removed.
Ben Darnell [Sat, 14 Oct 2023 01:27:20 +0000 (21:27 -0400)]
docs: Update intersphinx references for python 3.12
Intersphinx links are currently an unpinned dependency, so when
a new version of python is released it's possible (although relatively
rare) for it to break our links. 3.12 removed a few members of
the ssl module.
Ben Darnell [Wed, 23 Aug 2023 01:27:05 +0000 (21:27 -0400)]
escape: Use the standard library where possible
Many of these functions were necessary in Python 2, but are now
redundant. We can simply use the standard library in many cases.
The only major change is in xhtml_unescape, where we now reject
invalid character references such as surrogates and control characters.
Update docs throughout to be more specific about differences from the
standard library. Also be more complete about the ``plus`` option to
the url escaping functions.
Ben Darnell [Tue, 22 Aug 2023 03:03:39 +0000 (23:03 -0400)]
ioloop,concurrent: Fix reference cycles
In a few places we were referring to a future via a closure instead
of using the reference passed as an argument to the callback. This
sometimes causes a reference cycle that can slow GC. This commit
adds a test which covers two of the cases (chain_future and the
concurrent.future branch of add_future) while the third was found by
inspecting other calls to add_done_callback for obvious instances of
this pattern.
This test has recently become flaky on windows CI, and before
investigating further, see if it's just because the CI machines are
overloaded and subprocesses are slower on windows.
Ben Darnell [Fri, 11 Aug 2023 01:41:40 +0000 (21:41 -0400)]
httpserver_test: Add ExpectLog to fix CI
The github security advisory feature lets you make private PRs but
it apparently doesn't support CI so this log failure wasn't caught
until after the PR was merged.
Ben Darnell [Wed, 9 Aug 2023 01:55:02 +0000 (21:55 -0400)]
http1connection: Make content-length parsing more strict
Content-length and chunk size parsing now strictly matches the RFCs.
We previously used the python int() function which accepted leading
plus signs and internal underscores, which are not allowed by the
HTTP RFCs (it also accepts minus signs, but these are less problematic
in this context since they'd result in errors elsewhere)
It is important to fix this because when combined with certain proxies,
the lax parsing could result in a request smuggling vulnerability (if
both Tornado and the proxy accepted an invalid content-length but
interpreted it differently). This is known to occur with old versions
of haproxy, although the current version of haproxy is unaffected.
Ben Darnell [Thu, 27 Jul 2023 00:15:12 +0000 (20:15 -0400)]
autoreload: Add --until-success flag
This flag terminates the autoreload loop after the first successful
run. This makes it possible to cleanly shut down a process that is using
"python -m tornado.autoreload" without printing a traceback.
build(deps): bump certifi from 2022.12.7 to 2023.7.22
Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22)
Ben Darnell [Sun, 23 Jul 2023 02:10:18 +0000 (22:10 -0400)]
autoreload: Support directories in CLI wrapper
A previous commit added support for using autoreload within programs
that were started as directories; this commit supports them when
run with the -m tornado.autoreload wrapper.
This change may have side effects for file mode since we now use
runpy.run_path instead of executing the file by hand (I don't think
the run_path function existed when this code was originally written).
Ben Darnell [Fri, 14 Jul 2023 00:57:11 +0000 (20:57 -0400)]
autoreload: Support the ability to run a directory instead of a module
Running a directory has some but not all of the behavior of
running a module, including setting __spec__, so we must be careful
not to break things by assuming that __spec__ means module mode.
Ben Darnell [Sat, 8 Jul 2023 01:19:18 +0000 (21:19 -0400)]
asyncio: Remove atexit hook
This hook was added because of an only-in-CI issue, but we have since
improved our cleanup of the selector thread. As long as this passes
CI, I think we can remove the atexit hook.
Ben Darnell [Sat, 8 Jul 2023 00:04:27 +0000 (20:04 -0400)]
auth: Update facebook scope
The read_stream scope was replaced with user_posts; this change
was made to demos/facebook/facebook.py in #1674 but the corresponding
comment was not updated. The offline_access scope has also been removed
but seems irrelvant to this comment.
Ben Darnell [Fri, 9 Jun 2023 02:52:19 +0000 (22:52 -0400)]
*: Adapt to deprecation of datetime utc methods
Python 3.12 deprecates the utcnow and utcfromtimestamp methods and
discourages the use of naive datetimes to represent UTC. This was
previously the main way that Tornado used datetimes (since it was
the only option available in Python 2 before the introduction
of datetime.timezone.utc in Python 3.2).
- httpclient_test: Test-only change to test that both kinds of datetimes
are supported in If-Modified-Since (this just calls
httputil.format_timestamp)
- httputil: No functional changes, but format_timestamp's
support for both naive and aware datetimes is now tested.
- locale: format_timestamp now supports aware datetimes (in
addition to the existing support for naive datetimes).
- web: Cookie expirations internally use aware datetimes.
StaticFileHandler.get_modified_time now supports both and the
standard implementation returns aware.
It feels fragile that "naive" and "aware" datetimes are not distinct
types but subject to data-dependent behavior. This change uses
"aware" datetimes throughout Tornado, but some operations (comparisons
and subtraction) fail with mixed datetime types and if I missed any
in this change may cause errors if naive datetimes were used (where
previously naive datetimes would have been required). But that's
apparently the API we have to work with.
Ben Darnell [Mon, 19 Jun 2023 19:28:45 +0000 (15:28 -0400)]
asyncio_test: Use inequality when checking thread leaks
Sometimes we have a net reduction in the thread count
because there was an extra thread running at the time captured
the starting count, so use inequality instead of exact matches.
Ben Darnell [Wed, 17 May 2023 00:57:50 +0000 (20:57 -0400)]
asyncio: Manage the selector thread with an async generator
Async generators have a special shutdown protocol which allows
us to detect the end of the event loop and stop our thread.
This lets us clean up the thread reliably when the event loop
is started/stopped via the tornado IOLoop interfaces (which
explicitly know about the selector thread), or when the
latest asyncio interfaces are used (asyncio.run or manually
calling shutdown_asyncgens).
The thread is still leaked when older versions of the asyncio
interfaces are used (loop.close *without* shutdown_asyncgens), but
I've been unable to find a solution that does not print leak warnings
even in the event of a clean shutdown. Use of shutdown_asyncgens is
now effectively required for apps combining asyncio and tornado.
This is unfortunate since leaking a thread is relatively expensive
compared to the usual consequences of failing to call
shutdown_asyncgens, but it seems to be the best we can do.
Ben Darnell [Mon, 15 May 2023 01:03:52 +0000 (21:03 -0400)]
gen: Hold strong references to all asyncio.Tasks
Per the warning in the asyncio documentation, we need to hold a strong
reference to all asyncio Tasks to prevent premature GC. Following
discussions in cpython (https://github.com/python/cpython/issues/91887),
we hold these references on the IOLoop instance to ensure that they are
strongly held but do not cause leaks if the event loop itself is
discarded.
This is expected to fix all of the various "task was destroyed but
it is pending" warnings that have been reported. The
IOLoop._pending_tasks set is expected to become obsolete if
corresponding changes are made to asyncio in Python 3.13.
Fixes #3209
Fixes #3047
Fixes #2763
Some issues involve this warning as their most visible symptom,
but have an underlying cause that should still be addressed.
Updates #2914
Updates #2356
Ben Darnell [Sun, 14 May 2023 00:58:52 +0000 (20:58 -0400)]
web: Fix an open redirect in StaticFileHandler
Under some configurations the default_filename redirect could be exploited
to redirect to an attacker-controlled site. This change refuses to redirect
to URLs that could be misinterpreted.
A test case for the specific vulnerable configuration will follow after the
patch has been available.
Ben Darnell [Sun, 7 May 2023 21:03:33 +0000 (17:03 -0400)]
websocket: Add warning if client connection isn't closed cleanly
This gives a warning that is not dependent on GC for the issue
in #3257. This new warning covers all websocket client connections,
while the previous GC-dependent warning only affected those with
ping_interval set. This unfortunately introduces an effective
requirement to close all websocket clients explicitly for those
who are strict about warnings.
Ben Darnell [Wed, 3 May 2023 16:46:28 +0000 (12:46 -0400)]
setup: Include tox.ini in sdist
Also remove the demos directory from sdist. This inclusion was incomplete
and even if it were incomplete I don't think the sdist is a great way to
distribute these demos.
projects / thirdparty / tornado.git / log
