Ben Darnell [Fri, 11 Aug 2023 01:41:40 +0000 (21:41 -0400)]
httpserver_test: Add ExpectLog to fix CI
The github security advisory feature lets you make private PRs but
it apparently doesn't support CI so this log failure wasn't caught
until after the PR was merged.
Ben Darnell [Wed, 9 Aug 2023 01:55:02 +0000 (21:55 -0400)]
http1connection: Make content-length parsing more strict
Content-length and chunk size parsing now strictly matches the RFCs.
We previously used the python int() function which accepted leading
plus signs and internal underscores, which are not allowed by the
HTTP RFCs (it also accepts minus signs, but these are less problematic
in this context since they'd result in errors elsewhere)
It is important to fix this because when combined with certain proxies,
the lax parsing could result in a request smuggling vulnerability (if
both Tornado and the proxy accepted an invalid content-length but
interpreted it differently). This is known to occur with old versions
of haproxy, although the current version of haproxy is unaffected.
Ben Darnell [Thu, 27 Jul 2023 00:15:12 +0000 (20:15 -0400)]
autoreload: Add --until-success flag
This flag terminates the autoreload loop after the first successful
run. This makes it possible to cleanly shut down a process that is using
"python -m tornado.autoreload" without printing a traceback.
build(deps): bump certifi from 2022.12.7 to 2023.7.22
Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22)
Ben Darnell [Sun, 23 Jul 2023 02:10:18 +0000 (22:10 -0400)]
autoreload: Support directories in CLI wrapper
A previous commit added support for using autoreload within programs
that were started as directories; this commit supports them when
run with the -m tornado.autoreload wrapper.
This change may have side effects for file mode since we now use
runpy.run_path instead of executing the file by hand (I don't think
the run_path function existed when this code was originally written).
Ben Darnell [Fri, 14 Jul 2023 00:57:11 +0000 (20:57 -0400)]
autoreload: Support the ability to run a directory instead of a module
Running a directory has some but not all of the behavior of
running a module, including setting __spec__, so we must be careful
not to break things by assuming that __spec__ means module mode.
Ben Darnell [Sat, 8 Jul 2023 01:19:18 +0000 (21:19 -0400)]
asyncio: Remove atexit hook
This hook was added because of an only-in-CI issue, but we have since
improved our cleanup of the selector thread. As long as this passes
CI, I think we can remove the atexit hook.
Ben Darnell [Sat, 8 Jul 2023 00:04:27 +0000 (20:04 -0400)]
auth: Update facebook scope
The read_stream scope was replaced with user_posts; this change
was made to demos/facebook/facebook.py in #1674 but the corresponding
comment was not updated. The offline_access scope has also been removed
but seems irrelvant to this comment.
Ben Darnell [Fri, 9 Jun 2023 02:52:19 +0000 (22:52 -0400)]
*: Adapt to deprecation of datetime utc methods
Python 3.12 deprecates the utcnow and utcfromtimestamp methods and
discourages the use of naive datetimes to represent UTC. This was
previously the main way that Tornado used datetimes (since it was
the only option available in Python 2 before the introduction
of datetime.timezone.utc in Python 3.2).
- httpclient_test: Test-only change to test that both kinds of datetimes
are supported in If-Modified-Since (this just calls
httputil.format_timestamp)
- httputil: No functional changes, but format_timestamp's
support for both naive and aware datetimes is now tested.
- locale: format_timestamp now supports aware datetimes (in
addition to the existing support for naive datetimes).
- web: Cookie expirations internally use aware datetimes.
StaticFileHandler.get_modified_time now supports both and the
standard implementation returns aware.
It feels fragile that "naive" and "aware" datetimes are not distinct
types but subject to data-dependent behavior. This change uses
"aware" datetimes throughout Tornado, but some operations (comparisons
and subtraction) fail with mixed datetime types and if I missed any
in this change may cause errors if naive datetimes were used (where
previously naive datetimes would have been required). But that's
apparently the API we have to work with.
Ben Darnell [Mon, 19 Jun 2023 19:28:45 +0000 (15:28 -0400)]
asyncio_test: Use inequality when checking thread leaks
Sometimes we have a net reduction in the thread count
because there was an extra thread running at the time captured
the starting count, so use inequality instead of exact matches.
Ben Darnell [Wed, 17 May 2023 00:57:50 +0000 (20:57 -0400)]
asyncio: Manage the selector thread with an async generator
Async generators have a special shutdown protocol which allows
us to detect the end of the event loop and stop our thread.
This lets us clean up the thread reliably when the event loop
is started/stopped via the tornado IOLoop interfaces (which
explicitly know about the selector thread), or when the
latest asyncio interfaces are used (asyncio.run or manually
calling shutdown_asyncgens).
The thread is still leaked when older versions of the asyncio
interfaces are used (loop.close *without* shutdown_asyncgens), but
I've been unable to find a solution that does not print leak warnings
even in the event of a clean shutdown. Use of shutdown_asyncgens is
now effectively required for apps combining asyncio and tornado.
This is unfortunate since leaking a thread is relatively expensive
compared to the usual consequences of failing to call
shutdown_asyncgens, but it seems to be the best we can do.
Ben Darnell [Mon, 15 May 2023 01:03:52 +0000 (21:03 -0400)]
gen: Hold strong references to all asyncio.Tasks
Per the warning in the asyncio documentation, we need to hold a strong
reference to all asyncio Tasks to prevent premature GC. Following
discussions in cpython (https://github.com/python/cpython/issues/91887),
we hold these references on the IOLoop instance to ensure that they are
strongly held but do not cause leaks if the event loop itself is
discarded.
This is expected to fix all of the various "task was destroyed but
it is pending" warnings that have been reported. The
IOLoop._pending_tasks set is expected to become obsolete if
corresponding changes are made to asyncio in Python 3.13.
Fixes #3209
Fixes #3047
Fixes #2763
Some issues involve this warning as their most visible symptom,
but have an underlying cause that should still be addressed.
Updates #2914
Updates #2356
Ben Darnell [Sun, 14 May 2023 00:58:52 +0000 (20:58 -0400)]
web: Fix an open redirect in StaticFileHandler
Under some configurations the default_filename redirect could be exploited
to redirect to an attacker-controlled site. This change refuses to redirect
to URLs that could be misinterpreted.
A test case for the specific vulnerable configuration will follow after the
patch has been available.
Ben Darnell [Sun, 7 May 2023 21:03:33 +0000 (17:03 -0400)]
websocket: Add warning if client connection isn't closed cleanly
This gives a warning that is not dependent on GC for the issue
in #3257. This new warning covers all websocket client connections,
while the previous GC-dependent warning only affected those with
ping_interval set. This unfortunately introduces an effective
requirement to close all websocket clients explicitly for those
who are strict about warnings.
Ben Darnell [Wed, 3 May 2023 16:46:28 +0000 (12:46 -0400)]
setup: Include tox.ini in sdist
Also remove the demos directory from sdist. This inclusion was incomplete
and even if it were incomplete I don't think the sdist is a great way to
distribute these demos.
Ben Darnell [Tue, 2 May 2023 16:54:20 +0000 (12:54 -0400)]
ioloop: Deprecate add_callback_from_signal
I don't believe this method is currently working as intended, and I'm
not sure it ever has since the move to asyncio. I think this is
responsible for occasional test failures in CI.
Ben Darnell [Mon, 1 May 2023 21:10:27 +0000 (17:10 -0400)]
test: Streamline test configurations
- LANG tests were no longer having the intended effect because C locales
now default to utf8 instead of ascii. There's a new warning we can turn
on with an env var instead. (after cleaing up some tests)
- The tox install_command issue was reverted in tox 1.9
- Python now guarantees that __file__ is absolute
- Remove some obsolete warning manipulations
Ben Darnell [Sun, 23 Apr 2023 19:15:05 +0000 (15:15 -0400)]
test: Close a websocket client that causes occasional test failures
These failures occur on the build.yml workflow on the emulated arm64
platform: an ill-timed timer firing during test shutdown can result
in a message being logged and the test failing for dirty logs.
Ben Darnell [Sun, 9 Apr 2023 21:31:38 +0000 (21:31 +0000)]
ci: Update build workflow
Build wheels for Python 3.12 as well.
Update various dependencies. The upload/download artifact actions
were using deprecated versions, and we were using a deprecated
macos build image. While we're at it, update the other OS versions
and cibuildwheel.
Ben Darnell [Sat, 8 Apr 2023 19:42:05 +0000 (19:42 +0000)]
typing: Eagerly import all submodules in __init__.pyi
This makes the auto-import functionality compatible with mypy
and other typing-based tools such as autocomplete functionality.
Excluding these imports from static typing feels like a premature
optimization and made it much less appealing to make use of the
auto-imports.
This may slow down type checking of applications that use Tornado by
a little, since the type checker must now process all of Tornado and
not only the subset that was imported. However, the increasing use
of long-lived daemons for type checkers should mitigate this cost.
projects / thirdparty / tornado.git / log
