From: Ben Darnell Date: Sun, 4 Mar 2018 22:50:58 +0000 (-0500) Subject: docs: Updates for 5.0 X-Git-Tag: v5.0.0~1^2 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=a596a0d73331a9e582149af7668212852c0870ac;p=thirdparty%2Ftornado.git docs: Updates for 5.0 --- diff --git a/docs/coroutine.rst b/docs/coroutine.rst index 4db710095..144406ac3 100644 --- a/docs/coroutine.rst +++ b/docs/coroutine.rst @@ -4,7 +4,6 @@ Coroutines and concurrency .. toctree:: gen - concurrent locks queues process diff --git a/docs/gen.rst b/docs/gen.rst index cf42269fb..1f1b2c1d7 100644 --- a/docs/gen.rst +++ b/docs/gen.rst @@ -1,4 +1,4 @@ -``tornado.gen`` --- Simplify asynchronous code +``tornado.gen`` --- Generator-based coroutines ============================================== .. testsetup:: diff --git a/docs/guide/coroutines.rst b/docs/guide/coroutines.rst index 83e3048fa..8089c4f88 100644 --- a/docs/guide/coroutines.rst +++ b/docs/guide/coroutines.rst @@ -65,22 +65,6 @@ will work with ``await``:: executor = concurrent.futures.ThreadPoolExecutor() await tornado.gen.convert_yielded(executor.submit(g)) -While native coroutines are not visibly tied to a particular framework -(i.e. they do not use a decorator like `tornado.gen.coroutine` or -`asyncio.coroutine`), not all coroutines are compatible with each -other. There is a *coroutine runner* which is selected by the first -coroutine to be called, and then shared by all coroutines which are -called directly with ``await``. The Tornado coroutine runner is -designed to be versatile and accept awaitable objects from any -framework; other coroutine runners may be more limited (for example, -the ``asyncio`` coroutine runner does not accept coroutines from other -frameworks). For this reason, it is recommended to use the Tornado -coroutine runner for any application which combines multiple -frameworks. To call a coroutine using the Tornado runner from within a -coroutine that is already using the asyncio runner, use the -`tornado.platform.asyncio.to_asyncio_future` adapter. - - How it works ~~~~~~~~~~~~ @@ -165,42 +149,21 @@ used to start the ``main`` function of a batch-oriented program:: Coroutine patterns ~~~~~~~~~~~~~~~~~~ -Interaction with callbacks -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To interact with asynchronous code that uses callbacks instead of -`.Future`, wrap the call in a `.Task`. This will add the callback -argument for you and return a `.Future` which you can yield: - -.. testcode:: - - @gen.coroutine - def call_task(): - # Note that there are no parens on some_function. - # This will be translated by Task into - # some_function(other_args, callback=callback) - yield gen.Task(some_function, other_args) - -.. testoutput:: - :hide: - Calling blocking functions ^^^^^^^^^^^^^^^^^^^^^^^^^^ The simplest way to call a blocking function from a coroutine is to -use a `~concurrent.futures.ThreadPoolExecutor`, which returns +use `.IOLoop.run_in_executor`, which returns ``Futures`` that are compatible with coroutines:: - thread_pool = ThreadPoolExecutor(4) - @gen.coroutine def call_blocking(): - yield thread_pool.submit(blocking_func, args) + yield IOLoop.current().run_in_executor(blocking_func, args) Parallelism ^^^^^^^^^^^ -The coroutine decorator recognizes lists and dicts whose values are +The `.coroutine` decorator recognizes lists and dicts whose values are ``Futures``, and waits for all of those ``Futures`` in parallel: .. testcode:: @@ -224,6 +187,13 @@ The coroutine decorator recognizes lists and dicts whose values are .. testoutput:: :hide: +Lists and dicts must be wrapped in `tornado.gen.multi` for use with +``await``:: + + async def parallel_fetch(url1, url2): + resp1, resp2 = await gen.multi([http_client.fetch(url1), + http_client.fetch(url2)]) + Interleaving ^^^^^^^^^^^^ @@ -254,11 +224,12 @@ background processing. Looping ^^^^^^^ -Looping is tricky with coroutines since there is no way in Python -to ``yield`` on every iteration of a ``for`` or ``while`` loop and -capture the result of the yield. Instead, you'll need to separate -the loop condition from accessing the results, as in this example -from `Motor `_:: +In native coroutines, ``async for`` can be used. In older versions of +Python, looping is tricky with coroutines since there is no way to +``yield`` on every iteration of a ``for`` or ``while`` loop and +capture the result of the yield. Instead, you'll need to separate the +loop condition from accessing the results, as in this example from +`Motor `_:: import motor db = motor.MotorClient().test diff --git a/docs/guide/structure.rst b/docs/guide/structure.rst index e9f9506c0..0f20d3386 100644 --- a/docs/guide/structure.rst +++ b/docs/guide/structure.rst @@ -305,9 +305,10 @@ non-blocking way. This topic is covered in more detail in asynchronous techniques in `.RequestHandler` subclasses. The simplest way to make a handler asynchronous is to use the -`.coroutine` decorator. This allows you to perform non-blocking I/O -with the ``yield`` keyword, and no response will be sent until the -coroutine has returned. See :doc:`coroutines` for more details. +`.coroutine` decorator or ``async def``. This allows you to perform +non-blocking I/O with the ``yield`` or ``await`` keywords, and no +response will be sent until the coroutine has returned. See +:doc:`coroutines` for more details. In some cases, coroutines may be less convenient than a callback-oriented style, in which case the `.tornado.web.asynchronous` diff --git a/docs/locks.rst b/docs/locks.rst index 94184db2a..9f991880c 100644 --- a/docs/locks.rst +++ b/docs/locks.rst @@ -3,8 +3,10 @@ .. versionadded:: 4.2 -Coordinate coroutines with synchronization primitives analogous to those the -standard library provides to threads. +Coordinate coroutines with synchronization primitives analogous to +those the standard library provides to threads. These classes are very +similar to those provided in the standard library's `asyncio package +`_. .. warning:: diff --git a/docs/utilities.rst b/docs/utilities.rst index 1263944cc..55536626e 100644 --- a/docs/utilities.rst +++ b/docs/utilities.rst @@ -4,6 +4,7 @@ Utilities .. toctree:: autoreload + concurrent log options stack_context diff --git a/tornado/concurrent.py b/tornado/concurrent.py index 65da0ac83..1a79f1e10 100644 --- a/tornado/concurrent.py +++ b/tornado/concurrent.py @@ -12,13 +12,19 @@ # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. -"""Utilities for working with threads and ``Futures``. +"""Utilities for working with ``Future`` objects. ``Futures`` are a pattern for concurrent programming introduced in -Python 3.2 in the `concurrent.futures` package. This package defines -a mostly-compatible `Future` class designed for use from coroutines, -as well as some utility functions for interacting with the -`concurrent.futures` package. +Python 3.2 in the `concurrent.futures` package, and also adopted (in a +slightly different form) in Python 3.4's `asyncio` package. This +package defines a ``Future`` class that is an alias for `asyncio.Future` +when available, and a compatible implementation for older versions of +Python. It also includes some utility functions for interacting with +``Future`` objects. + +While this package is an important part of Tornado's internal +implementation, applications rarely need to interact with it +directly. """ from __future__ import absolute_import, division, print_function @@ -404,7 +410,11 @@ def run_on_executor(*args, **kwargs): pass This decorator should not be confused with the similarly-named - `.IOLoop.run_in_executor`. + `.IOLoop.run_in_executor`. In general, using ``run_in_executor`` + when *calling* a blocking method is recommended instead of using + this decorator when *defining* a method. If compatibility with older + versions of Tornado is required, consider defining an executor + and using ``executor.submit()`` at the call site. .. versionchanged:: 4.2 Added keyword arguments to use alternative attributes. @@ -441,6 +451,10 @@ def return_future(f): """Decorator to make a function that returns via callback return a `Future`. + This decorator was provided to ease the transition from + callback-oriented code to coroutines. It is not recommended for + new code. + The wrapped function should take a ``callback`` keyword argument and invoke it with one argument when it has finished. To signal failure, the function can simply raise an exception (which will be @@ -475,6 +489,7 @@ def return_future(f): Note that ``@return_future`` and ``@gen.engine`` can be applied to the same function, provided ``@return_future`` appears first. However, consider using ``@gen.coroutine`` instead of this combination. + """ replacer = ArgReplacer(f, 'callback') diff --git a/tornado/gen.py b/tornado/gen.py index 0ceda1d56..762b44f9c 100644 --- a/tornado/gen.py +++ b/tornado/gen.py @@ -1,6 +1,20 @@ -"""``tornado.gen`` is a generator-based interface to make it easier to -work in an asynchronous environment. Code using the ``gen`` module -is technically asynchronous, but it is written as a single generator +"""``tornado.gen`` implements generator-based coroutines. + +.. note:: + + The "decorator and generator" approach in this module is a + precursor to native coroutines (using ``async def`` and ``await``) + which were introduced in Python 3.5. Applications that do not + require compatibility with older versions of Python should use + native coroutines instead. Some parts of this module are still + useful with native coroutines, notably `multi`, `sleep`, + `WaitIterator`, and `with_timeout`. Some of these functions have + counterparts in the `asyncio` module which may be used as well, + although the two may not necessarily be 100% compatible. + +Coroutines provide an easier way to work in an asynchronous +environment than chaining callbacks. Code using coroutines is +technically asynchronous, but it is written as a single generator instead of a collection of separate functions. For example, the following asynchronous handler: @@ -706,6 +720,10 @@ def multi(children, quiet_exceptions=()): This function is available under the names ``multi()`` and ``Multi()`` for historical reasons. + Cancelling a `.Future` returned by ``multi()`` does not cancel its + children. `asyncio.gather` is similar to ``multi()``, but it does + cancel its children. + .. versionchanged:: 4.2 If multiple yieldables fail, any exceptions after the first (which is raised) will be logged. Added the ``quiet_exceptions`` @@ -886,6 +904,10 @@ def with_timeout(timeout, future, quiet_exceptions=()): Does not support `YieldPoint` subclasses. + The wrapped `.Future` is not canceled when the timeout expires, + permitting it to be reused. `asyncio.wait_for` is similar to this + function but it does cancel the wrapped `.Future` on timeout. + .. versionadded:: 4.0 .. versionchanged:: 4.1 @@ -894,6 +916,7 @@ def with_timeout(timeout, future, quiet_exceptions=()): .. versionchanged:: 4.4 Added support for yieldable objects other than `.Future`. + """ # TODO: allow YieldPoints in addition to other yieldables? # Tricky to do with stack_context semantics. diff --git a/tornado/queues.py b/tornado/queues.py index e792e0c04..23b8bb9ca 100644 --- a/tornado/queues.py +++ b/tornado/queues.py @@ -12,7 +12,9 @@ # License for the specific language governing permissions and limitations # under the License. -"""Asynchronous queues for coroutines. +"""Asynchronous queues for coroutines. These classes are very similar +to those provided in the standard library's `asyncio package +`_. .. warning:: @@ -20,6 +22,7 @@ are *not* thread-safe. To use these queues from another thread, use `.IOLoop.add_callback` to transfer control to the `.IOLoop` thread before calling any queue methods. + """ from __future__ import absolute_import, division, print_function diff --git a/tornado/web.py b/tornado/web.py index 8abe4bc2a..2fce30960 100644 --- a/tornado/web.py +++ b/tornado/web.py @@ -46,12 +46,14 @@ Thread-safety notes ------------------- In general, methods on `RequestHandler` and elsewhere in Tornado are -not thread-safe. In particular, methods such as +not thread-safe. In particular, methods such as `~RequestHandler.write()`, `~RequestHandler.finish()`, and -`~RequestHandler.flush()` must only be called from the main thread. If +`~RequestHandler.flush()` must only be called from the main thread. If you use multiple threads it is important to use `.IOLoop.add_callback` to transfer control back to the main thread before finishing the -request. +request, or to limit your use of other threads to +`.IOLoop.run_in_executor` and ensure that your callbacks running in +the executor do not refer to Tornado objects. """