From: Nick Coghlan Date: Tue, 8 Nov 2011 12:11:21 +0000 (+1000) Subject: Issue #13237: Forward port from 3.2 of subprocess documentation updates. Needed quite... X-Git-Tag: v3.3.0a1~908 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=217f05b3edbcab4dfcf5ce79cbc963ac1bd6c2a1;p=thirdparty%2FPython%2Fcpython.git Issue #13237: Forward port from 3.2 of subprocess documentation updates. Needed quite a few adjustments to account for new features coming in 3.3 --- 217f05b3edbcab4dfcf5ce79cbc963ac1bd6c2a1 diff --cc Doc/library/subprocess.rst index 2c7613027bd0,19a29cb56193..fa91c0f283e0 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@@ -25,7 -25,227 +25,260 @@@ modules and functions can be found in t Using the subprocess Module --------------------------- - This module defines one class called :class:`Popen`: + The recommended approach to invoking subprocesses is to use the following + convenience functions for all use cases they can handle. For more advanced + use cases, the underlying :class:`Popen` interface can be used directly. + + -.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False) ++.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + + Run the command described by *args*. Wait for command to complete, then + return the :attr:`returncode` attribute. + + The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the slightly odd notation in - the abbreviated signature). The full function signature is the same as - that of the :class:`Popen` constructor - this functions passes all - supplied arguments directly through to that interface. ++ in :ref:`frequently-used-arguments` (hence the use of keyword-only notation ++ in the abbreviated signature). The full function signature is largely the ++ same as that of the :class:`Popen` constructor - this function passes all ++ supplied arguments other than *timeout* directly through to that interface. ++ ++ The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout ++ expires, the child process will be killed and then waited for again. The ++ :exc:`TimeoutExpired` exception will be re-raised after the child process ++ has terminated. + + Examples:: + + >>> subprocess.call(["ls", "-l"]) + 0 + + >>> subprocess.call("exit 1", shell=True) + 1 + + .. warning:: + + Invoking the system shell with ``shell=True`` can be a security hazard + if combined with untrusted input. See the warning under + :ref:`frequently-used-arguments` for details. + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As + the pipes are not being read in the current process, the child + process may block if it generates enough output to a pipe to fill up + the OS pipe buffer. + ++ .. versionchanged:: 3.3 ++ *timeout* was added. ++ + -.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False) ++.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + + Run command with arguments. Wait for command to complete. If the return + code was zero then return, otherwise raise :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`returncode` attribute. + + The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the slightly odd notation in - the abbreviated signature). The full function signature is the same as - that of the :class:`Popen` constructor - this functions passes all - supplied arguments directly through to that interface. ++ in :ref:`frequently-used-arguments` (hence the use of keyword-only notation ++ in the abbreviated signature). The full function signature is largely the ++ same as that of the :class:`Popen` constructor - this function passes all ++ supplied arguments other than *timeout* directly through to that interface. ++ ++ The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout ++ expires, the child process will be killed and then waited for again. The ++ :exc:`TimeoutExpired` exception will be re-raised after the child process ++ has terminated. + + Examples:: + + >>> subprocess.check_call(["ls", "-l"]) + 0 + + >>> subprocess.check_call("exit 1", shell=True) + Traceback (most recent call last): + ... + subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 + - .. versionadded:: 2.5 - + .. warning:: + + Invoking the system shell with ``shell=True`` can be a security hazard + if combined with untrusted input. See the warning under + :ref:`frequently-used-arguments` for details. + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As + the pipes are not being read in the current process, the child + process may block if it generates enough output to a pipe to fill up + the OS pipe buffer. + ++ .. versionchanged:: 3.3 ++ *timeout* was added. ++ + -.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False) ++.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) + + Run command with arguments and return its output as a byte string. + + If the return code was non-zero it raises a :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`returncode` attribute and any output in the :attr:`output` + attribute. + + The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the slightly odd notation in - the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor, except that *stdout* is - not permitted as it is used internally. All other supplied arguments are - passed directly through to the :class:`Popen` constructor. ++ in :ref:`frequently-used-arguments` (hence the use of keyword-only notation ++ in the abbreviated signature). The full function signature is largely the ++ same as that of the :class:`Popen` constructor - this functions passes all ++ supplied arguments other than *timeout* directly through to that interface. ++ In addition, *stdout* is not permitted as an argument, as it is used ++ internally to collect the output from the subprocess. ++ ++ The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout ++ expires, the child process will be killed and then waited for again. The ++ :exc:`TimeoutExpired` exception will be re-raised after the child process ++ has terminated. + + Examples:: + + >>> subprocess.check_output(["echo", "Hello World!"]) + b'Hello World!\n' + + >>> subprocess.check_output(["echo", "Hello World!"], universal_newlines=True) + 'Hello World!\n' + + >>> subprocess.check_output("exit 1", shell=True) + Traceback (most recent call last): + ... + subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 + + By default, this function will return the data as encoded bytes. The actual + encoding of the output data may depend on the command being invoked, so the + decoding to text will often need to be handled at the application level. + + This behaviour may be overridden by setting *universal_newlines* to + :const:`True` as described below in :ref:`frequently-used-arguments`. + + To also capture standard error in the result, use + ``stderr=subprocess.STDOUT``:: + + >>> subprocess.check_output( + ... "ls non_existent_file; exit 0", + ... stderr=subprocess.STDOUT, + ... shell=True) + 'ls: non_existent_file: No such file or directory\n' + - .. versionadded:: 2.7 ++ .. versionadded:: 3.1 + + .. warning:: + + Invoking the system shell with ``shell=True`` can be a security hazard + if combined with untrusted input. See the warning under + :ref:`frequently-used-arguments` for details. + + .. note:: + + Do not use ``stderr=PIPE`` with this function. As the pipe is not being + read in the current process, the child process may block if it + generates enough output to the pipe to fill up the OS pipe buffer. + ++ .. versionchanged:: 3.3 ++ *timeout* was added. ++ ++ ++.. data:: DEVNULL ++ ++ Special value that can be used as the *stdin*, *stdout* or *stderr* argument ++ to :class:`Popen` and indicates that the special file :data:`os.devnull` ++ will be used. ++ ++ .. versionadded:: 3.3 ++ + + .. data:: PIPE + + Special value that can be used as the *stdin*, *stdout* or *stderr* argument + to :class:`Popen` and indicates that a pipe to the standard stream should be + opened. + + + .. data:: STDOUT + + Special value that can be used as the *stderr* argument to :class:`Popen` and + indicates that standard error should go into the same handle as standard + output. + + + .. _frequently-used-arguments: + + Frequently Used Arguments + ^^^^^^^^^^^^^^^^^^^^^^^^^ + + To support a wide variety of use cases, the :class:`Popen` constructor (and + the convenience functions) accept a large number of optional arguments. For + most typical use cases, many of these arguments can be safely left at their + default values. The arguments that are most commonly needed are: + + *args* is required for all calls and should be a string, or a sequence of + program arguments. Providing a sequence of arguments is generally + preferred, as it allows the module to take care of any required escaping + and quoting of arguments (e.g. to permit spaces in file names). If passing + a single string, either *shell* must be :const:`True` (see below) or else + the string must simply name the program to be executed without specifying + any arguments. + + *stdin*, *stdout* and *stderr* specify the executed program's standard input, + standard output and standard error file handles, respectively. Valid values - are :data:`PIPE`, an existing file descriptor (a positive integer), an - existing file object, and ``None``. :data:`PIPE` indicates that a new pipe - to the child should be created. With the default settings of ``None``, no - redirection will occur; the child's file handles will be inherited from the - parent. Additionally, *stderr* can be :data:`STDOUT`, which indicates that - the stderr data from the child process should be captured into the same file - handle as for stdout. ++ are :data:`PIPE`, :data:`DEVNULL`, an existing file descriptor (a positive ++ integer), an existing file object, and ``None``. :data:`PIPE` indicates ++ that a new pipe to the child should be created. :data:`DEVNULL` indicates ++ that the special file :data:`os.devnull` will be used. With the default ++ settings of ``None``, no redirection will occur; the child's file handles ++ will be inherited from the parent. Additionally, *stderr* can be ++ :data:`STDOUT`, which indicates that the stderr data from the child ++ process should be captured into the same file handle as for *stdout*. + + When *stdout* or *stderr* are pipes and *universal_newlines* is + :const:`True` then the output data is assumed to be encoded as UTF-8 and + will automatically be decoded to text. All line endings will be converted + to ``'\n'`` as described for the universal newlines `'U'`` mode argument + to :func:`open`. + + If *shell* is :const:`True`, the specified command will be executed through + the shell. This can be useful if you are using Python primarily for the + enhanced control flow it offers over most system shells and still want + access to other shell features such as filename wildcards, shell pipes and + environment variable expansion. + + .. warning:: + + Executing shell commands that incorporate unsanitized input from an + untrusted source makes a program vulnerable to `shell injection + `_, + a serious security flaw which can result in arbitrary command execution. + For this reason, the use of *shell=True* is **strongly discouraged** in cases + where the command string is constructed from external input:: + + >>> from subprocess import call + >>> filename = input("What file would you like to display?\n") + What file would you like to display? + non_existent; rm -rf / # + >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly... + + ``shell=False`` disables all shell based features, but does not suffer + from this vulnerability; see the Note in the :class:`Popen` constructor + documentation for helpful hints in getting ``shell=False`` to work. + + These options, along with all of the other options, are described in more + detail in the :class:`Popen` constructor documentation. + + + Popen Constuctor + ^^^^^^^^^^^^^^^^ + + The underlying process creation and management in this module is handled by + the :class:`Popen` class. It offers a lot of flexibility so that developers + are able to handle the less common cases not covered by the convenience + functions. .. class:: Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0, restore_signals=True, start_new_session=False, pass_fds=()) @@@ -122,16 -329,15 +362,16 @@@ You don't need ``shell=True`` to run a batch file, nor to run a console-based executable. - *stdin*, *stdout* and *stderr* specify the executed programs' standard input, + *stdin*, *stdout* and *stderr* specify the executed program's standard input, standard output and standard error file handles, respectively. Valid values - are :data:`PIPE`, an existing file descriptor (a positive integer), an - existing :term:`file object`, and ``None``. :data:`PIPE` indicates that a - new pipe to the child should be created. With the default settings of - ``None``, no redirection will occur; the child's file handles will be - inherited from the parent. Additionally, *stderr* can be :data:`STDOUT`, - which indicates that the stderr data from the applications should be - captured into the same file handle as for stdout. + are :data:`PIPE`, :data:`DEVNULL`, an existing file descriptor (a positive + integer), an existing :term:`file object`, and ``None``. :data:`PIPE` + indicates that a new pipe to the child should be created. :data:`DEVNULL` - indicates that the special file :data:`os.devnull` will be used. With ``None``, - no redirection will occur; the child's file handles will be inherited from - the parent. Additionally, *stderr* can be :data:`STDOUT`, which indicates - that the stderr data from the applications should be captured into the same - file handle as for stdout. ++ indicates that the special file :data:`os.devnull` will be used. With the ++ default settings of ``None``, no redirection will occur; the child's file ++ handles will be inherited from the parent. Additionally, *stderr* can be ++ :data:`STDOUT`, which indicates that the stderr data from the applications ++ should be captured into the same file handle as for stdout. If *preexec_fn* is set to a callable object, this object will be called in the child process just before the child is executed. @@@ -391,18 -452,10 +486,19 @@@ when trying to execute a non-existent f A :exc:`ValueError` will be raised if :class:`Popen` is called with invalid arguments. - check_call() will raise :exc:`CalledProcessError`, if the called process returns - a non-zero return code. + :func:`check_call` and :func:`check_output` will raise + :exc:`CalledProcessError` if the called process returns a non-zero return + code. +All of the functions and methods that accept a *timeout* parameter, such as +:func:`call` and :meth:`Popen.communicate` will raise :exc:`TimeoutExpired` if +the timeout expires before the process exits. + +Exceptions defined in this module all inherit from :exc:`SubprocessError`. + + .. versionadded:: 3.3 + The :exc:`SubprocessError` base class was added. + Security ^^^^^^^^ diff --cc Misc/NEWS index af4775a7dbdc,0d5f98116d84..1a226a53b183 --- a/Misc/NEWS +++ b/Misc/NEWS @@@ -1780,8 -1169,6 +1780,11 @@@ C-AP Documentation ------------- ++- Issue #13237: Reorganise subprocess documentation to emphasise convenience ++ functions and the most commonly needed arguments to Popen. ++ +- Issue #13141: Demonstrate recommended style for socketserver examples. + - Issue #11818: Fix tempfile examples for Python 3.