[3.14] gh-135676: Reword the Operators & Delimiters section(s) (GH-137713) (#138457)

Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Blaise Pabon <blaise@gmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst
index e320eedfa67a27db9d58b424419e4c3881d79478..83db7646f1673f0a407d00a3040715c63fdf309e 100644 (file)
--- a/Doc/reference/lexical_analysis.rst
+++ b/Doc/reference/lexical_analysis.rst
@@ -1351,67 +1351,62 @@ Formally, imaginary literals are described by the following lexical definition:
    imagnumber: (`floatnumber` | `digitpart`) ("j" | "J")
 
 
-.. _operators:
-
-Operators
-=========
-
-.. index:: single: operators
-
-The following tokens are operators:
-
-.. code-block:: none
-
-
-   +       -       *       **      /       //      %      @
-   <<      >>      &       |       ^       ~       :=
-   <       >       <=      >=      ==      !=
-
-
 .. _delimiters:
-
-Delimiters
-==========
-
-.. index:: single: delimiters
-
-The following tokens serve as delimiters in the grammar:
-
-.. code-block:: none
-
-   (       )       [       ]       {       }
-   ,       :       !       .       ;       @       =
-
-The period can also occur in floating-point and imaginary literals.
-
+.. _operators:
 .. _lexical-ellipsis:
 
-A sequence of three periods has a special meaning as an
-:py:data:`Ellipsis` literal:
-
-.. code-block:: none
-
-   ...
-
-The following *augmented assignment operators* serve
-lexically as delimiters, but also perform an operation:
+Operators and delimiters
+========================
 
-.. code-block:: none
-
-   ->      +=      -=      *=      /=      //=     %=
-   @=      &=      |=      ^=      >>=     <<=     **=
-
-The following printing ASCII characters have special meaning as part of other
-tokens or are otherwise significant to the lexical analyzer:
-
-.. code-block:: none
-
-   '       "       #       \
+.. index::
+   single: operators
+   single: delimiters
 
-The following printing ASCII characters are not used in Python.  Their
-occurrence outside string literals and comments is an unconditional error:
+The following grammar defines :dfn:`operator` and :dfn:`delimiter` tokens,
+that is, the generic :data:`~token.OP` token type.
+A :ref:`list of these tokens and their names <token_operators_delimiters>`
+is also available in the :mod:`!token` module documentation.
 
-.. code-block:: none
+.. grammar-snippet::
+   :group: python-grammar
 
-   $       ?       `
+   OP:
+      | assignment_operator
+      | bitwise_operator
+      | comparison_operator
+      | enclosing_delimiter
+      | other_delimiter
+      | arithmetic_operator
+      | "..."
+      | other_op
+
+   assignment_operator:   "+=" | "-=" | "*=" | "**=" | "/="  | "//=" | "%=" |
+                          "&=" | "|=" | "^=" | "<<=" | ">>=" | "@="  | ":="
+   bitwise_operator:      "&"  | "|"  | "^"  | "~"   | "<<"  | ">>"
+   comparison_operator:   "<=" | ">=" | "<"  | ">"   | "=="  | "!="
+   enclosing_delimiter:   "("  | ")"  | "["  | "]"   | "{"   | "}"
+   other_delimiter:       ","  | ":"  | "!"  | ";"   | "="   | "->"
+   arithmetic_operator:   "+"  | "-"  | "**" | "*"   | "//"  | "/"   | "%"
+   other_op:              "."  | "@"
+
+.. note::
+
+   Generally, *operators* are used to combine :ref:`expressions <expressions>`,
+   while *delimiters* serve other purposes.
+   However, there is no clear, formal distinction between the two categories.
+
+   Some tokens can serve as either operators or delimiters, depending on usage.
+   For example, ``*`` is both the multiplication operator and a delimiter used
+   for sequence unpacking, and ``@`` is both the matrix multiplication and
+   a delimiter that introduces decorators.
+
+   For some tokens, the distinction is unclear.
+   For example, some people consider ``.``, ``(``, and ``)`` to be delimiters, while others
+   see the :py:func:`getattr` operator and the function call operator(s).
+
+   Some of Python's operators, like ``and``, ``or``, and ``not in``, use
+   :ref:`keyword <keywords>` tokens rather than "symbols" (operator tokens).
+
+A sequence of three consecutive periods (``...``) has a special
+meaning as an :py:data:`Ellipsis` literal.