Peter Maydell [Mon, 21 Jun 2021 15:11:33 +0000 (16:11 +0100)]
Merge remote-tracking branch 'remotes/jsnow-gitlab/tags/python-pull-request' into staging
Python Pull request
Moves QMP-related tools not used for build or automatic testing from
scripts/ to python/qemu/qmp/ where they will be protected from bitrot by
the check-python-* CI jobs.
stub forwarders are left in the old locations for now.
# gpg: Signature made Sat 19 Jun 2021 00:02:40 BST
# gpg: using RSA key F9B7ABDBBCACDF95BE76CBD07DEF8106AAFC390E
# gpg: Good signature from "John Snow (John Huston) <jsnow@redhat.com>" [full]
# Primary key fingerprint: FAEB 9711 A12C F475 812F 18F2 88A9 064D 1835 61EB
# Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76 CBD0 7DEF 8106 AAFC 390E
* remotes/jsnow-gitlab/tags/python-pull-request: (72 commits)
scripts/qmp-shell: add redirection shim
python: add qmp-shell entry point
scripts/qmp-shell: move to python/qemu/qmp/qmp_shell.py
scripts/qmp-shell: add docstrings
scripts/qmp-shell: make QMPShellError inherit QMPError
scripts/qmp-shell: remove double-underscores
scripts/qmp-shell: convert usage comment to docstring
scripts/qmp-shell: Remove too-broad-exception
scripts/qmp-shell: Fix empty-transaction invocation
scripts/qmp-shell: remove TODO
scripts/qmp-shell: use logging to show warnings
scripts/qmp-shell: Use context manager instead of atexit
python/qmp: return generic type from context manager
scripts/qmp-shell: unprivatize 'pretty' property
scripts/qmp-shell: Accept SocketAddrT instead of string
scripts/qmp-shell: add mypy types
python/qmp: add QMPObject type alias
scripts/qmp-shell: initialize completer early
scripts/qmp-shell: refactor QMPCompleter
scripts/qmp-shell: Fix "FuzzyJSON" parser
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Peter Maydell [Mon, 21 Jun 2021 10:26:04 +0000 (11:26 +0100)]
Merge remote-tracking branch 'remotes/ehabkost-gl/tags/x86-next-pull-request' into staging
x86 queue, 2021-06-18
Features:
* Add ratelimit for bus locks acquired in guest (Chenyi Qiang)
Documentation:
* SEV documentation updates (Tom Lendacky)
* Add a table showing x86-64 ABI compatibility levels (Daniel P. Berrangé)
Automated changes:
* Update Linux headers to 5.13-rc4 (Eduardo Habkost)
# gpg: Signature made Fri 18 Jun 2021 20:51:26 BST
# gpg: using RSA key 5A322FD5ABC4D3DBACCFD1AA2807936F984DC5A6
# gpg: issuer "ehabkost@redhat.com"
# gpg: Good signature from "Eduardo Habkost <ehabkost@redhat.com>" [full]
# Primary key fingerprint: 5A32 2FD5 ABC4 D3DB ACCF D1AA 2807 936F 984D C5A6
* remotes/ehabkost-gl/tags/x86-next-pull-request:
scripts: helper to generate x86_64 CPU ABI compat info
docs: add a table showing x86-64 ABI compatibility levels
docs/interop/firmware.json: Add SEV-ES support
docs: Add SEV-ES documentation to amd-memory-encryption.txt
doc: Fix some mistakes in the SEV documentation
i386: Add ratelimit for bus locks acquired in guest
Update Linux headers to 5.13-rc4
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Peter Maydell [Sun, 20 Jun 2021 20:20:13 +0000 (21:20 +0100)]
Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-06-15-v2' into staging
nbd patches for 2021-06-15
- bug fixes in coroutine aio context handling
- rework NBD client connection logic to perform more work in coroutine
# gpg: Signature made Fri 18 Jun 2021 18:29:39 BST
# gpg: using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg: aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg: aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2 F3AA A7A1 6B4A 2527 436A
* remotes/ericb/tags/pull-nbd-2021-06-15-v2: (34 commits)
block/nbd: safer transition to receiving request
block/nbd: add nbd_client_connected() helper
block/nbd: reuse nbd_co_do_establish_connection() in nbd_open()
nbd/client-connection: add option for non-blocking connection attempt
block/nbd: split nbd_co_do_establish_connection out of nbd_reconnect_attempt
block-coroutine-wrapper: allow non bdrv_ prefix
nbd/client-connection: return only one io channel
block/nbd: drop BDRVNBDState::sioc
block/nbd: don't touch s->sioc in nbd_teardown_connection()
block/nbd: use negotiation of NBDClientConnection
block/nbd: split nbd_handle_updated_info out of nbd_client_handshake()
nbd/client-connection: shutdown connection on release
nbd/client-connection: implement connection retry
nbd/client-connection: add possibility of negotiation
nbd/client-connection: use QEMU_LOCK_GUARD
nbd: move connection code from block/nbd to nbd/client-connection
block/nbd: introduce nbd_client_connection_release()
block/nbd: introduce nbd_client_connection_new()
block/nbd: rename NBDConnectThread to NBDClientConnection
block/nbd: make nbd_co_establish_connection_cancel() bs-independent
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
John Snow [Mon, 7 Jun 2021 20:06:47 +0000 (16:06 -0400)]
scripts/qmp-shell: move to python/qemu/qmp/qmp_shell.py
The script will be unavailable for a commit or two, which will help
preserve development history attached to the new file. A forwarder will
be added shortly afterwards.
With qmp_shell in the python qemu.qmp package, now it is fully type
checked, linted, etc. via the Python CI. It will be quite a bit harder
to accidentally break it again in the future.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-41-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Mon, 7 Jun 2021 20:06:45 +0000 (16:06 -0400)]
scripts/qmp-shell: make QMPShellError inherit QMPError
In preparation for moving qmp-shell into the qemu.qmp package, make
QMPShellError inherit from QMPError so that all custom errors in this
package all derive from QMPError.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-39-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Mon, 7 Jun 2021 20:06:44 +0000 (16:06 -0400)]
scripts/qmp-shell: remove double-underscores
They're not needed; single underscore is enough to express intent that
these methods are "internal". double underscore is used as a weak name
mangling, but that isn't beneficial for us here.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-38-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
calling "transaction( )" is pointless, but valid. Rework the parser to
allow this kind of invocation. This helps clean up exception handling
later by removing accidental breakages of the parser that aren't
explicitly forbidden.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-35-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Mon, 7 Jun 2021 20:06:38 +0000 (16:06 -0400)]
scripts/qmp-shell: Use context manager instead of atexit
We can invoke the shell history writing when we leave the QMPShell scope
instead of relying on atexit. Doing so may be preferable to avoid global
state being registered from within a class instead of from the
application logic directly.
Use QMP's context manager to hook this history saving at close time,
which gets invoked when we leave the context block.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-32-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Mon, 7 Jun 2021 20:06:34 +0000 (16:06 -0400)]
scripts/qmp-shell: add mypy types
As per my usual, this patch is annotations only. Any changes with side
effects are done elsewhere.
Note: pylint does not understand the subscripts for Collection in Python 3.6,
so use the stronger Sequence type as a workaround. Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-28-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Mon, 7 Jun 2021 20:06:30 +0000 (16:06 -0400)]
scripts/qmp-shell: Fix "FuzzyJSON" parser
I'm not sure when this regressed (Or maybe if it was ever working right
to begin with?), but the Python AST requires you to change "Names" to
"Constants" in order to truly convert `false` to `False`.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-24-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Mon, 7 Jun 2021 20:06:29 +0000 (16:06 -0400)]
scripts/qmp-shell: move the REPL functionality into QMPShell
Instead of doing this in main, move it into the class itself. (This
makes it easier to put into the qemu.qmp package later by removing as
much as we can from the main() function.)
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-23-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Mon, 7 Jun 2021 20:06:26 +0000 (16:06 -0400)]
scripts/qmp-shell: Make verbose a public attribute
No real reason to hide this behind an underscore; make it part of the
initializer and make it a regular RW attribute. Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-20-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Mon, 7 Jun 2021 20:06:22 +0000 (16:06 -0400)]
scripts/qmp-shell: remove if-raise-else patterns
Shushes pylint. I don't always mind these patterns personally, but I'm
not as sure that I want to remove the warning from pylint's repertoire
entirely. Oh well.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210607200649.1840382-16-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Fri, 4 Jun 2021 15:55:31 +0000 (11:55 -0400)]
python/qemu-ga-client: add entry point
Remove the shebang, and add a package-defined entry point instead. Now,
it can be accessed using 'qemu-ga-client' from the command line after
installing the package.
The next commit adds a forwarder shim that allows the running of this
script without needing to install the package again.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-11-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Fri, 4 Jun 2021 15:55:30 +0000 (11:55 -0400)]
scripts/qemu-ga-client: move to python/qemu/qmp/qemu_ga_client.py
The script itself will be unavailable for a few commits before being
restored, with no way to run it right after this commit. This helps move
git history into the new file. To prevent linter regressions, though, we
do need to immediately touch up the filename to remove dashes (to make
the module importable), and remove the executable bit.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-10-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Fri, 4 Jun 2021 15:55:29 +0000 (11:55 -0400)]
scripts/qemu-ga-client: add mypy type hints
This script is in slightly rough shape, but it still works. A lot of
care went into its initial development. In good faith, I'm updating it
to the latest Python coding standards. If there is in interest in this
script, though, I'll be asking for a contributor to take care of it
further.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-9-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Fri, 4 Jun 2021 15:55:28 +0000 (11:55 -0400)]
python/qmp: Correct type of QMPReturnValue
It's only a Dict[str, Any] most of the time. It's not actually
guaranteed to be anything in particular. Fix this type to be
more accurate to the reality we live in.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-8-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Fri, 4 Jun 2021 15:55:27 +0000 (11:55 -0400)]
scripts/qemu-ga-client: apply (most) pylint rules
I'm only doing a very quick best-effort to preserve this script, to help
keep it from breaking further. I think there are pending ideas swirling
on the right way to implement better SDKs and better clients, and this
script might be a handy reference for those discussions. It presents
some interesting design problems, like static type safety when using a
dynamic RPC mechanism.
I believe it's worth preserving the effort and care that went into
making this script by updating it to work with our current
infrastructure. However, I am disabling the requirement for docstrings
in this file.
If you would like to help improve this script, please add docstrings
alongside any refactors or rejuvenations you might apply at that time.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210604155532.1499282-7-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:19 +0000 (20:37 -0400)]
python/qmp: add fuse command to 'qom' tools
The 'fuse' command will be unavailable if 'fusepy' is not installed. It
will simply not load and subsequently be unavailable as a subcommand.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-20-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:18 +0000 (20:37 -0400)]
scripts/qom-fuse: add redirection shim to python/qemu/qmp/qom-fuse.py
By leaving the script absent for a commit, git-blame travels to the new
file instead of staying on the shim.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-19-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:17 +0000 (20:37 -0400)]
scripts/qom-fuse: move to python/qemu/qmp/qom_fuse.py
Move qom-fuse over to the python package now that it passes the
linter. Update the import paradigms so that it continues to pass in the
context of the Python package.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-18-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:16 +0000 (20:37 -0400)]
python: add optional FUSE dependencies
In preparation for moving qom-fuse over to the python package, we need
some new dependencies to support it.
Add an optional 'fusepy' dependency that users of the package can opt
into with e.g. "pip install qemu[fuse]" which installs the requirements
necessary to obtain the additional functionality.
Add the same fusepy dependency to the 'devel' extras group --
unfortunately I do not see a way for optional groups to imply other
optional groups at present, so the dependency is repeated. The
development group needs to include the full set of dependencies for the
purpose of static analysis of all features offered by this library.
Lastly, add the [fuse] extras group to tox's configuration as a
workaround so that if a stale tox environment is found when running
`make check-tox`, tox will know to rebuild its environments.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-17-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:15 +0000 (20:37 -0400)]
scripts/qom-fuse: add static type hints
Because fusepy does not have type hints, add some targeted warning
suppressions.
Namely, we need to allow subclassing something of an unknown type (in
qom_fuse.py), and we need to allow missing imports (recorded against
fuse itself) because mypy will be unable to import fusepy (even when
installed) as it has no types nor type stubs available.
Note: Until now, it was possible to run invocations like 'mypy qemu/'
from ./python and have that work. However, these targeted suppressions
require that you run 'mypy -p qemu/' instead. The correct, canonical
invocation is recorded in ./python/tests/mypy.sh and all of the various
CI invocations always use this correct form.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-16-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
- Use FuseOSError to signal ENOENT instead of returning it
- Wrap qom-get in str(), as we don't always know its type
- The empty return should be b'', not ''.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-15-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:13 +0000 (20:37 -0400)]
scripts/qom-fuse: use QOMCommand.qom_list()
the qom_list method provides a type-safe object that's easier to type
check, so switch to using it.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-14-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:12 +0000 (20:37 -0400)]
scripts/qom-fuse: Convert to QOMCommand
Move qom-fuse onto the QOMCommand base established in
python/qemu/qmp/qom_common.py. The interface doesn't change
incompatibly, "qom-fuse mountpoint" still works as an invocation, and
QMP_SOCKET is still used as the environment variable.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-13-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:10 +0000 (20:37 -0400)]
scripts/qom-fuse: Apply pylint rules
- Catch specific exceptions from QMP
- Reraise errors with explicit context
- method parameters should match parent's names
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-11-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:09 +0000 (20:37 -0400)]
python: Add 'fh' to known-good variable names
fd and fh are fine: we often use these for "file descriptor" or "file
handle" accordingly. It is rarely the case that you need to enforce a
more semantically meaningful name beyond "This is the file we are using
right now."
While we're here: add comments for all of the non-standard pylint
names. (And the underscore.)
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-10-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:08 +0000 (20:37 -0400)]
scripts/qom-fuse: apply flake8 rules
flake8 still has one warning because of the sys.path hack, but that will
be going away by the end of this patch series.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-9-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:07 +0000 (20:37 -0400)]
scripts/qom-fuse: apply isort rules
Hint: you can use symlinks to create qom_fuse.py in python/qemu/qmp/ and
point to scripts/qom-fuse to apply the standard linting rules to this
script.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-8-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:06 +0000 (20:37 -0400)]
scripts/qmp: redirect qom-xxx scripts to python/qemu/qmp/
Redirect to the new qom scripts. These forwarders can be deleted
eventually when there has been more time for the dust on the Python
packaging to settle and people understand how to find these commands.
Note: You can run these by setting $PYTHONPATH in your shell and then
running "python3 -m qemu.qmp.qom", or you can install the qemu namespace
package and use the "qom" or "qom-set" scripts.
I've written how to install the package elsewhere, but for the sake of
git-blame, cd to ./python, and then do:
- pip3 install [--user] [-e] .
--user will install to your local user install (will not work inside of
a venv), omitting this flag installs to your system-wide packages
(outside of a venv) or to your current virtual environment (inside the
venv).
When installing to a venv or to your system-wide packages, "qom"
should be in your $PATH already. If you do a user install, you may
need to add ~/.local/bin to your $PATH if you haven't already.
-e installs in editable mode: the installed package is effectively just
a symlink to this folder; so changes to your git working tree are
reflected in the installed package.
Note: installing these packages to an environment outside a venv can be
dangerous: Many QEMU scripts will begin to prefer the installed version
instead of the version directly in the tree. Use with caution. editable
mode is recommended when working outside of a venv.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-7-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:05 +0000 (20:37 -0400)]
python/qmp: add qom script entry points
Add the 'qom', 'qom-set', 'qom-get', 'qom-list', and 'qom-tree' scripts
to the qemu.qmp package. When you install this package, these scripts
will become available on your command line.
(e.g. when inside of a venv, `cd python && pip install .` will add
'qom', 'qom-set', etc to your $PATH.)
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-6-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:03 +0000 (20:37 -0400)]
python/qmp: add parse_address classmethod
This takes the place of qmp-shell's __get_address function. It also
allows other utilities to share the same parser and syntax for
specifying QMP locations.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-4-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:02 +0000 (20:37 -0400)]
python/qmp: Fix type of SocketAddrT
In porting the qom tools, qmp-shell, etc; it becomes evident that this
type is wrong.
This is an integer, not a string. We didn't catch this before because
none of QEMUMonitorProtocol's *users* happen to be checked, and the
internal logic of this class is otherwise self-consistent. Additionally,
mypy was not introspecting into the socket() interface to realize we
were passing a bad type for AF_INET. Fixed now.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210603003719.1321369-3-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
John Snow [Thu, 3 Jun 2021 00:37:01 +0000 (20:37 -0400)]
python/pipenv: Update Pipfile.lock
In a previous commit, I added tox to the development requirements of the
Python library. I never bothered to add them to the Pipfile, because
they aren't needed there. Here, I sync it anyway in its own commit so
that when we add new packages later that the diffstats will not
confusingly appear to pull in lots of extra packages.
Ideally I could tell Pipenv simply not to install these, but it doesn't
seem to support that, exactly. The alternative is removing Tox from the
development requires, which I'd rather not do.
The other alternative is re-specifying all of the dependencies of
setup.cfg in the Pipfile, which I'd also rather not do.
Picking what feels least-worst here.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20210603003719.1321369-2-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>
req->receiving is a flag of request being in one concrete yield point
in nbd_co_do_receive_one_chunk().
Such kind of boolean flag is always better to unset before scheduling
the coroutine, to avoid double scheduling. So, let's be more careful.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-33-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
We already have two similar helpers for other state. Let's add another
one for convenience.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-32-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
block/nbd: reuse nbd_co_do_establish_connection() in nbd_open()
The only last step we need to reuse the function is coroutine-wrapper.
nbd_open() may be called from non-coroutine context. So, generate the
wrapper and use it.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-31-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
nbd/client-connection: add option for non-blocking connection attempt
We'll need a possibility of non-blocking nbd_co_establish_connection(),
so that it returns immediately, and it returns success only if a
connections was previously established in background.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-30-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
block/nbd: split nbd_co_do_establish_connection out of nbd_reconnect_attempt
Split out the part that we want to reuse for nbd_open().
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210610100802.5888-29-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Eric Blake <eblake@redhat.com>
We are going to reuse the script to generate a nbd_ function in
further commit. Prepare the script now.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-28-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
block/nbd doesn't need underlying sioc channel anymore. So, we can
update nbd/client-connection interface to return only one top-most io
channel, which is more straight forward.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-27-vsementsov@virtuozzo.com>
[eblake: squash in Vladimir's fixes for uninit usage caught by clang] Signed-off-by: Eric Blake <eblake@redhat.com>
Currently sioc pointer is used just to pass from socket-connection to
nbd negotiation. Drop the field, and use local variables instead. With
next commit we'll update nbd/client-connection.c to behave
appropriately (return only top-most ioc, not two channels).
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-26-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
block/nbd: don't touch s->sioc in nbd_teardown_connection()
Negotiation during reconnect is now done in a thread, and s->sioc is
not available during negotiation. Negotiation in thread will be
cancelled by nbd_client_connection_release() called from
nbd_clear_bdrvstate(). So, we don't need this code chunk anymore.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-25-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
Now that we can opt in to negotiation as part of the client connection
thread, use that to simplify connection_co. This is another step on
the way to moving all reconnect code into NBDClientConnection.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-24-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
block/nbd: split nbd_handle_updated_info out of nbd_client_handshake()
To be reused in the following patch.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Roman Kagan <rvkagan@yandex-team.ru> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-23-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
nbd/client-connection: shutdown connection on release
Now, when a thread can do negotiation and retry, it may run relatively
long. We need a mechanism to stop it, when the user is not interested
in a result any more. So, on nbd_client_connection_release() let's
shutdown the socket, and do not retry connection if thread is detached.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210610100802.5888-22-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Eric Blake <eblake@redhat.com>
Add an option for a thread to retry connecting until it succeeds. We'll
use nbd/client-connection both for reconnect and for initial connection
in nbd_open(), so we need a possibility to use same NBDClientConnection
instance to connect once in nbd_open() and then use retry semantics for
reconnect.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210610100802.5888-21-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: grammar tweak] Signed-off-by: Eric Blake <eblake@redhat.com>
nbd/client-connection: add possibility of negotiation
Add arguments and logic to support nbd negotiation in the same thread
after successful connection.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210610100802.5888-20-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Eric Blake <eblake@redhat.com>
We don't update connect_thread_func() to use QEMU_LOCK_GUARD, as it
will get more complex critical sections logic in further commit, where
QEMU_LOCK_GUARD doesn't help.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210610100802.5888-19-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Eric Blake <eblake@redhat.com>
This is a last step of creating bs-independent nbd connection
interface. With next commit we can finally move it to separate file.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210610100802.5888-17-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Eric Blake <eblake@redhat.com>
This is a step of creating bs-independent nbd connection interface.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Roman Kagan <rvkagan@yandex-team.ru> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-16-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
block/nbd: rename NBDConnectThread to NBDClientConnection
We are going to move the connection code to its own file, and want
clear names and APIs first.
The structure is shared between user and (possibly) several runs of
connect-thread. So it's wrong to call it "thread". Let's rename to
something more generic.
Appropriately rename connect_thread and thr variables to conn.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Roman Kagan <rvkagan@yandex-team.ru> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-15-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
block/nbd: make nbd_co_establish_connection_cancel() bs-independent
nbd_co_establish_connection_cancel() actually needs only pointer to
NBDConnectThread. So, make it clean.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Roman Kagan <rvkagan@yandex-team.ru> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-14-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
block/nbd: bs-independent interface for nbd_co_establish_connection()
We are going to split connection code to a separate file. Now we are
ready to give nbd_co_establish_connection() clean and bs-independent
interface.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Roman Kagan <rvkagan@yandex-team.ru> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-13-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
We don't need all these states. The code refactored to use two boolean
variables looks simpler.
While moving the comment in nbd_co_establish_connection() rework it to
give better information. Also, we are going to move the connection code
to separate file and mentioning drained section would be confusing.
Improve also the comment in NBDConnectThread, while dropping removed
state names from it.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210610100802.5888-12-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: comment tweak] Signed-off-by: Eric Blake <eblake@redhat.com>
block/nbd: simplify waking of nbd_co_establish_connection()
Instead of managing connect_bh, bh_ctx, and wait_connect fields, we
can use a single link to the waiting coroutine with proper mutex
protection.
So new logic is:
nbd_co_establish_connection() sets wait_co under the mutex, releases
the mutex, then yield()s. Note that wait_co may be scheduled by the
thread immediately after unlocking the mutex. Still, the main thread
(or iothread) will not reach the code for entering the coroutine until
the yield(), so we are safe.
connect_thread_func() and nbd_co_establish_connection_cancel() do
the following to handle wait_co:
Under the mutex, if thr->wait_co is not NULL, make it NULL and
schedule it. This way, we avoid scheduling the coroutine twice.
Still scheduling is a bit different:
In connect_thread_func() we can just call aio_co_wake under mutex,
after commit
[async: the main AioContext is only "current" if under the BQL]
we are sure that aio_co_wake() will not try to acquire the aio context
and do qemu_aio_coroutine_enter() but simply schedule the coroutine by
aio_co_schedule().
nbd_co_establish_connection_cancel() will be called from non-coroutine
context in further patch and will be able to go through
qemu_aio_coroutine_enter() path of aio_co_wake(). So keep current
behavior of waking the coroutine after the critical section.
Also, this commit reduces the dependence of
nbd_co_establish_connection() on the internals of bs (we now use a
generic pointer to the coroutine, instead of direct use of
s->connection_co). This is a step towards splitting the connection
API out of nbd.c.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210610100802.5888-11-vsementsov@virtuozzo.com> Reviewied-by: Eric Blake <eblake@redhat.com> Signed-off-by: Eric Blake <eblake@redhat.com>
block/nbd: BDRVNBDState: drop unused connect_err and connect_status
These fields are write-only. Drop them.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Roman Kagan <rvkagan@yandex-team.ru> Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210610100802.5888-10-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
block/nbd: nbd_client_handshake(): fix leak of s->ioc
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Roman Kagan <rvkagan@yandex-team.ru>
Message-Id: <20210610100802.5888-9-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake <eblake@redhat.com>
projects / thirdparty / qemu.git / log
