Cole Robinson [Mon, 9 Dec 2019 14:02:56 +0000 (09:02 -0500)]
Fix src/ pkg-config API XML paths
The .pc files in src/ are intended for use with the ./run script,
to ease building bindings against an uninstalled libvirt build.
The pointer to the API XML files is incorrect though, it needs to
point into the build tree.
This fixes use of the run script for building libvirt-python, ex:
/path/to/libvirt.git/run ./setup.py build
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Cole Robinson <crobinso@redhat.com>
Peter Krempa [Tue, 3 Dec 2019 16:38:13 +0000 (17:38 +0100)]
qemu: Add support for VIR_DOMAIN_CAPS_FEATURE_BACKUP
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Tue, 3 Dec 2019 16:36:35 +0000 (17:36 +0100)]
conf: domaincaps: Add 'backup' feature flag
This flag will allow figuring out whether the hypervisor supports the
incremental backup and checkpoint features.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Tue, 26 Nov 2019 17:25:49 +0000 (18:25 +0100)]
qemu: backup: Implement stats gathering while the job is running
We can use the output of 'query-jobs' to figure out some useful
information about a backup job. That is progress in case of a push job
and scratch file use in case of a pull job.
Add a worker which will total up the data and call it from
qemuDomainGetJobStatsInternal.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Tue, 26 Nov 2019 21:21:22 +0000 (22:21 +0100)]
doc: Document quirk of getting block job info for a 'backup' blockjob
The stats reported for a blockjob which is member of a domain pull
backup refer to the utilization of the scratch file rather than the
progress of the backup as the progress of the backup depends on the
client. Note this quirk in the docs.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Tue, 26 Nov 2019 17:11:19 +0000 (18:11 +0100)]
conf: backup: Add fields for tracking stats of completed sub-jobs
We need a place to store stats of completed sub-jobs so that we can
later report accurate stats.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Wed, 18 Sep 2019 10:45:31 +0000 (12:45 +0200)]
tests: qemustatusxml2xml: Add test for 'pull' type backup job
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Wed, 16 Oct 2019 07:39:32 +0000 (09:39 +0200)]
qemu: blockjob: Track internal data for 'backup' blockjob
A backup blockjob needs to be able to notify the parent backup job as
well as track all data to be able to clean up the bitmap and blockdev
used for the backup.
Add the data structure, job allocation function and status XML formatter
and parser.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Wed, 18 Sep 2019 09:27:05 +0000 (11:27 +0200)]
qemu: domain: Track backup job data in the status XML
Store the data of a backup job along with the index counter for new
backup jobs in the status XML. Currently we will support only one
backup job and thus there's no necessity to add arrays of jobs.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Fri, 27 Sep 2019 15:28:48 +0000 (17:28 +0200)]
qemu: monitor: Add support for blockdev-backup via 'transaction'
Implement the transaction actions generator for blockdev-backup.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Fri, 18 Oct 2019 13:10:33 +0000 (15:10 +0200)]
Add 'backup' block job type
A backup job may consist of many backup sub-blockjobs. Add the new
blockjob type and add all type converter strings.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Fri, 22 Nov 2019 16:19:49 +0000 (17:19 +0100)]
qemu: domain: Introduce QEMU_ASYNC_JOB_BACKUP async job type
We will want to use the async job infrastructure along with all the APIs
and event for the backup job so add the backup job as a new async job
type.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Fri, 22 Nov 2019 16:30:05 +0000 (17:30 +0100)]
qemu: Add infrastructure for statistics of a backup job
Introduce QEMU_DOMAIN_JOB_STATS_TYPE_BACKUP and the convertors and other
plumbing to be able to report statistics for the backup job.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Eric Blake [Thu, 22 Aug 2019 01:42:45 +0000 (20:42 -0500)]
backup: Implement virsh support for backup
Introduce virsh commands for performing backup jobs.
Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Tue, 3 Dec 2019 12:49:04 +0000 (13:49 +0100)]
tests: genericxml2xml: Add testing of backup XML files
Now that the parser and formatter are in place we can exercise it on
the test files.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Eric Blake [Thu, 22 Aug 2019 01:42:43 +0000 (20:42 -0500)]
backup: Implement backup APIs for remote driver
This one is fairly straightforward - the generator already does what
we need.
Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Fri, 22 Nov 2019 15:35:27 +0000 (16:35 +0100)]
API: Add domain job operation for backups
Introduce VIR_DOMAIN_JOB_OPERATION_BACKUP into virDomainJobOperation.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Eric Blake [Thu, 22 Aug 2019 01:42:42 +0000 (20:42 -0500)]
backup: Introduce virDomainBackup APIs
Introduce a few new public APIs related to incremental backups. This
builds on the previous notion of a checkpoint (without an existing
checkpoint, the new API is a full backup, differing from
virDomainBlockCopy in the point of time chosen and in operation on
multiple disks at once); and also allows creation of a new checkpoint
at the same time as starting the backup (after all, an incremental
backup is only useful if it covers the state since the previous
backup).
A backup job also affects filtering a listing of domains, as well as
adding event reporting for signaling when a push model backup
completes (where the hypervisor creates the backup); note that the
pull model does not have an event (starting the backup lets a third
party access the data, and only the third party knows when it is
finished).
The full list of new APIs:
virDomainBackupBegin;
virDomainBackupGetXMLDesc;
Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Eric Blake [Thu, 22 Aug 2019 01:42:41 +0000 (20:42 -0500)]
backup: Document new XML for backups
Prepare for new backup APIs by describing the XML that will represent
a backup. The XML resembles snapshots and checkpoints in being able
to select actions for a set of disks, but has other differences. It
can support both push model (the hypervisor does the backup directly
into the destination file) and pull model (the hypervisor exposes an
access port for a third party to grab what is necessary). Add
testsuite coverage for some minimal uses of the XML.
The <disk> element within <domainbackup> tries to model the same
elements as a <disk> under <domain>, but sharing the RNG grammar
proved to be hairy. That is in part because while <domain> use
<source> to describe a host resource in use by the guest, a backup job
is using a host resource that is not visible to the guest: a push
backup action is instead describing a <target> (which ultimately could
be a remote network resource, but for simplicity the RNG just
validates a local file for now), and a pull backup action is instead
describing a temporary local file <scratch> (which probably should not
be a remote resource). A future refactoring may thus introduce some
way to parameterize RNG to accept <disk type='FOO'>...</disk> so that
the name of the subelement can be <source> for domain, or <target> or
<scratch> as needed for backups. Future patches may improve this area
of code.
Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Mon, 25 Nov 2019 16:24:42 +0000 (17:24 +0100)]
virsh: Implement VIR_DOMAIN_JOB_DISK_TEMP_(USED|TOTAL) in cmdDomjobinfo
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Mon, 25 Nov 2019 16:19:48 +0000 (17:19 +0100)]
API: Introduce field for reporting temporary disk space usage of a domain job
A pull mode backup job uses temporary disk images to hold the changed
parts of the disk while the client is copying the changes. Since usage
of the temporary space can be monitored but doesn't really fit any of
the existing stats fields introduce new fields for reporting this data.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Tue, 3 Dec 2019 14:17:23 +0000 (15:17 +0100)]
qemu: domain: Export qemuDomainGetImageIds
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Cole Robinson [Mon, 9 Dec 2019 16:34:45 +0000 (11:34 -0500)]
qemu: domain: raise an error when no emulator is found
$ cat f | grep -e arch -e emulator
<type arch='mipsel'>hvm</type>
$ sudo virsh define f
error: Failed to define domain from f
error: An error occurred, but the cause is unknown
After:
$ sudo virsh define f
error: Failed to define domain from f
error: unsupported configuration: No emulator found for arch 'mipsel'
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Cole Robinson <crobinso@redhat.com>
Peter Krempa [Mon, 9 Dec 2019 11:44:41 +0000 (12:44 +0100)]
qemu: snapshot: Mark file becoming backingStore as read-only
For any backing file we set 'read-only' to true, but didn't do this when
modifying the recorded backing store when creating external snapshots.
This meant that qemu would attempt to open the backing-file read-write.
This would fail for example when selinux is used as qemu doesn't have
write permission for the backing file.
qemu: stop creating capabilities at driver startup
Now that nearly all internal APIs use the QEMU capabilities or other
QEMU driver data directly, there's no compelling benefit to create
virCapsPtr at driver startup.
Skipping this means we don't probe capabilities for all 30 system
emulator targets at startup, only those emulators which are referenced
by an XML doc. This massively improves libvirtd startup time when the
capabilities cache is not populated. It even improves startup time
when the cache is up to date, as we don't bother to load files from
the cache until we need them.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
We always refresh the capabilities object when using virResctrlInfo
during process startup. This is undesirable overhead, because we can
just directly create a virResctrlInfo instead.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Annoyingly there was no existing constructor, and identifying all the
places which do a VIR_ALLOC(cpu) is a bit error prone. Hopefully this
has found & converted them all.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
conf: move NUMA capabilities into self contained object
The NUMA cells are stored directly in the virCapsHostPtr
struct. This moves them into their own struct allowing
them to be stored independantly of the rest of the host
capabilities. The change is used as an excuse to switch
the representation to use a GPtrArray too.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
conf: move seclabel validation into post-parse phase
Currently the disk and chardev seclabels are validated immediately at
the time their data is parsed. This forces the parser to fill in the
top level secmodel at time of parsing which is an undesirable thing.
This validation conceptually should be done in the post-parse phase
instead.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
conf: pass netprefix in the domain parser config struct
Instead of using the virCapsPtr information, pass the driver specific
netprefix in the domain parser struct. This eliminates one more use of
virCapsPtr from the XML parsing/formatting code.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
conf: don't use passed in caps in post parse method
To enable the virCapsPtr parameter to the post parse method to be
eliminated, the drivers must fetch the virCapsPtr from their own
driver via the opaque parameter, or use an alternative approach
to validate the parsed data.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
conf: move virt type / os type / arch validation to post-parse
The XML parser currently calls virCapabilitiesDomainDataLookup during
parsing to find the domain capabilities matching the triple
(virt type, os type, arch)
This is, however, bogus with the QEMU driver as it assumes that there
is an emulator known to the default driver capabilities that matches
this triple. It is entirely possible for the driver to be parsing an
XML file with a custom emulator path specified pointing to a binary
that doesn't exist in the default driver capabilities. This will,
for example be the case on a RHEL host which only installs the host
native emulator to /usr/bin. The user can have built a custom QEMU
for non-native arches into $HOME and wish to use that.
Aside from validation, this call is also used to fill in a machine type
for the guest if not otherwise specified. Again, this data may be
incorrect for the QEMU driver because it is not taking account of
the emulator binary that is referenced.
To start fixing this, move the validation to the post-parse callbacks
where more intelligent driver specific logic can be applied.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
conf: pass in default architecture via domain XML options
When parsing the guest XML we must fill in the default guest arch if it
is not already present because later parts of the parsing process need
this information.
If no arch is specified we lookup the first guest in the capabilities
data matching the os type and virt type. In most cases this will result
in picking the host architecture but there are some exceptions...
- The test driver is hardcoded to always use i686 arch
- The VMWare/ESX drivers will always place i686 guests ahead
of x86_64 guests in capabilities, so effectively they always
use i686
- The QEMU driver can potentially return any arch at all
depending on what combination of QEMU binaries are installed.
The domain XML hardware configurations are inherently architecture
specific in many places. As a result whomever/whatever created the
domain XML will have had a particular architecture in mind when
specifying the config. In pretty much any sensible case this arch
will have been the native host architecture. i686 on x86_64 is
the only sensible divergance because both these archs are
compatible from a domaain XML config POV.
IOW, although the QEMU driver can pick an almost arbitrary arch as its
default, in the real world no application or user is likely to be
relying on this default arch being anything other than native.
With all this in mind, it is reasonable to change the XML parser to
allow the default architecture to be passed via the domain XML options
struct. If no info is explicitly given then it is safe & sane to pick
the host native architecture as the default for the guest.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Moving their instance parameter to be the first one, and give consistent
ordering of other parameters across all functions. Ensure that the xml
options are passed into both functions in prep for future work.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Our normal practice is for the object type to be the name prefix, and
the object instance be the first parameter passed in.
Rename these to virDomainObjSave and virDomainDefSave moving their
primary parameter to be the first one. Ensure that the xml options
are passed into both functions in prep for future work.
Finally enforce checking of the return type and mark all parameters
as non-NULL.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
qemu: add machines & arch to fake test capabilities cache
Currently the virQEMUCapsPtr objects are just empty. Future patches are
going to expect them to contain real data. Start off by populating the
machine types and arch information.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
As part of a goal to eliminate the need to use virCapsPtr for anything
other than the virConnectGetCapabilies() API impl, cache the host arch
against the QEMU driver struct and use that field directly.
In the tests we move virArchFromHost() globally in testutils.c so that
every test runs with a fixed default architecture reported.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Normally the TLS certificate from the destination host must match the
host's name for TLS verification to succeed. When the certificate does
not match the destination hostname and the expected cetificate's
hostname is known, this parameter can be used to pass this expected
hostname when starting the migration.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
Jiri Denemark [Tue, 3 Dec 2019 13:58:32 +0000 (14:58 +0100)]
qemu: Add support for setting string migration params
The functions for converting migration typed parameters to QEMU
migration parameters and back were only implemented for integer types.
This patch adds support for string parameters.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
Peter Krempa [Fri, 6 Dec 2019 17:50:46 +0000 (18:50 +0100)]
qemu: migration: Properly setup mirror for blockdev configurations
With blockdev we need to refer to the nodename of the disk source image
as the source argument for the blockdev-mirror operation while still
keeping the old job name. With blockdev we must also persist the job in
qemu.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Fri, 6 Dec 2019 16:47:46 +0000 (17:47 +0100)]
qemu: migration: Split out setup of the migration target
Separate out allocation of the virStorageSource corresponding to the
target NBD export of the migration.
As part of the splitout we allocate the export name explicitly as that
one must not change regardless whether blockdev is used or not to
provide compatibility.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Fri, 6 Dec 2019 16:46:29 +0000 (17:46 +0100)]
qemu: blockjob: Allow NULL 'mirror' for block copy jobs due to migration
The non-shared-storage migration tracks the storage source used
explicitly in the migration data so we must allow for processing of the
block job which has NULL mirror as the mirror will not be populated.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Fri, 6 Dec 2019 13:06:55 +0000 (14:06 +0100)]
qemu: driver: Use appropriate job name when setting blockjob speed
qemuDomainBlockJobSetSpeed was not converted to get the job name from
the block job data. This means that after enabling blockdev the API call
would fail as we wouldn't use the appropriate name.
projects / thirdparty / libvirt.git / log
