From: Adrian Vovk Date: Sat, 22 Jun 2024 18:48:50 +0000 (-0400) Subject: sysupdate: Introduce optional features X-Git-Tag: v257-rc1~179^2~4 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=e1384cfb096ad3561e3d20f193c37f59d5379768;p=thirdparty%2Fsystemd.git sysupdate: Introduce optional features Optional features allow distros to define sets of transfers that can be enabled or disabled by the system administrator. This is useful for situations where a distro may want to ship some resources version-locked to the core OS, but many people have no need for the resource, such as: development tools/compilers, drivers for specialized hardware, language packs, etc We also rename sysupdate.d/*.conf -> sysupdate.d/*.transfer, because now there are more than one type of definition in sysupdate.d/. For backwards compat, we still load *.conf files as long as no *.transfer files are found and the *.conf files don't try to declare themselves as part of any features Fixes https://github.com/systemd/systemd/issues/33343 Fixes https://github.com/systemd/systemd/issues/33344 --- diff --git a/TODO b/TODO index 1f8f33c46ff..c5d793c0c8d 100644 --- a/TODO +++ b/TODO @@ -1429,9 +1429,6 @@ Features: - "systemd-sysupdate update --all" support, that iterates through all components defined on the host, plus all images installed into /var/lib/machines/, /var/lib/portable/ and so on. - - figure out what to do about system extensions (i.e. they need to imply an - update component, since otherwise sysupdate.d/ files would override the - host's update files.) - Allow invocation with a single transfer definition, i.e. with --definitions= pointing to a file rather than a dir. - add ability to disable implicit decompression of downloaded artifacts, diff --git a/man/rules/meson.build b/man/rules/meson.build index 080aa44b3de..5f69f01cc48 100644 --- a/man/rules/meson.build +++ b/man/rules/meson.build @@ -1192,6 +1192,7 @@ manpages = [ ['systemd.unit', '5', [], ''], ['systemd.v', '7', [], ''], ['sysupdate.d', '5', [], 'ENABLE_SYSUPDATE'], + ['sysupdate.features', '5', [], 'ENABLE_SYSUPDATE'], ['sysusers.d', '5', [], 'ENABLE_SYSUSERS'], ['telinit', '8', [], 'HAVE_SYSV_COMPAT'], ['timedatectl', '1', [], 'ENABLE_TIMEDATECTL'], diff --git a/man/sysupdate.d.xml b/man/sysupdate.d.xml index 070d6307355..bec423705c7 100644 --- a/man/sysupdate.d.xml +++ b/man/sysupdate.d.xml @@ -23,21 +23,23 @@ - /etc/sysupdate.d/*.conf - /run/sysupdate.d/*.conf - /usr/local/lib/sysupdate.d/*.conf - /usr/lib/sysupdate.d/*.conf + /etc/sysupdate.d/*.transfer + /run/sysupdate.d/*.transfer + /usr/local/lib/sysupdate.d/*.transfer + /usr/lib/sysupdate.d/*.transfer Description - sysupdate.d/*.conf files describe how specific resources on the local system - shall be updated from a remote source. Each such file defines one such transfer: typically a remote - HTTP/HTTPS resource as source; and a local file, directory or partition as target. This may be used as a - simple, automatic, atomic update mechanism for the OS itself, for containers, portable services or system - extension images — but in fact may be used to update any kind of file from a remote source. + These files describe how specific resources on the local system shall be updated from a remote + source. + Each such file defines one such transfer: typically a remote HTTP/HTTPS resource as source; and a local + file, directory or partition as target. + This may be used as a simple, automatic, atomic update mechanism for the OS itself, for containers, + portable services or system extension images — but in fact may be used to update any kind of file from a + remote source. The systemd-sysupdate8 @@ -48,14 +50,22 @@ versions, in order to implement flexible update schemes, e.g. A/B updating (or a superset thereof, e.g. A/B/C, A/B/C/D, …). - Each *.conf file defines one transfer, i.e. describes one resource to - update. Typically, multiple of these files (i.e. multiple of such transfers) are defined together, and + Each *.transfer file defines one transfer, i.e. describes one resource to + update. + Typically, multiple of these files (i.e. multiple of such transfers) are defined together, and are bound together by a common version identifier in order to update multiple resources at once on each update operation, for example to update a kernel, a root file system and a Verity partition in a single, combined, synchronized operation, so that only a combined update of all three together constitutes a - complete update. + complete update. + We'll call such a collection of transfers a target. + systemd-sysupdate always operates on a single target. - Each *.conf file contains three sections: [Transfer], [Source] and [Target]. + Transfers may be grouped together into sets that can be individually enabled or disabled by the + system administrator, called "Optional Features": + sysupdate.features5. + + + Each *.transfer file contains three sections: [Transfer], [Source] and [Target]. @@ -106,8 +116,8 @@ An update can only complete if the relevant URLs provide their resources for the same version, i.e. for the same value of @v. - The above may be translated into three *.conf files in - sysupdate.d/, one for each resource to transfer. The *.conf + The above may be translated into three *.transfer files in + sysupdate.d/, one for each resource to transfer. The *.transfer files configure the type of download, and what place to write the download to (i.e. whether to a partition or a file in the file system). Most importantly these files contain the URL, partition name and filename patterns shown above that describe how these resources are called on the source and how they @@ -130,11 +140,11 @@ lists file names and their SHA256 hashes. - Transfers are done in the alphabetical order of the .conf file names they are + Transfers are done in the alphabetical order of the .transfer file names they are defined in. First, the resource data is downloaded directly into a target file/directory/partition. Once this is completed for all defined transfers, in a second step the files/directories/partitions are renamed to their final names as defined by the target MatchPattern=, again in the - order the .conf transfer file names dictate. This step is not atomic, however it is + order the .transfer transfer file names dictate. This step is not atomic, however it is guaranteed to be executed strictly in order with suitable disk synchronization in place. Typically, when updating an OS one of the transfers defines the entry point when booting. Thus it is generally a good idea to order the resources via the transfer configuration file names so that the entry point is written @@ -519,6 +529,39 @@ + + + Features= + + A space-separated list of + sysupdate.features5 + that this transfer belongs to, by name. + This option may be specified more than once, in which case the specified list of features is merged. + If the empty string is assigned to this option, the list is reset and all prior assignments will have + no effect. + For example: Features=foo bar specifies that the transfer belongs to + foo.feature and bar.feature. + + If the list of features is empty, then this transfer is always used. + If this transfer belongs to more than one feature, then it will be used if any + one of the listed features is enabled. + A name that does not correspond to a defined feature will resolve to an implicit feature that is + always disabled. + + + + + + RequisiteFeatures= + + This is like Features=, except that all + features listed here must be enabled for this transfer to be enabled. + If both options are specified, then they both apply: the transfer will be enabled only if all + features specified here are enabled, and at least one feature listed in Features= + is enabled. + + + @@ -869,7 +912,7 @@ partition, and a unified kernel image to update as one. This example is an extension of the example discussed earlier in this man page. - # /usr/lib/sysupdate.d/50-verity.conf + # /usr/lib/sysupdate.d/50-verity.transfer [Transfer] ProtectVersion=%A @@ -899,7 +942,7 @@ ReadOnly=1 systemd-gpt-auto-generator8 suggests). - # /usr/lib/sysupdate.d/60-root.conf + # /usr/lib/sysupdate.d/60-root.transfer [Transfer] ProtectVersion=%A @@ -918,7 +961,7 @@ ReadOnly=1 The above defines a matching transfer definition for the root file system. - # /usr/lib/sysupdate.d/70-kernel.conf + # /usr/lib/sysupdate.d/70-kernel.transfer [Transfer] ProtectVersion=%A diff --git a/man/sysupdate.features.xml b/man/sysupdate.features.xml new file mode 100644 index 00000000000..d33df94490f --- /dev/null +++ b/man/sysupdate.features.xml @@ -0,0 +1,338 @@ + + + + + + + + sysupdate.features + systemd + + + + sysupdate.features + 5 + + + + sysupdate.features + Definition Files for Optional Features + + + + + /etc/sysupdate.d/*.feature + /run/sysupdate.d/*.feature + /usr/local/lib/sysupdate.d/*.feature + /usr/lib/sysupdate.d/*.feature + + + + + Description + + "Optional Features" are functionality provided by + systemd-sysupdate8, + that allow a distribution to define sets of + sysupdate.d5 + transfer definitions that are intended to be enabled or disabled by the system administrator. + + When a feature is enabled, transfers belonging to it will be considered when checking for and + downloading updates, when vacuuming, and in all other situations. + In effect, transfers belonging to a feature will always be updated in lock-step with the rest of their + target. + This is the primary difference an Optional Feature and a systemd-sysupdate component. + When a feature is disabled, its transfers will not be considered when checking for and downloading updates, + but systemd-sysupdate will still consider them while vacuuming and in other situations + where it needs to determine ownership over previously downloaded system resources. + systemd-sysupdate will clean up all instances of the feature's transfers whenever it + is disabled, effectively uninstalling it. + + Optional Features are described by sysupdate.d/*.feature files, which are + defined below. + Transfers can declare that they belong to a feature via the Features= setting. + Feature definitions support drop-in files, which are most commonly used to override the + Enabled= setting). + They can also be masked out to hide the availability of the feature entirely. + + Each *.feature file contains one section: [Feature]. + + + + [Feature] Section Options + + This section defines general properties of this feature. + + + + Description= + + A short human readable description of this feature. + This may be used as a label for this feature, so the string should meaningfully identify the feature + among the features available in sysupdate.d/. + + + + + + Documentation= + + A user-presentable URL to documentation about this feature. + This setting supports specifier expansion; see below for details on supported specifiers. + + + + + + AppStream= + + A URL to an + AppStream catalog + XML file. + This may be used by software centers (such as GNOME Software or KDE Discover) to present rich + metadata about this feature. + This includes display names, chagnelogs, icons, and more. + This setting supports specifier expansion; see below for details on supported specifiers. + + + + + + Enabled= + + Whether or not this feature is enabled. If unspecified, the feature is disabled + by default. + + + + + + + + Specifiers + + Specifiers may be used in the Documentation= and AppStream= + settings. The following expansions are understood: + + + Specifiers available + + + + + + + Specifier + Meaning + Details + + + + + + + + + + + + + + + + + + + + +
+ + + + Examples + + + Development Tools for Image-Based OS + + We'll use the hypothetical "foobarOS" described in + sysupdate.d5 as our + example base OS. + The vast majority of foobarOS's users have no need for a compiler, build system, debugger, and other + such development tools to be part of their OS. + However, the developers of foobarOS itself need this build tooling to be available. + So, foobarOS needs to provide a system extension image (see + systemd-sysext8) + containing these development tools, and this image must be updated in lock-step with the underlying + base OS. + This is a great use case for an optional OS feature, so let's define one: + + + # /usr/lib/sysupdate.d/devel.feature +[Feature] +Description=Development Tools +Documentation=https://developer.example.com/foobarOS/getting-started +Enabled=false + + + The above defines the devel feature, and disables it by default. + Now let's a define a transfer that's associated with this feature: + + # /usr/lib/sysupdate.d/50-devel.transfer +[Transfer] +Features=devel +ProtectVersion=%A + +[Source] +Type=url-file +Path=https://download.example.com/ +MatchPattern=foobarOS_@v_devel.raw.xz + +[Target] +Type=regular-file +Path=/var/lib/extensions +MatchPattern=foobarOS_@v_devel.raw +Mode=0444 +InstancesMax=2 + + + With these two files, we have created a feature called devel that, when + enabled, will download and decompress the appropriate version of + https://download.example.com/foobarOS_@v_devel.raw.xz into + /var/lib/extensions/foobarOS_@v_devel.raw during each OS update. + + The developers of foobarOS can enable the devel feature on their workstations + by creating the following drop-in: + + # /etc/sysupdate.d/devel.feature.d/enable.conf +[Feature] +Enabled=true + + + + + Proprietary Drivers + + Suppose that many of foobarOS's users have a GPU manufactured by the MVISUAL corporation. + Due to lack of documentation and difficulty in reverse-engineering the hardware, the open-source + drivers for MVISUAL GPUs are unable to make proper use of available graphics and compute performance. + MVISUAL provides a redistributable proprietary driver for their cards, and foobarOS's developers + distribute them to address their users' needs. + + MVISUAL's driver has a couple different parts that must be installed for it to function: a UKI + addon to configure the kernel command-line, an initrd system extension image to add the MVISUAL kernel + module into the initrd, and a regular system extension image to add the proprietary OpenGL and Vulkan + userspace drivers. + All of these should be version-locked to the core OS. + + Let's start by defining an optional feature named mvisual-driver: + + # /usr/lib/sysupdate.d/mvisual-driver.feature +[Feature] +Description=MVISUAL Proprietary GPU Driver +Documentation=https://support.example.com/foobarOS/mvisual +AppStream=https://metadata.example.com/mvisual-driver-%A.xml.gz + + + Note that we define AppStream metadata for this feature, because we want software centers to + present it to end-users. + Next, let's define the corresponding transfers: + + # /usr/lib/sysupdate.d/50-mvisual-userspace.transfer +[Transfer] +Features=mvisual-driver +ProtectVersion=%A + +[Source] +Type=url-file +Path=https://download.example.com/ +MatchPattern=foobarOS_@v_mvisual_userspace.raw.xz + +[Target] +Type=regular-file +Path=/var/lib/extensions +MatchPattern=foobarOS_@v_mvisual.raw +Mode=0444 +InstancesMax=2 + + + # /usr/lib/sysupdate.d/70-mvisual-initrd.transfer +[Transfer] +Features=mvisual-driver +ProtectVersion=%A + +[Source] +Type=url-file +Path=https://download.example.com/ +MatchPattern=foobarOS_@v_mvisual_initrd.raw.xz + +[Target] +Type=regular-file +Path=/EFI/Linux +PathRelativeTo=boot +MatchPattern=foobarOS_@v.efi.extra.d/foobarOS_mvisual.raw +Mode=0444 +InstancesMax=2 + + + # /usr/lib/sysupdate.d/90-mvisual-addon.transfer +[Transfer] +Features=mvisual-driver +ProtectVersion=%A + +[Source] +Type=url-file +Path=https://download.example.com/ +MatchPattern=foobarOS_@v_mvisual_addon.efi.xz + +[Target] +Type=regular-file +Path=/EFI/Linux +PathRelativeTo=boot +MatchPattern=foobarOS_@v.efi.extra.d/foobarOS_mvisual.addon.efi +Mode=0444 +InstancesMax=2 + + + + + Intersecting Features + + Suppose that MVISUAL releases special tooling to help a distribution developer troubleshoot + crashes in their proprietary driver. + Let's define a transfer: + + # /usr/lib/sysupdate.d/50-mvisual-debugger.transfer +[Transfer] +RequisiteFeatures=devel mvisual-driver +ProtectVersion=%A + +[Source] +Type=url-file +Path=https://download.example.com/ +MatchPattern=foobarOS_@v_devel.raw.xz + +[Target] +Type=regular-file +Path=/var/lib/extensions +MatchPattern=foobarOS_@v_devel.raw +Mode=0444 +InstancesMax=2 + + + This transfer will be used only if both the devel and + mvisual-driver features are enabled. + + + + + See Also + + systemd1 + systemd-sysupdate8 + sysupdate.d5 + + + + diff --git a/src/sysupdate/meson.build b/src/sysupdate/meson.build index 9186a454f2a..0c7c469a620 100644 --- a/src/sysupdate/meson.build +++ b/src/sysupdate/meson.build @@ -2,6 +2,7 @@ systemd_sysupdate_sources = files( 'sysupdate-cache.c', + 'sysupdate-feature.c', 'sysupdate-instance.c', 'sysupdate-partition.c', 'sysupdate-pattern.c', diff --git a/src/sysupdate/sysupdate-feature.c b/src/sysupdate/sysupdate-feature.c new file mode 100644 index 00000000000..05c1924d353 --- /dev/null +++ b/src/sysupdate/sysupdate-feature.c @@ -0,0 +1,121 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ + +#include "conf-parser.h" +#include "hash-funcs.h" +#include "path-util.h" +#include "sysupdate-feature.h" +#include "sysupdate.h" +#include "web-util.h" + +static Feature *feature_free(Feature *f) { + if (!f) + return NULL; + + free(f->id); + + free(f->description); + free(f->documentation); + free(f->appstream); + + return mfree(f); +} + +Feature *feature_new(void) { + Feature *f; + + f = new(Feature, 1); + if (!f) + return NULL; + + *f = (Feature) { + .n_ref = 1, + }; + + return f; +} + +DEFINE_TRIVIAL_REF_UNREF_FUNC(Feature, feature, feature_free); + +DEFINE_HASH_OPS_WITH_VALUE_DESTRUCTOR(feature_hash_ops, + char, string_hash_func, string_compare_func, + Feature, feature_unref); + +static int config_parse_url_specifiers( + const char *unit, + const char *filename, + unsigned line, + const char *section, + unsigned section_line, + const char *lvalue, + int ltype, + const char *rvalue, + void *data, + void *userdata) { + char **s = ASSERT_PTR(data); + _cleanup_free_ char *resolved = NULL; + int r; + + assert(rvalue); + + if (isempty(rvalue)) { + *s = mfree(*s); + return 0; + } + + r = specifier_printf(rvalue, NAME_MAX, specifier_table, arg_root, NULL, &resolved); + if (r < 0) { + log_syntax(unit, LOG_WARNING, filename, line, r, + "Failed to expand specifiers in %s=, ignoring: %s", lvalue, rvalue); + return 0; + } + + if (!http_url_is_valid(resolved)) { + log_syntax(unit, LOG_WARNING, filename, line, 0, + "%s= URL is not valid, ignoring: %s", lvalue, rvalue); + return 0; + } + + return free_and_replace(*s, resolved); +} + + +int feature_read_definition(Feature *f, const char *path, const char *const *dirs) { + assert(f); + + ConfigTableItem table[] = { + { "Feature", "Description", config_parse_string, 0, &f->description }, + { "Feature", "Documentation", config_parse_url_specifiers, 0, &f->documentation }, + { "Feature", "AppStream", config_parse_url_specifiers, 0, &f->appstream }, + { "Feature", "Enabled", config_parse_bool, 0, &f->enabled }, + {} + }; + + _cleanup_free_ char *filename = NULL; + int r; + + assert(path); + assert(dirs); + + r = path_extract_filename(path, &filename); + if (r < 0) + return log_error_errno(r, "Failed to extract filename from path '%s': %m", path); + + r = config_parse_many( + STRV_MAKE_CONST(path), + dirs, + strjoina(filename, ".d"), + arg_root, + "Feature\0", + config_item_table_lookup, table, + CONFIG_PARSE_WARN, + /* userdata= */ NULL, + /* stats_by_path= */ NULL, + /* drop_in_files= */ NULL); + if (r < 0) + return r; + + *ASSERT_PTR(endswith(filename, ".feature")) = 0; /* Remove the file extension */ + f->id = TAKE_PTR(filename); + + return 0; +} diff --git a/src/sysupdate/sysupdate-feature.h b/src/sysupdate/sysupdate-feature.h new file mode 100644 index 00000000000..4b52b0a4472 --- /dev/null +++ b/src/sysupdate/sysupdate-feature.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ + +#include "hash-funcs.h" +#include "macro.h" +#include "sysupdate-transfer.h" + +typedef struct Feature { + unsigned n_ref; + + char *id; + + char *description; + char *documentation; + char *appstream; + + bool enabled; +} Feature; + +Feature *feature_new(void); + +Feature *feature_ref(Feature *f); +Feature *feature_unref(Feature *f); +DEFINE_TRIVIAL_CLEANUP_FUNC(Feature*, feature_unref); + +extern const struct hash_ops feature_hash_ops; + +int feature_read_definition(Feature *f, const char *path, const char *const *conf_file_dirs); diff --git a/src/sysupdate/sysupdate-transfer.c b/src/sysupdate/sysupdate-transfer.c index 508fd73355f..350a12ee0f8 100644 --- a/src/sysupdate/sysupdate-transfer.c +++ b/src/sysupdate/sysupdate-transfer.c @@ -28,6 +28,7 @@ #include "stdio-util.h" #include "strv.h" #include "sync-util.h" +#include "sysupdate-feature.h" #include "sysupdate-pattern.h" #include "sysupdate-resource.h" #include "sysupdate-transfer.h" @@ -44,7 +45,6 @@ Transfer* transfer_free(Transfer *t) { t->temporary_path = rm_rf_subvolume_and_free(t->temporary_path); - free(t->definition_path); free(t->min_version); strv_free(t->protected_versions); free(t->current_symlink); @@ -93,12 +93,6 @@ Transfer* transfer_new(Context *ctx) { return t; } -static const Specifier specifier_table[] = { - COMMON_SYSTEM_SPECIFIERS, - COMMON_TMP_SPECIFIERS, - {} -}; - static int config_parse_protect_version( const char *unit, const char *filename, @@ -466,11 +460,33 @@ static int config_parse_partition_flags( return 0; } -int transfer_read_definition(Transfer *t, const char *path) { - int r; +static bool transfer_decide_if_enabled(Transfer *t, Hashmap *known_features) { + assert(t); + + /* Requisite feature disabled -> transfer disabled */ + STRV_FOREACH(id, t->requisite_features) { + Feature *f = hashmap_get(known_features, *id); + if (!f || !f->enabled) /* missing features are implicitly disabled */ + return false; + } + + /* No features defined -> transfer implicitly enabled */ + if (strv_isempty(t->features)) + return true; + + /* At least one feature enabled -> transfer enabled */ + STRV_FOREACH(id, t->features) { + Feature *f = hashmap_get(known_features, *id); + if (f && f->enabled) + return true; + } + + /* All listed features disabled -> transfer disabled */ + return false; +} +int transfer_read_definition(Transfer *t, const char *path, const char **dirs, Hashmap *known_features) { assert(t); - assert(path); ConfigTableItem table[] = { { "Transfer", "MinVersion", config_parse_min_version, 0, &t->min_version }, @@ -478,6 +494,8 @@ int transfer_read_definition(Transfer *t, const char *path) { { "Transfer", "Verify", config_parse_bool, 0, &t->verify }, { "Transfer", "ChangeLog", config_parse_url_specifiers, 0, &t->changelog }, { "Transfer", "AppStream", config_parse_url_specifiers, 0, &t->appstream }, + { "Transfer", "Features", config_parse_strv, 0, &t->features }, + { "Transfer", "RequisiteFeatures", config_parse_strv, 0, &t->requisite_features }, { "Source", "Type", config_parse_resource_type, 0, &t->source.type }, { "Source", "Path", config_parse_resource_path, 0, &t->source }, { "Source", "PathRelativeTo", config_parse_resource_path_relto, 0, &t->source.path_relative_to }, @@ -501,17 +519,34 @@ int transfer_read_definition(Transfer *t, const char *path) { {} }; - r = config_parse(NULL, path, NULL, - "Transfer\0" - "Source\0" - "Target\0", - config_item_table_lookup, table, - CONFIG_PARSE_WARN, - t, - NULL); + _cleanup_free_ char *filename = NULL; + int r; + + assert(path); + assert(dirs); + + r = path_extract_filename(path, &filename); + if (r < 0) + return log_error_errno(r, "Failed to extract filename from path '%s': %m", path); + + r = config_parse_many( + STRV_MAKE_CONST(path), + dirs, + strjoina(filename, ".d"), + arg_root, + "Transfer\0" + "Source\0" + "Target\0", + config_item_table_lookup, table, + CONFIG_PARSE_WARN, + /* userdata= */ NULL, + /* stats_by_path= */ NULL, + /* drop_in_files= */ NULL); if (r < 0) return r; + t->enabled = transfer_decide_if_enabled(t, known_features); + if (!RESOURCE_IS_SOURCE(t->source.type)) return log_syntax(NULL, LOG_ERR, path, 1, SYNTHETIC_ERRNO(EINVAL), "Source Type= must be one of url-file, url-tar, tar, regular-file, directory, subvolume."); @@ -704,6 +739,8 @@ int transfer_vacuum( assert(instances_max >= 1); if (instances_max == UINT64_MAX) /* Keep infinite instances? */ limit = UINT64_MAX; + else if (space == UINT64_MAX) /* forcibly delete all instances? */ + limit = 0; else if (space > instances_max) return log_error_errno(SYNTHETIC_ERRNO(ENOSPC), "Asked to delete more instances than total maximum allowed number of instances, refusing."); @@ -713,7 +750,7 @@ int transfer_vacuum( else limit = instances_max - space; - if (t->target.type == RESOURCE_PARTITION) { + if (t->target.type == RESOURCE_PARTITION && space != UINT64_MAX) { uint64_t rm, remain; /* If we are looking at a partition table, we also have to take into account how many @@ -767,7 +804,11 @@ int transfer_vacuum( assert(oldest->resource); - log_info("%s Removing old '%s' (%s).", special_glyph(SPECIAL_GLYPH_RECYCLING), oldest->path, resource_type_to_string(oldest->resource->type)); + log_info("%s Removing %s '%s' (%s).", + special_glyph(SPECIAL_GLYPH_RECYCLING), + space == UINT64_MAX ? "disabled" : "old", + oldest->path, + resource_type_to_string(oldest->resource->type)); switch (t->target.type) { diff --git a/src/sysupdate/sysupdate-transfer.h b/src/sysupdate/sysupdate-transfer.h index 497cdccd7c1..841161846f4 100644 --- a/src/sysupdate/sysupdate-transfer.h +++ b/src/sysupdate/sysupdate-transfer.h @@ -15,12 +15,13 @@ typedef struct Transfer Transfer; #include "sysupdate.h" struct Transfer { - char *definition_path; char *min_version; char **protected_versions; char *current_symlink; bool verify; + bool enabled; + Resource source, target; uint64_t instances_max; @@ -55,11 +56,10 @@ struct Transfer { typedef int (*TransferProgress)(const Transfer *t, const Instance *inst, unsigned percentage); Transfer* transfer_new(Context *ctx); - Transfer* transfer_free(Transfer *t); DEFINE_TRIVIAL_CLEANUP_FUNC(Transfer*, transfer_free); -int transfer_read_definition(Transfer *t, const char *path); +int transfer_read_definition(Transfer *t, const char *path, const char **dirs, Hashmap *features); int transfer_resolve_paths(Transfer *t, const char *root, const char *node); diff --git a/src/sysupdate/sysupdate.c b/src/sysupdate/sysupdate.c index 490466228d1..0cdcb34a588 100644 --- a/src/sysupdate/sysupdate.c +++ b/src/sysupdate/sysupdate.c @@ -25,9 +25,11 @@ #include "set.h" #include "signal-util.h" #include "sort-util.h" +#include "specifier.h" #include "string-util.h" #include "strv.h" #include "sysupdate.h" +#include "sysupdate-feature.h" #include "sysupdate-transfer.h" #include "sysupdate-update-set.h" #include "sysupdate-util.h" @@ -57,10 +59,21 @@ STATIC_DESTRUCTOR_REGISTER(arg_component, freep); STATIC_DESTRUCTOR_REGISTER(arg_image_policy, image_policy_freep); STATIC_DESTRUCTOR_REGISTER(arg_transfer_source, freep); +const Specifier specifier_table[] = { + COMMON_SYSTEM_SPECIFIERS, + COMMON_TMP_SPECIFIERS, + {} +}; + typedef struct Context { Transfer **transfers; size_t n_transfers; + Transfer **disabled_transfers; + size_t n_disabled_transfers; + + Hashmap *features; /* Defined features, keyed by ID */ + UpdateSet **update_sets; size_t n_update_sets; @@ -77,6 +90,12 @@ static Context* context_free(Context *c) { transfer_free(*tr); free(c->transfers); + FOREACH_ARRAY(tr, c->disabled_transfers, c->n_disabled_transfers) + transfer_free(*tr); + free(c->disabled_transfers); + + hashmap_free(c->features); + FOREACH_ARRAY(us, c->update_sets, c->n_update_sets) update_set_free(*us); free(c->update_sets); @@ -93,83 +112,145 @@ static Context* context_new(void) { return new0(Context, 1); } -static int context_read_definitions( +static void free_transfers(Transfer **array, size_t n) { + FOREACH_ARRAY(t, array, n) + transfer_free(*t); + free(array); +} + +static int read_definitions( Context *c, - const char *directory, - const char *component, - const char *root, + const char **dirs, + const char *suffix, const char *node) { _cleanup_strv_free_ char **files = NULL; + Transfer **transfers = NULL, **disabled = NULL; + size_t n_transfers = 0, n_disabled = 0; int r; + CLEANUP_ARRAY(transfers, n_transfers, free_transfers); + CLEANUP_ARRAY(disabled, n_disabled, free_transfers); + assert(c); + assert(dirs); + assert(suffix); - if (directory) - r = conf_files_list_strv(&files, ".conf", NULL, CONF_FILES_REGULAR|CONF_FILES_FILTER_MASKED, (const char**) STRV_MAKE(directory)); - else if (component) { - _cleanup_strv_free_ char **n = NULL; + r = conf_files_list_strv(&files, suffix, arg_root, CONF_FILES_REGULAR|CONF_FILES_FILTER_MASKED, dirs); + if (r < 0) + return log_error_errno(r, "Failed to enumerate sysupdate.d/*%s definitions: %m", suffix); + + STRV_FOREACH(p, files) { + _cleanup_(transfer_freep) Transfer *t = NULL; + Transfer **appended; + + t = transfer_new(c); + if (!t) + return log_oom(); + + r = transfer_read_definition(t, *p, dirs, c->features); + if (r < 0) + return r; + + r = transfer_resolve_paths(t, arg_root, node); + if (r < 0) + return r; + + if (t->enabled) + appended = GREEDY_REALLOC_APPEND(transfers, n_transfers, &t, 1); + else + appended = GREEDY_REALLOC_APPEND(disabled, n_disabled, &t, 1); + if (!appended) + return log_oom(); + TAKE_PTR(t); + } + + c->transfers = TAKE_PTR(transfers); + c->n_transfers = n_transfers; + c->disabled_transfers = TAKE_PTR(disabled); + c->n_disabled_transfers = n_disabled; + return 0; +} + +static int context_read_definitions(Context *c, const char* node) { + _cleanup_strv_free_ char **dirs = NULL, **files = NULL; + int r; + + assert(c); + + if (arg_definitions) + dirs = strv_new(arg_definitions); + else if (arg_component) { char **l = CONF_PATHS_STRV(""); - size_t k = 0; + size_t i = 0; - n = new0(char*, strv_length(l) + 1); - if (!n) + dirs = new0(char*, strv_length(l) + 1); + if (!dirs) return log_oom(); - STRV_FOREACH(i, l) { + STRV_FOREACH(dir, l) { char *j; - j = strjoin(*i, "sysupdate.", component, ".d"); + j = strjoin(*dir, "sysupdate.", arg_component, ".d"); if (!j) return log_oom(); - n[k++] = j; + dirs[i++] = j; } - - r = conf_files_list_strv(&files, ".conf", root, CONF_FILES_REGULAR|CONF_FILES_FILTER_MASKED, (const char**) n); } else - r = conf_files_list_strv(&files, ".conf", root, CONF_FILES_REGULAR|CONF_FILES_FILTER_MASKED, (const char**) CONF_PATHS_STRV("sysupdate.d")); + dirs = strv_new(CONF_PATHS("sysupdate.d")); + if (!dirs) + return log_oom(); + + r = conf_files_list_strv(&files, + ".feature", + arg_root, + CONF_FILES_REGULAR|CONF_FILES_FILTER_MASKED, + (const char**) dirs); if (r < 0) - return log_error_errno(r, "Failed to enumerate *.conf files: %m"); + return log_error_errno(r, "Failed to enumerate sysupdate.d/*.feature definitions: %m"); - STRV_FOREACH(f, files) { - _cleanup_(transfer_freep) Transfer *t = NULL; + STRV_FOREACH(p, files) { + _cleanup_(feature_unrefp) Feature *f = NULL; - if (!GREEDY_REALLOC(c->transfers, c->n_transfers + 1)) + f = feature_new(); + if (!f) return log_oom(); - t = transfer_new(c); - if (!t) - return log_oom(); + r = feature_read_definition(f, *p, (const char**) dirs); + if (r < 0) + return r; - t->definition_path = strdup(*f); - if (!t->definition_path) - return log_oom(); + r = hashmap_ensure_put(&c->features, &feature_hash_ops, f->id, f); + if (r < 0) + return log_error_errno(r, "Failed to insert feature '%s' into map: %m", f->id); + TAKE_PTR(f); + } + + r = read_definitions(c, (const char**) dirs, ".transfer", node); + if (r < 0) + return r; - r = transfer_read_definition(t, *f); + if (c->n_transfers + c->n_disabled_transfers == 0) { + /* Backwards-compat: If no .transfer defs are found, fall back to trying .conf! */ + r = read_definitions(c, (const char**) dirs, ".conf", node); if (r < 0) return r; - c->transfers[c->n_transfers++] = TAKE_PTR(t); + if (c->n_transfers + c->n_disabled_transfers > 0) + log_warning("As of v257, transfer definitions should have the '.transfer' extension."); } if (c->n_transfers == 0) { if (arg_component) return log_error_errno(SYNTHETIC_ERRNO(ENOENT), - "No transfer definitions for component '%s' found.", arg_component); + "No transfer definitions for component '%s' found.", + arg_component); return log_error_errno(SYNTHETIC_ERRNO(ENOENT), "No transfer definitions found."); } - FOREACH_ARRAY(tr, c->transfers, c->n_transfers) { - Transfer *t = *tr; - - r = transfer_resolve_paths(t, root, node); - if (r < 0) - return r; - } - return 0; } @@ -191,6 +272,17 @@ static int context_load_installed_instances(Context *c) { return r; } + FOREACH_ARRAY(tr, c->disabled_transfers, c->n_disabled_transfers) { + Transfer *t = *tr; + + r = resource_load_instances( + &t->target, + arg_verify >= 0 ? arg_verify : t->verify, + &c->web_cache); + if (r < 0) + return r; + } + return 0; } @@ -203,7 +295,6 @@ static int context_load_available_instances(Context *c) { FOREACH_ARRAY(tr, c->transfers, c->n_transfers) { Transfer *t = *tr; - assert(t); r = resource_load_instances( &t->source, @@ -750,6 +841,7 @@ static int context_vacuum( uint64_t space, const char *extra_protected_version) { + size_t disabled_count = 0; int r, count = 0; assert(c); @@ -773,15 +865,29 @@ static int context_vacuum( count = MAX(count, r); } + FOREACH_ARRAY(tr, c->disabled_transfers, c->n_disabled_transfers) { + r = transfer_vacuum(*tr, UINT64_MAX /* wipe all instances */, NULL); + if (r < 0) + return r; + if (r > 0) + disabled_count++; + } + if (FLAGS_SET(arg_json_format_flags, SD_JSON_FORMAT_OFF)) { - if (count > 0) + if (count > 0 && disabled_count > 0) + log_info("Removed %i instances, and %zu disabled transfers.", count, disabled_count); + else if (count > 0) log_info("Removed %i instances.", count); + else if (disabled_count > 0) + log_info("Removed %zu disabled transfers.", disabled_count); else - log_info("Removed no instances."); + log_info("Found nothing to remove."); } else { _cleanup_(sd_json_variant_unrefp) sd_json_variant *json = NULL; - r = sd_json_buildo(&json, SD_JSON_BUILD_PAIR_INTEGER("removed", count)); + r = sd_json_buildo(&json, + SD_JSON_BUILD_PAIR_INTEGER("removed", count), + SD_JSON_BUILD_PAIR_UNSIGNED("disabledTransfers", disabled_count)); if (r < 0) return log_error_errno(r, "Failed to create JSON: %m"); @@ -806,7 +912,7 @@ static int context_make_offline(Context **ret, const char *node) { if (!context) return log_oom(); - r = context_read_definitions(context, arg_definitions, arg_component, arg_root, node); + r = context_read_definitions(context, node); if (r < 0) return r; diff --git a/src/sysupdate/sysupdate.h b/src/sysupdate/sysupdate.h index 572b4305c72..ba9fd449686 100644 --- a/src/sysupdate/sysupdate.h +++ b/src/sysupdate/sysupdate.h @@ -4,6 +4,8 @@ #include #include +#include "specifier.h" + /* Forward declare this type so that Transfers can point at it */ typedef struct Context Context; @@ -11,3 +13,5 @@ extern bool arg_sync; extern uint64_t arg_instances_max; extern char *arg_root; extern char *arg_transfer_source; + +extern const Specifier specifier_table[];