core: turn unit_load_fragment_and_dropin_optional() into a flag
unit_load_fragment_and_dropin() and unit_load_fragment_and_dropin_optional()
are really the same, with one minor difference in behaviour. Let's drop
the second function.
"_optional" in the name suggests that it's the "dropin" part that is optional.
(Which it is, but in this case, we mean the fragment to be optional.)
I think the new version with a flag is easier to understand.
parse_hwdb: fix compatibility with pyparsing 2.4.*
pyparsing 2.3.1/2.4.0 had some changes to grouping of And matches, and as a
result we'd report 0 properties and 0 matches, and not really do any checks.
With this change we get identical behaviour for pyparsing 2.3.1, 2.4.0, 2.4.2:
$ hwdb/parse_hwdb.py
hwdb/60-evdev.hwdb: 72 match groups, 94 matches, 262 properties
hwdb/60-input-id.hwdb: 3 match groups, 3 matches, 4 properties
hwdb/60-keyboard.hwdb: 173 match groups, 256 matches, 872 properties
Keycode KBD_LCD_MENU1 unknown
Keycode KBD_LCD_MENU4 unknown
Keycode KBD_LCD_MENU2 unknown
Keycode KBD_LCD_MENU3 unknown
hwdb/60-sensor.hwdb: 101 match groups, 120 matches, 105 properties
hwdb/70-joystick.hwdb: 2 match groups, 3 matches, 2 properties
hwdb/70-mouse.hwdb: 104 match groups, 119 matches, 123 properties
hwdb/70-pointingstick.hwdb: 8 match groups, 30 matches, 11 properties
hwdb/70-touchpad.hwdb: 6 match groups, 9 matches, 6 properties
parse_hwdb: bail with an error if no matches or groups are detected
pyparsing sometimes changes behaviour and stops giving matches. This should
allow us to detect such scenario. With this change, parse_hwdb fails with
pyparsing 2.4 on F31.
This change is only about the source tree. We have tmpfiles.d/, modprobe.d/,
sysctl.d/, and sysusers.d/, but for historical reasons, rules/ didn't fit this
pattern. We also *install* it as rules.d/. Let's rename to be consistent.
Hans de Goede [Mon, 23 Sep 2019 09:00:50 +0000 (11:00 +0200)]
hwdb: Update Primebook C11B sensor entry to also work with older BIOS versions
Older Primebook C11B BIOS versions use "Primebook C11B" as product name
instead of "PRIMEBOOK C11B", update the Primebook C11B 60-sensor.hwdb entries
to match on both spellings.
For executables which take a verb, we should list the verbs first, and
then options which modify those verbs second. The general layout of
the man page is from general description to specific details, usually
Overview, Commands, Options, Return Value, Examples, References.
test/TEST-31-DEVICE-ENUMERATION: do not use -x to avoid grep loop
https://github.com/systemd/systemd/pull/13746#issuecomment-539410752:
> [grep] now matches the grep command itself, as it's logged into journal as well, thanks to set -x.
Also, use journalctl --grep and -t to make things a bit quicker.
test: add function to reduce copied setup boilerplate
Many tests were also masking systemd-machined.service. But machined
should only start when activated, so having it not masked shouldn't be
noticable. TEST-25-IMPORT needs it.
test: drop redirection to tty in integration tests
I *think* this was originally added to make it easier to see what was happening
in tests. Later we added the functionality to print the journal on failure, so
this redirection has stopped being useful.
In https://github.com/systemd/systemd/pull/13719#issuecomment-539292650
@filbranden shows that grep tries to write to stdout and fails. In general,
we should not assume that writing to the console it always possible. We have
special code to handle this in pid1 after all:
Tim Teichmann [Sat, 5 Oct 2019 13:52:37 +0000 (15:52 +0200)]
Add missing license file and information for tools/chromeos/gen_autosuspend_rules.py (#13729)
The license file for the python script that was commited with b61d777abeecd8b6c76035e11899aae210633534 was missing. The license was copied from https://chromium.googlesource.com/chromiumos/platform2/+/master/LICENSE.
tty-ask-pwd-agent: treat SIGINT as a request to exit immediately
Unlike SIGTERM, SIGINT is now treated as a request to exit as soon as
possible. IOW, if SIGINT is received, the agent wont process all remaining
passwords before exiting.
This allows a more comprehensive behavior when C-c is pressed and when the
agent is spawned by systemctl.
Before that patch, pressing C-c killed systemctl but left the agent waiting
for a password since SIGINT was blocked. The result was pretty clumsy.
Semmle Security Reports report:
> The problem occurs on the way realloc is being used. When a size
> bigger than the chunk that wants to be reallocated is passed, realloc
> try to malloc a bigger size, however in the case that malloc fails
> (for example, by forcing a big allocation) realloc will return NULL.
>
> According to the man page:
> "The realloc() function returns a pointer to the newly allocated
> memory, which is suitably aligned for any built-in type and may be
> different from ptr, or NULL if the request fails. If size was
> equal to 0, either NULL or a pointer suitable to be passed to free()
> is returned. If realloc() fails, the original block is left
> untouched; it is not freed or moved."
>
> The problem occurs when the memory ptr passed to the first argument of
> realloc is the same as the one used for the result, for example in
> this case:
>
> dmesg = realloc(dmesg, dmesg_size + strlen(pe->dirent.d_name) +
> strlen(":\n") + pe->content_size + 1);
>
> https://lgtm.com/projects/g/systemd/systemd/snapshot/f8bcb81955f9e93a4787627e28f43fffb2a84836/files/src/pstore/pstore.c?sort=name&dir=A
> SC&mode=heatmap#L300
>
> If the malloc inside that realloc fails, then the original memory
> chunk will never be free but since realloc will return NULL, the
> pointer to that memory chunk will be lost and a memory leak will
> occur.
>
> In case you are curious, this is the query we used to find this problem:
> https://lgtm.com/query/8650323308193591473/
Let's use a more standard pattern: allocate memory using greedy_realloc, and
instead of freeing it when we wrote out a chunk, let's just move the cursor
back to the beginning and reuse the memory we allocated previously.
If we fail to allocate the memory for dmesg contents, don't write the dmesg
entry, but let's still process the files to move them out of pstore.
Ryan Attard [Fri, 4 Oct 2019 12:52:49 +0000 (07:52 -0500)]
ata_id: Add check for fixed format sense codes (#13654)
Original revisions of the SAT (SCSI-ATA Translation) specification
required that all sense data be reported in Descriptor Format (72h).
Later revisions specifcally allow and account for sense data being
reported in Fixed Format (70h).
The current code checks for a Descriptor Format sense structure (0x72),
then looks specifically at the first byte of the first descriptor for the
ATA specific code 0x9, cross referencing it with the first byte which is
just a length field 0x0c (as a sanity check).
In the Fixed Format case(0x70), we can fall back to using the top-level
SCSI Sense data for the Additional Sense code (0x0) and then the
Additional Sense Code Qualifier (0x1d),
That identifies that the sense data is of the format associated with:
`ATA PASS THROUGH INFORMATION AVAILABLE`.
This fallback mechanism retains support for SATLs compliant with ANSI
INCITS 431-2007, and enables support for Fixed Format Sense data
enabled by SATLs with later revisions.
Glad to do so. This patch allows ata_id to export attributes correctly. I believe that any drive can potentially return information in this format on any SATL using the libata-scsi (the Linux builtin SATL), but in this particular case, it appears it is the SATL itself. Attaching the disk to the AHCI controller changes the behavior impacted here. (Not entirely surprisingly, SATLs are are pretty inconsistent).
Test:
This case specifically is an LSI SATL. I'll illustrate that without the patch, ata_id does not return
any output for a valid SATA drive but after the patch does.
1. Verify the device is ATA, by looking at the vpd page specific to ATA drives
```
root@machine:~# sg_vpd -p ai /dev/sdn
ATA information VPD page:
SAT Vendor identification: LSI
SAT Product identification: LSI SATL
SAT Product revision level: 0008
Device signature indicates SATA transport
ATA command IDENTIFY DEVICE response summary:
model: HGST HUH728080ALE604
serial number: ZZZZH3VX
firmware revision: A4GNW7J0
```
2. Look at what udev thinks of the disk, it says ID_BUS=scsi
ATA information says ID_MODEL should be HGST_HUH728080ALE604
udev says it is HGST_HUH728080AL (Missing E604, 4 bytes), and no ATA attributes are
populated.
If I install the same disk into a machine using an ATA driver, this behavior changes:
```
root@machine2:~# sg_vpd -p ai /dev/sdb
ATA information VPD page:
SAT Vendor identification: linux
SAT Product identification: libata
SAT Product revision level: 3.00
Device signature indicates SATA transport
ATA command IDENTIFY DEVICE response summary:
model: HGST HUH728080ALE604
serial number: ZZZZH3VX
firmware revision: A4GNW7J0
root@machine-2:~# /lib/udev/ata_id -x /dev/sdb
ID_ATA=1
ID_TYPE=disk
ID_BUS=ata
ID_MODEL=HGST_HUH728080ALE604
ID_MODEL_ENC=HGST\x20HUH728080ALE604\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
ID_REVISION=A4GNW7J0
<truncated>
```
The ChromeOS ecosystem has a large amount of testing, both automated
and manual across devices including measurement of power regressions.
It's safe to assume that any of these devices will handle USB
auto-suspend appropriately. Use the script from ChromeOS
https://chromium.googlesource.com/chromiumos/platform2/+/master/power_manager/udev/gen_autosuspend_rules.py
to generate udev rules at build time.
This script in systemd `tools/chromeos/gen_autosuspend_rules.py` should be kept
in sync with the ChromeOS version of the script.
Manually added autosuspend devices should be placed in the new
template `rules/61-autosuspend-manual.rules`
Suggested-by: Hans de Goede <hdegoede@redhat.com> Signed-off-by: Mario Limonciello <mario.limonciello@dell.com>
Zach Smith [Fri, 4 Oct 2019 01:09:35 +0000 (18:09 -0700)]
systemd-tmpfiles: rename force to append_or_force
The force field of the Item struct is used to indicate
force creation or appending in different context. This
change renames the field to append_or_force to improve
readability.
In case PID1 is going to switch to a new root and execute a new system manager
which is not systemd, we should restore the original environment as the new
manager might expect some variables to be set by default (more specifically
$HOME).
Dan Streetman [Mon, 16 Sep 2019 16:34:55 +0000 (12:34 -0400)]
test: add temporarily blacklisted tests
This temporarily blacklists some tests when run under Ubuntu CI.
This is the upstream side of the Debian 'upstream' test MR:
https://salsa.debian.org/systemd-team/systemd/merge_requests/52
The tests blacklisted here should only be temporarily blacklisted
until they can be fixed; the intention is that these blacklist files
will be added and removed over time while debugging/fixing flaky
and/or regressed tests, without causing test failure noise for other
PRs.
Chris Down [Thu, 3 Oct 2019 12:21:29 +0000 (13:21 +0100)]
cgroup: analyze: Report memory configurations that deviate from systemd
This is the most basic consumer of the new systemd-vs-kernel checker,
both acting as a reasonable standalone exerciser of the code, and also
as a way for easy inspection of deviations from systemd internal state.
Chris Down [Mon, 30 Sep 2019 15:13:32 +0000 (16:13 +0100)]
cgroup: Allow checking systemd-internal limits against the kernel
We currently don't have any mitigations against another privileged user
on the system messing with the cgroup hierarchy, bringing the system out
of line with what we've set in systemd. We also don't have any real way
to surface this to the user (we do have logs, but you have to know to
look in the first place).
There are a few possible solutions:
1. Maintaining our own cgroup tree with the new fsopen API and having a
read-only copy for everyone else. However, there are some
complications on this front, and this may be infeasible in some
environments. I'd rate this as a longer term effort that's tangential
to this patch.
2. Actively checking for changes with {fa,i}notify and changing them
back afterwards to match our configuration again. This is also
possible, but it's also good to have a way to do passive monitoring
of the situation without taking hard action. Also, currently daemons
like senpai do actually need to modify the tree behind systemd's
back (although hopefully this should be more integrated soon).
This patch implements another option, where one can, on demand, monitor
deviations in cgroup memory configuration from systemd's internal state.
Currently the only consumer is `systemd-analyze dump`, but the interface
is generic enough that it can also be exposed elsewhere later (for
example, over D-Bus).
Currently only memory limit style properties are supported, but later I
also plan to expand this out to other properties that systemd should
have ultimate control over.
hwdb: Add trackpoint rules for Lenovo Thinkpad 70, 80, 90
Extend the existing rules to match the Thinkpad models for the
previous 3 generations. It will work if a Synaptic Trackpoint is
built into the notebook. It will not work for Elantech trackpoints.
Dan Streetman [Sun, 29 Sep 2019 21:16:55 +0000 (17:16 -0400)]
src/core/automount: use DirectoryMode when calling mkdir -p
mkdir -p is called both when setting up the autofs mount, as well
as after being notified that the real mount unit should be called.
However the first mkdir -p is hardcoded with 0555, while the second
uses the value specified to DirectoryMode in the automount unit; the
second mkdir -p is only needed when called from coldplug, so under
normal operation the dirs are incorrectly created with mode 0555.
This replaces the hardcoded 0555 mode with the value of DirectoryMode.
sd-dhcp-client: merge client_send_release() into sd_dhcp_client_send_release()
The public function and the implementation were split into two for
no particular reason.
We would assert() on the internal state of the client. This should not be done
in a function that is directly called from a public function. (I.e., we should
not crash if the public function is called at the wrong time.)
assert() is changed to assert_return().
And before anyone asks: I put the assert_returns() *above* the internal
variables on purpose. This makes it easier to see that the assert_returns()
are about the state that is passed in, and if they are not satisfied, the
function returns immediately. The compiler doesn't care either way, so
the ordering that is clearest to the reader should be chosen.
It seems that other parts of link_stop_clients() should be skipped
when restarting, but I don't know enough about those other clients to have
an opinion if it is better to stop&start them on restart or not.
Anyway, that can be done in later patches now that the support for restarts
is there.
The code supports SIGTERM and SIGINT to termiante the process. It would
be possible to reporpose one of those signals for the restart operation,
but I think it's better to use a completely different signal to avoid
misunderstandings.
core: rework how logging level is calculated for kill operations
Setting the log level based on the signal made sense when signals that
were used were fixed. Since we allow signals to be configured, it doesn't
make sense to log at notice level about e.g. a restart or stop operation
just because the signal used is different.
This avoids messages like:
six.service: Killing process 210356 (sleep) with signal SIGINT.
core: add support for RestartKillSignal= to override signal used for restart jobs
v2:
- if RestartKillSignal= is not specified, fall back to KillSignal=. This is necessary
to preserve backwards compatibility (and keep KillSignal= generally useful).
nspawn: rename UNIFIED_CGROUP_HIERARCHY to SYSTEMD_NSPAWN_UNIFIED_HIERARCHY
We should never have used an unprefixed environment variable name.
All other systemd-nspawn variables have the "SYSTEMD_NSPAWN_" prefix,
and all other systemd variables have the "SYSTEMD_" prefix.
The new variable name takes precedence, but we fall back to checking the
old one. If only the old one is found, a warning is emitted.
In addition, SYSTEMD_NSPAWN_UNIFIED_HIERARCHY="" is accepted as an override
to avoid looking for the old variable name.
We have a variable with the same name ($UNIFIED_CGROUP_HIERARCHY) in tests,
which governs both systemd-nspawn and qemu behaviour. It is not renamed.
nspawn: consistenly fail if parsing the environment fails
We would parse the environment twice (to re-apply settings after reading
config from disk), but we would not check the return code first time.
This means that for some settings we would ignore invalid values, while
for others, we'd fail at some point.
Let's just consistently fail. Those environment variables define important
aspects of behaviour, and it is better for the user if we ignore invalid
values. (Unknown settings are still ignored, so forward compatibility is
maintained.)
Jay Strict [Thu, 26 Sep 2019 13:54:29 +0000 (15:54 +0200)]
cryptsetup: bump minimum libcryptsetup version to v2.0.1
libcryptsetup v2.0.1 introduced new API calls, supporting 64 bit wide
integers for `keyfile_offset`. This change invokes the new function
call, gets rid of the warning that was added in #7689, and removes
redundant #ifdefery and constant definitions.
See https://gitlab.com/cryptsetup/cryptsetup/issues/359.
Chris Down [Mon, 30 Sep 2019 17:36:13 +0000 (18:36 +0100)]
cgroup: Mark memory protections as explicitly set in transient units
A later version of the DefaultMemory{Low,Min} patch changed these to
require explicitly setting memory_foo_set, but we only set that in
load-fragment, not dbus-cgroup.
Without these, we may fall back to either DefaultMemoryFoo or
CGROUP_LIMIT_MIN when we really shouldn't.
Chris Down [Mon, 30 Sep 2019 17:25:09 +0000 (18:25 +0100)]
cgroup: Respect DefaultMemoryMin when setting memory.min
This is an oversight from https://github.com/systemd/systemd/pull/12332.
Sadly the tests didn't catch it since it requires a real cgroup
hierarchy to see, and it wasn't seen in prod since we're only currently
using DefaultMemoryLow, not DefaultMemoryMin. :-(
Chris Down [Wed, 25 Sep 2019 16:09:38 +0000 (17:09 +0100)]
util-lib: Don't propagate EACCES from find_binary PATH lookup to caller
On one of my test machines, test-path-util was failing because the
find_binary("xxxx-xxxx") was returning -EACCES instead of -ENOENT. This
happens because the PATH entry on that host contains a directory which
the user in question doesn't have access to. Typically applications
ignore permission errors when searching through PATH, for example in
bash:
$ whoami
cdown
$ PATH=/root:/bin type sh
sh is /bin/sh
This behaviour is present on zsh and other shells as well, though. This
patch brings our PATH search behaviour closer to other major Unix tools.
projects / thirdparty / systemd.git / log
