Frantisek Sumsal [Tue, 26 Dec 2023 18:39:12 +0000 (19:39 +0100)]
coccinelle: dial back warnings about performance
Turns out I _really_ underestimated the impact of
--include-headers-for-types, as it significantly reduces both runtime
and storage penalties. For example, on my machine the runtime of
uncached run goes down from ~15 minutes to ~2 minutes, and similarly the
total storage needed by the cache goes from ~15 GiB down to ~3 GiB.
Frantisek Sumsal [Tue, 26 Dec 2023 17:55:31 +0000 (18:55 +0100)]
coccinelle: help Coccinelle with some more complex macros
Drop the original macro file, since it's not needed anymore thanks to
resolving includes properly, but introduce a similar file -
parsing_hacks.h - that helps Coccinelle in some specific corner cases.
This eliminates most of the outstanding parsing errors in source files.
The remaining ones are limitations of the parsing engine (see the FIXMEs
in pasing_hacks.h) and need further investigation.
Frantisek Sumsal [Tue, 26 Dec 2023 17:44:21 +0000 (18:44 +0100)]
coccinelle: search the system include path for header files as well
Since Coccinelle is originally a kernel tool, it doesn't search the
system include path by default for header files. Without this we're
missing a lot of types provides by stdlib (and other libraries we make
use of).
Frantisek Sumsal [Mon, 25 Dec 2023 10:28:26 +0000 (11:28 +0100)]
coccinelle: rework how we run the Coccinelle transformations
Turns out that the original way we did things was quite broken, as it
skipped a _lot_ of code. This was because we just threw everything into
one pile and tried to spatch it, but this made Coccinelle sad, like when
man page examples redefined some of our macros, causing typedef
conflicts.
For example, with a minimal reproducer that defines a cleanup macro in
two source files, Coccinelle has no issues when spatch-ing each one
separately:
$ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c
init_defs_builtins: /usr/lib64/coccinelle/standard.h
HANDLING: main.c
SPECIAL NAMES: adding _cleanup_ as a attribute with arguments
SPECIAL NAMES: adding _cleanup_free_ as a attribute
$ spatch --verbose-parsing --sp-file zz-drop-braces.cocci
logcontrol-example.c
init_defs_builtins: /usr/lib64/coccinelle/standard.h
HANDLING: logcontrol-example.c
SPECIAL NAMES: adding _cleanup_ as a attribute with arguments
But when you try to spatch both of them at once, Coccinelle starts
complaining and skipping the "bad" code:
$ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c
init_defs_builtins: /usr/lib64/coccinelle/standard.h
HANDLING: main.c logcontrol-example.c
SPECIAL NAMES: adding _cleanup_ as a attribute with arguments
SPECIAL NAMES: adding _cleanup_free_ as a attribute
remapping: _cleanup_ to an ident in macro name
ERROR-RECOV: found sync end of #define, line 44
parsing pass2: try again
ERROR-RECOV: found sync end of #define, line 44
parse error
= File "logcontrol-example.c", line 44, column 21, charpos = 1719
around = '__attribute__',
whole content = #define _cleanup_(f) __attribute__((cleanup(f)))
badcount: 2
bad: #include <systemd/sd-journal.h>
bad:
BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f)))
This was, unfortunately, hidden as it is visible only with
--verbose-parsing (or --parse-error-msg).
Another issue was how we handled includes. The original way of throwing
them into the pile of source files doesn't really work, leading up to
similar issues as above. The better way is to let Coccinelle properly
resolve all includes by telling it where to find our own include files
(basically the same thing we already do during compilation).
After fixing all this, Coccinelle now has a chance to process much more
of our code (there are still some issues in more complex macros, but
that requires further investigation). However, there's a huge downside
from all of this - doing a _proper_ code analysis is surprisingly time
and resource heavy; meaning that processing just one Coccinelle rule now
takes 15 - 30 minutes.
To make this slightly less painful, Coccinelle supports caching the
generated ASTs, which actually helps a lot - it gets the runtime of one
rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own
downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB).
However, even with the aggressive AST caching you're still looking at
~1 hour for one full Coccinelle run, which is a bit annoying, but I
guess that's the price of doing things _properly_ (but I'll definitely
look into ways of further optimizing this).
David Tardon [Mon, 25 Dec 2023 09:50:13 +0000 (10:50 +0100)]
man: use <refsect1> instead of <refsection>
DocBook document model doesn't allow mixing of <refsection> with the
numbered variants (<refsect1> etc.). Therefore, any document that
included something from standard-conf.xml was invalid. Fortunately, all
the includes are at the 1st level, hence let's just change
standard-conf.xml to use <refsect1> to fix that.
David Tardon [Sat, 23 Dec 2023 16:22:04 +0000 (17:22 +0100)]
man: make ID valid
The id attribute is of type ID, defined at
https://www.w3.org/TR/1998/REC-xml-19980210#id . It may contain only
selected non-alphanumeric characters; '@' is not among them.
Frantisek Sumsal [Sun, 24 Dec 2023 17:26:47 +0000 (18:26 +0100)]
resolve: don't add sockets to the graveyard on shutdown
Since in that case the event loop is already finished and we'd hit an
assertion:
[ 1295.993300] testsuite-75.sh[50]: + systemctl stop systemd-resolved.service
[ 1296.005152] systemd-resolved[298]: Assertion 'e->state != SD_EVENT_FINISHED' failed at src/libsystemd/sd-event/sd-event.c:1252, function sd_event_add_io(). Aborting.
Thread 1 (Thread 0x7f17d25e2940 (LWP 298)):
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x00007f17d16ac8a3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2 0x00007f17d165c668 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x00007f17d16444b8 in __GI_abort () at abort.c:79
#4 0x00007f17d2402d2d in log_assert_failed (text=<optimized out>, file=<optimized out>, line=<optimized out>, func=<optimized out>) at ../build/src/basic/log.c:968
#5 0x00007f17d240401c in log_assert_failed_return (text=text@entry=0x7f17d2533f13 "e->state != SD_EVENT_FINISHED", file=file@entry=0x7f17d25195d9 "src/libsystemd/sd-event/sd-event.c", line=line@entry=1252, func=func@entry=0x7f17d2567260 <__func__.140> "sd_event_add_io") at ../build/src/basic/log.c:987
#6 0x00007f17d24d011a in sd_event_add_io (e=0x55e5cb497270, ret=0x55e5cb4a5120, fd=fd@entry=26, events=events@entry=1, callback=callback@entry=0x55e5caff5466 <on_io_event>, userdata=0x55e5cb4a5110) at ../build/src/libsystemd/sd-event/sd-event.c:1252
#7 0x000055e5caff571c in manager_add_socket_to_graveyard (m=0x55e5cb43cf00, fd=26) at ../build/src/resolve/resolved-socket-graveyard.c:117
#8 0x000055e5cafd4253 in dns_transaction_close_connection (t=t@entry=0x55e5cb57c7d0, use_graveyard=use_graveyard@entry=true) at ../build/src/resolve/resolved-dns-transaction.c:78
#9 0x000055e5cafd8444 in dns_transaction_complete (t=t@entry=0x55e5cb57c7d0, state=state@entry=DNS_TRANSACTION_ABORTED) at ../build/src/resolve/resolved-dns-transaction.c:427
#10 0x000055e5cafc4969 in dns_scope_abort_transactions (s=s@entry=0x55e5cb4b1a70) at ../build/src/resolve/resolved-dns-scope.c:91
#11 0x000055e5cafc6aee in dns_scope_free (s=0x55e5cb4b1a70) at ../build/src/resolve/resolved-dns-scope.c:106
#12 0x000055e5cafe72d1 in link_free (l=0x55e5cb4a5160) at ../build/src/resolve/resolved-link.c:94
#13 0x000055e5cafedefc in manager_free (m=0x55e5cb43cf00) at ../build/src/resolve/resolved-manager.c:697
#14 0x000055e5caff99b6 in manager_freep (p=p@entry=0x7ffd71fab8f8) at ../build/src/resolve/resolved-manager.h:198
#15 0x000055e5caff9d66 in run (argc=argc@entry=1, argv=argv@entry=0x7ffd71faba78) at ../build/src/resolve/resolved.c:25
#16 0x000055e5caff9fe3 in main (argc=1, argv=0x7ffd71faba78) at ../build/src/resolve/resolved.c:99
djantti [Sat, 23 Dec 2023 21:08:41 +0000 (23:08 +0200)]
udev: add hwdb execution for hidraw subsystem devices
Hwdb call for hidraw subsystem is missing and AV controller devices defined in hwdb.d/70-av-production.hwdb never get the proper permissions for /dev/hidraw*. This patch implements hwdb execution also for hidraw devices.
Frantisek Sumsal [Sun, 24 Dec 2023 13:49:23 +0000 (14:49 +0100)]
busctl: avoid asserting on NULL message
Avoid passing a NULL message to sd_bus_message_is_signal(), to not trip
over an assertion:
[ 132.869436] H testsuite-82.sh[614]: + systemctl --no-block --check-inhibitors=yes soft-reboot
[ 132.967386] H systemd[1]: Created slice system-systemd\x2dcoredump.slice.
[ 133.018292] H systemd[1]: Starting inhibit.service...
[ 133.122610] H systemd[1]: Started systemd-coredump@0-665-0.service.
[ 133.163643] H systemd[1]: Started inhibit.service.
[ 133.206836] H testsuite-82.sh[614]: + exec sleep infinity
[ 133.236762] H systemd-logind[611]: The system will reboot now!
[ 135.891607] H systemd-coredump[667]: [🡕] Process 663 (busctl) of user 0 dumped core.
Frantisek Sumsal [Sun, 24 Dec 2023 11:53:53 +0000 (12:53 +0100)]
test: flush the socket once the triggered unit exits
Since the triggered unit intentionally fails without consuming any data
from the socket, we'd try to trigger it again and again, and we might
try to check the unit state in one of the "in-between" states, failing
the test:
[ 165.271698] H testsuite-07.sh[1032]: + systemctl start badbin_assert.socket
[ 165.977637] H testsuite-07.sh[1032]: + socat - ABSTRACT-CONNECT:badbin_assert.socket
[ 165.983787] H systemd[1]: Cannot find unit for notify message of PID 1039, ignoring.
[ 166.817187] H testsuite-07.sh[1032]: + timeout 10 sh -c 'while systemctl is-active badbin_assert.service; do sleep .5; done'
[ 167.049218] H testsuite-07.sh[1065]: active
[ 167.146854] H systemd[1]: Listening on badbin_assert.socket.
[ 167.163473] H systemd[1]: badbin_assert.socket: Incoming traffic
[ 167.542626] H systemd[1]: Cannot find unit for notify message of PID 1065, ignoring.
[ 167.543437] H (badbin)[1062]: badbin_assert.service: Failed to execute /tmp/badbin: Exec format error
[ 167.548346] H systemd[1]: badbin_assert.service: Main process exited, code=exited, status=203/EXEC
[ 167.549482] H systemd[1]: badbin_assert.service: Failed with result 'exit-code'.
[ 167.561537] H systemd[1]: badbin_assert.socket: Incoming traffic
[ 167.933390] H systemd[1]: Started badbin_assert.service.
[ 167.950489] H (badbin)[1070]: badbin_assert.service: Failed to execute /tmp/badbin: Exec format error
[ 167.956318] H systemd[1]: badbin_assert.service: Main process exited, code=exited, status=203/EXEC
[ 167.957173] H systemd[1]: badbin_assert.service: Failed with result 'exit-code'.
[ 167.974609] H systemd[1]: badbin_assert.socket: Incoming traffic
[ 168.042838] H testsuite-07.sh[1072]: failed
[ 168.094431] H testsuite-07.sh[1075]: ++ systemctl show -P ExecMainStatus badbin_assert.service
[ 168.704022] H systemd[1]: Started badbin_assert.service.
[ 168.778680] H (badbin)[1074]: badbin_assert.service: Failed to execute /tmp/badbin: Exec format error
[ 168.826881] H systemd[1]: badbin_assert.service: Main process exited, code=exited, status=203/EXEC
[ 168.833825] H systemd[1]: badbin_assert.service: Failed with result 'exit-code'.
[ 168.923931] H testsuite-07.sh[1032]: + [[ 0 == 203 ]]
[ 168.951492] H systemd[1]: Cannot find unit for notify message of PID 1075, ignoring.
[ 168.999862] H testsuite-07.sh[615]: + echo 'Subtest /usr/lib/systemd/tests/testdata/units/testsuite-07.issue-30412.sh failed'
[ 168.999862] H testsuite-07.sh[615]: Subtest /usr/lib/systemd/tests/testdata/units/testsuite-07.issue-30412.sh failed
Luca Boccassi [Sat, 23 Dec 2023 08:56:31 +0000 (09:56 +0100)]
meson: check for pefile dependency before enabling ukify
ukify (and all the tests, including the autogenerated check-version-ukify)
does not work unless pefile is available, so track it as a dependency
in meson to avoid unit test failures later
Yu Watanabe [Fri, 17 Nov 2023 19:18:44 +0000 (04:18 +0900)]
log: make assert_return() critical when -Dmode=developer
Triggering assert_return() should be a bug in general, and we should
really fix that. But, previously, it is hard to notice such bug, as
it was not critical.
This is for making CI or our testing environment fail if we unexpectedly
trigger assert_return(). So, hopefully we can easily find such bugs.
Yu Watanabe [Sat, 23 Dec 2023 16:49:57 +0000 (01:49 +0900)]
test: make assert_return() critical by default on fuzzer and unit tests
Several test cases intentionally trigger assert_return(). So, to avoid
the entire test fails, this introduces several macros that tentatively
make assert_return() not critical.