Lasse Collin [Wed, 10 Dec 2025 17:11:27 +0000 (19:11 +0200)]
tuklib_integer/CMake: Don't run unaligned access checks unnecessarily
If the cache variable TUKLIB_FAST_UNALIGNED_ACCESS is already set,
the autodetection result isn't needed because the option() command
does nothing when the cache variable is already set.
This is largely white space change to indent the if...endif block.
Lasse Collin [Wed, 10 Dec 2025 14:49:55 +0000 (16:49 +0200)]
tuklib_integer: Use CMAKE_C_COMPILER_ARCHITECTURE_ID when available
CMake >= 4.1 sets CMAKE_<LANG>_COMPILER_ARCHITECTURE_ID on many
platforms. The list of possible values are documented. Use this
variable when available. On older CMake versions CMAKE_SYSTEM_PROCESSOR
is still used, thus the regexes have to include values like ^amd64 still.
With old CMake versions, checking CMAKE_C_COMPILER_ARCHITECTURE_ID
is somewhat useful with MSVC because CMAKE_SYSTEM_PROCESSOR might
not match the target architecture.
Lasse Collin [Wed, 10 Dec 2025 14:49:55 +0000 (16:49 +0200)]
tuklib_integer: Autodetect when -mstrict-align is used with GCC on ARM64
On ARM64, support for fast unaligned memory access was autodetected by
checking if __ARM_FEATURE_UNALIGNED is defined. However, at least GCC
versions up to 15.2.0 define the macro even when -mstrict-align has
been specified. Thus, autodetection with GCC doesn't work correctly,
and binaries built using -mstrict-align can be much slower than they
need to be, unless the user also passes --disable-unaligned-access
to configure or -DTUKLIB_FAST_UNALIGNED_ACCESS=OFF to cmake.
Lasse Collin [Tue, 9 Dec 2025 10:55:45 +0000 (12:55 +0200)]
xz: Never translate "Failed to enable the sandbox"
This was already the case in practice because I had forgotten to list
src/xz/sandbox.c in po/POTFILES.in. However, it seems better to never
translate this particular error message. It should almost never occur
and if it does, an untranslated message is should make it easier to
find bug reports about it.
Lasse Collin [Tue, 9 Dec 2025 10:13:36 +0000 (12:13 +0200)]
CMake: Autodetect unaligned access support on LoongArch
This still relies on CMAKE_SYSTEM_PROCESSOR. CMake 4.1 added more
CMAKE_<LANG>_COMPILER_ARCHITECTURE_ID values to detect the arch in
a more defined manner, but 4.1 is too new to require for now.
Thanks-to: Li Chenggang <lichenggang@deepin.org> Closes: https://github.com/tukaani-project/xz/pull/186
Lasse Collin [Tue, 9 Dec 2025 10:13:36 +0000 (12:13 +0200)]
Autotools: Autodetect unaligned access support on LoongArch
According to [1] sections 7.4, 8.1, and 8.2, desktop and server
processors support fast unaligned access, but embedded systems likely
don't.
It's important that TUKLIB_FAST_UNALIGNED_ACCESS isn't defined when
-mstrict-align is in use because it will result in slower binaries
even if running on a processor that supports fast unaligned access.
It's because compilers will translate multibyte memcpy() to multiple
byte-by-byte instructions instead of wider loads and stores. The
compression times from [2] show this well:
Unaligned access CFLAGS Compression time
enabled -O2 -mno-strict-align 66.1 s
disabled -O2 -mno-strict-align 79.5 s
disabled -O2 -mstrict-align 79.9 s
enabled -O2 -mstrict-align 129.1 s
There currently (GCC 15.2) is no preprocessor macro on LoongArch
to detect if -mstrict-align or -mno-strict-align is in effect (the
default is -mno-strict-align). Use heuristics to detect which of the
flags is in effect.
Lasse Collin [Sun, 2 Nov 2025 10:17:50 +0000 (12:17 +0200)]
liblzma: Silence two Coverity warnings
lzma_lzma_decoder_memusage() returns UINT64_MAX if lc/lp/pb aren't
valid. alone_decoder.c and lzip_decoder.c didn't check the return
value because in both it is known that lc/lp/pb are valid. Make them
call the _nocheck() variant instead which skips the validation (it
already existed for LZMA2's internal use).
Lasse Collin [Sun, 2 Nov 2025 10:17:50 +0000 (12:17 +0200)]
liblzma: Fix a harmless read of shared variable without mutex
The partial_update_mode enumeration had three states, _DISABLED,
_START, and _ENABLED. Main thread changed it from _DISABLED to _START
while holding a mutex. Once set to _START, worker thread changed it
to _ENABLED without a mutex. Later main thread read it without a mutex,
so it could see either _START or _ENABLED. However, it made no
difference because the main thread checked for != _DISABLED, so
it didn't matter if it saw _START or _ENABLED.
Nevertheless, such things must not be done. It's clear it was a mistake
because there were two comments that directly contradicted each
other about how the variable was accessed.
Split the enumeration into two booleans:
- partial_update_enabled: A worker thread locks the mutex to read
this variable and the main thread locks the mutex to change the
value. Because only the main thread modifies the variable, the
main thread can read the value without locking the mutex.
This variable replaces the _DISABLED -> _START transition.
- partial_update_started is for worker thread's internal use and thus
needs no mutex. This replaces the _START -> _ENABLED transition.
Fixes: Coverity CID 456025 Fixes: bd93b776c1bd ("liblzma: Fix a deadlock in threaded decoder.")
Lasse Collin [Tue, 25 Nov 2025 11:55:35 +0000 (13:55 +0200)]
Change the sorting order in THANKS
In short, sort the names with this command (-k1,1 isn't needed because
the lines with names start with " -"):
LC_ALL=en_US.UTF-8 sort -k2,2 -k3,3 -k4,4 -k5,5
When THANKS was created, I wrote the names as "First Last" and attempted
to keep them sorted by last name / surname / family name. This works
with many names in THANKS, but it becomes complicated with names that
don't fit that pattern. For example, names that are written as
"Last First" can be manually sorted by family name, but only if one
knows which part of the name is the family name.[*] And of course,
the concept of first/last name doesn't apply to all names.
[*] xz had a co-maintainer who could help me with such names,
but fortunately he isn't working on the project anymore.
Adding the names in chronological order could have worked too, although
if something is contributed by multiple people, one would still have to
decide how to sort the names within the batch. Another downside would
be that if THANKS is updated in more than one work-in-progress branch,
merge conflicts would occur more often.
Don't attempt to sort by last name. Let's be happy that people tend to
provide names that can be expressed in a reasonable number of printable
Unicode characters. In practice, people have been even nicer: if the
native language doesn't use a Latin script alphabet, people often provide
a transliterated name (only or in addition to the original spelling),
which is very much appreciated by those who don't know the native script.
Treat the names as opaque strings or space-separated strings for sorting
purposes. This means that most names will now be sorted by first name.
There still are many choices how to sort:
(1) LC_ALL=en_US.UTF-8 sort
The project is in English, so this may sound like a logical choice.
However, spaces have a lower weight than letters, which results in
this order:
- A Ba
- Ab C
- A Bc
- A Bd
(2) LC_ALL=en_US.UTF-8 sort -k2,2
This first sorts by the first word and then by the rest of the
string. It's -k2,2 instead of -k1,1 to skip the leading dash.
- A Ba
- A Bc
- A Bd
- Ab C
I like this more than (1). One could add -k3,3 -k4,4 -k5,5 ... too.
With current THANKS it makes no difference but it might some day.
NOTE: The ordering in en_US.UTF-8 can differ between libc versions
and operating systems. Luckily it's not a big deal in THANKS.
(3) LC_ALL=en_US.UTF-8 sort -f -k2,2
Passing -f (--ignore-case) to sort affects sorting of single-byte
characters but not multibyte characters (GNU coreutils 9.9):
No -f With -f LC_ALL=C
Aa A.A A.A
A.A Aa Aa
Ää Ää Ä.Ä
Ä.Ä Ä.Ä Ää
In GNU coreutils, the THANKS file is sorted using "sort -f -k1,1".
There is also a basic check that the en_US.UTF-8 locale is
behaving as expected.
(4) LC_ALL=C sort
This sorts by byte order which in UTF-8 is the same as Unicode
code point order. With the strings in (1) and (2), this produces
the same result as in (2). The difference in (3) can be seen above.
The results differ from en_US.UTF-8 when a name component starts
with a lower case ASCII letter (like "von" or "de"). Worse, any
non-ASCII characters sort after ASCII chars. These properties might
look weird in English language text, although it's good to remember
that en_US.UTF-8 sorting can appear weird too if one's native
language isn't English.
The choice between (2) and (4) was difficult but I went with (2).
Lasse Collin [Sun, 23 Nov 2025 18:13:49 +0000 (20:13 +0200)]
Landlock: Workaround a bug in RHEL 9 kernel
If one runs xz 5.8.0 or 5.8.1 from some other distribution in a container
on RHEL 9, xz will fail with the message "Failed to enable the sandbox".
RHEL 9 kernel since 5.14.0-603.el9 (2025-07-30) claims to support
Landlock ABI version 6, but it lacks support for LANDLOCK_SCOPE_SIGNAL.
The issue is still present in 5.14.0-643.el9 (2025-11-22). Red Hat is
aware of the issue, but I don't know when it will be fixed.
The sandbox is meant to be transparent to users, thus there isn't and
won't be a command line option to disable it. Instead, add a workaround
to keep xz working on the buggy RHEL 9 kernels.
Lasse Collin [Mon, 3 Nov 2025 10:41:31 +0000 (12:41 +0200)]
mythread.h: Fix the build on Emscripten when threading is disabled
To make a non-threaded liblzma-only build work with WASI SDK, <signal.h>
and mythread_sigmask() were omitted from mythread.h in the commit 81db3b889830. This broke non-threaded full build with Emscripten because
src/xz/signals.c needs mythread_sigmask() (liblzma-only build was fine).
If __wasm__ is defined, omit <signal.h> and mythread_sigmask() in
non-threaded builds only when __EMSCRIPTEN__ isn't defined.
Reported-by: Marcus Tillmanns Thanks-to: ChanTsune Fixes: https://github.com/tukaani-project/xz/issues/161 Fixes: 81db3b889830 ("mythread.h: Disable signal functions in builds targeting Wasm + WASI.")
xz: Look at resource limits when determining the default memlimit
When no memory usage limits have been set by the user, the default
for multithreaded mode has been 1/4 of total RAM. If this limit is
too high and memory allocation fails, liblzma (and xz) fail. Perhaps
liblzma should handle it better by reducing the number of threads
and continuing with the amount of memory it can allocate, but currently
that isn't the case.
If resource limits were set to about 1/4 of RAM or lower, then xz
could fail for the above reason. This commit makes xz look at
RLIMIT_DATA, RLIMIT_AS, and RLIMIT_VMEM when they are available,
and set the limit 64 MiB below the lowest of those limits. This is
more or less a hack just like the 1/4-of-RAM method is, but this is
simple and quick to implement.
On Linux, there are other limits like cgroup v2 memory.max which
can still make xz fail. The same is likely possible with FreeBSD's
rctl(8).
Co-authored-by: Lasse Collin <lasse.collin@tukaani.org> Thanks-to: Fangrui Song Fixes: https://github.com/tukaani-project/xz/issues/195 Closes: https://github.com/tukaani-project/xz/pull/196
Lasse Collin [Wed, 1 Oct 2025 09:46:26 +0000 (12:46 +0300)]
CI: Remove windows-2019 (which had VS 2019)
GitHub has removed the runner image.
A breakage with CLMUL CRC code occurred with VS 2019 but not 2022,
see b5a5d9e3f702. MS supports VS 2019 for a few more years, so it's
unfortunate that it can no longer be tested on GitHub.
Lasse Collin [Mon, 29 Sep 2025 15:37:19 +0000 (18:37 +0300)]
liblzma: Document that lzma_allocator.free(opaque, NULL) is possible
It feels better to fix the docs than change the code because this
way newly-written applications will be forced to be compatible with
the lzma_allocator behavior of old liblzma versions. It can matter
if someone builds the application against an older liblzma version.
Lasse Collin [Tue, 13 May 2025 17:42:26 +0000 (20:42 +0300)]
Scripts: Add shellcheck directives to silence warnings
Set also shell because the xz*.in files start with '#!@POSIX_SHELL@'.
SC1003 and SC2016 are only info messages, not warnings. Several other
shellcheck info messages remain. They are safe to ignore, but I didn't
want to disable them now.
Lasse Collin [Fri, 23 May 2025 10:09:14 +0000 (13:09 +0300)]
CI: Add timeout-minutes
Sometimes the VM workflows (like FreeBSD VM on Ubuntu) get stuck
and the default timeout is six hours. While at it, set a sensible
timeout for all workflows.
Lasse Collin [Sat, 3 May 2025 09:37:28 +0000 (12:37 +0300)]
Tests: Silence a warning from GCC 15.1
It was (probably) intentionally without the null terminator, but the test
works with null terminator too (the test still fails with xz <= 5.0.3),
so simply omit one character to silence the warning.
tests/test_bcj_exact_size.c:30:32: error: initializer-string for array of ‘unsigned char’ truncates NUL terminator but destination lacks ‘nonstring’ attribute (17 chars into 16 available) [-Werror=unterminated-string-initialization]
30 | const uint8_t in[16] = "0123456789ABCDEF";
| ^~~~~~~~~~~~~~~~~~
Fixes: d8db706acb83 ("liblzma: Fix possibility of incorrect LZMA_BUF_ERROR.") Fixes: https://github.com/tukaani-project/xz/issues/176
xz, xzdec: Capsicum sandbox: Fix incorrect use of cap_rights_clear()
cap_rights_clear() with no additional arguments acts as a no-op, so
instead of removing all capability rights from STDIN_FILENO, the same
rights were allowed for STDIN_FILENO as were allowed for src_fd.
Fixes: a0eecc235d3b ("xz: Make Capsicum sandbox more strict with stdin and stdout.")
(The commit message says "stdout". It should have said "stderr".)
Lasse Collin [Tue, 22 Apr 2025 18:12:50 +0000 (21:12 +0300)]
CI: Valgrind: Improve the skipping of traced child processes
Use --trace-children-skip instead of --trace-children-skip-by-arg
so that the skipping is only done based on the executable names.
(--trace-children-skip-by-arg can match other args than argv[0].)
Update the list of executables to skip to match what the scripts run.
Do not skip bash or sh. If Valgrind didn't trace the shell, then the
xz and xzdec programs run by the shell wouldn't be analyzed either.
Fixes: 7e99856f66c0 ("CI: Speed up Valgrind job by using --trace-children-skip-by-arg=...")
Lasse Collin [Tue, 22 Apr 2025 18:12:50 +0000 (21:12 +0300)]
CI: Valgrind: Test with static liblzma
If shared liblzma is built, tests/test_* and src/xz/xz are wrapper
scripts created by Libtool. The wrappers set library search path
so that the freshly-built shared library is found.
With a static liblzma, no wrapper scripts are needed, and Libtool
places the real executables to the aforementioned locations. This
speeds up the tests under Valgrind dramatically.
Lasse Collin [Tue, 22 Apr 2025 13:21:50 +0000 (16:21 +0300)]
Doxygen: Set HAVE_DOT = NO
Debian and Ubuntu have a patch that changes the upstream default to
HAVE_DOT = YES. Undo it to have more consistent results across distros.
This was noticed in Ubuntu CI runner where "doxygen" tried to run "dot"
but that failed due to "dot" not being installed. "doxygen" still
finished with exit status 0 until the commit that turned warnings to
errors with WARN_AS_ERROR = FAIL_ON_WARNINGS.
Lasse Collin [Tue, 22 Apr 2025 13:21:50 +0000 (16:21 +0300)]
CI: Revise MSYS2
Re-enable CLANG64 environment. Add CLANGARM64. Don't add MINGW64
to slightly reduce the number of runner VMs needed.
Install the required packages using the setup-msys2 action instead
of running the commands separately.
Test Autotools and CMake in the same job to reduce the number of VMs.
This doesn't slow it down too much because the msys2-setup step is
needed by both. However, do only the full builds on ARM64 because
those runners seem to be slower.
Test fewer build configurations. The point of testing on MSYS2 is to
catch Windows-related issues. It should be enough that the more unusual
build configurations are tested in ci.yml.
Run the build commands directly instead of using ci_build.bash. This
makes it easier to see what commands are run even if it is a little
more verbose now.
Run the workflow automatically when commit are pushed to master.
With the fewer build variants it's not too slow.
Lasse Collin [Mon, 21 Apr 2025 09:23:37 +0000 (12:23 +0300)]
liblzma: Update lzma_lzip_decoder() docs about trailing data
Don't say that the .lz format allows trailing data. According to the
lzip 1.25 manual, trailing data isn't part of the file format at all.
However, tools are still expected to behave as usefully as possible
when there is trailing data.
Fix the description of lzip >= 1.20 behavior when some of the first
bytes of trailing data match the magic bytes. While the lzip 1.25 manual
recommends that none of the first four bytes in trailing data should
match the magic bytes, the default behavior of lzip 1.25 treats
trailing data as a corrupt member header only if two or three bytes
match the magic bytes; one matching byte isn't enough.
projects / thirdparty / xz.git / log
