xz, xzdec: Capsicum sandbox: Fix incorrect use of cap_rights_clear()
cap_rights_clear() with no additional arguments acts as a no-op, so
instead of removing all capability rights from STDIN_FILENO, the same
rights were allowed for STDIN_FILENO as were allowed for src_fd.
Fixes: a0eecc235d3b ("xz: Make Capsicum sandbox more strict with stdin and stdout.")
(The commit message says "stdout". It should have said "stderr".)
Lasse Collin [Thu, 3 Apr 2025 11:34:42 +0000 (14:34 +0300)]
liblzma: mt dec: Fix lack of parallelization in single-shot decoding
Single-shot decoding means calling lzma_code() by giving it the whole
input at once and enough output buffer space to store the uncompressed
data, and combining this with LZMA_FINISH and no timeout
(lzma_mt.timeout = 0). This way the file is decoded with a single
lzma_code() call if possible.
The bug prevented the decoder from starting more than one worker thread
in single-shot mode. The issue was noticed when reviewing the code;
there are no bug reports. Thus maybe few have tried this mode.
Fixes: 64b6d496dc81 ("liblzma: Threaded decoder: Always wait for output if LZMA_FINISH is used.")
(cherry picked from commit 0c80045ab82c406858d9d5bcea9f48ebc3d0a81d)
Lasse Collin [Thu, 3 Apr 2025 11:34:42 +0000 (14:34 +0300)]
liblzma: mt dec: Don't modify thr->in_size in the worker thread
Don't set thr->in_size = 0 when returning the thread to the stack of
available threads. Not only is it useless, but the main thread may
read the value in SEQ_BLOCK_THR_RUN. With valid inputs, it made
no difference if the main thread saw the original value or 0. With
invalid inputs (when worker thread stops early), thr->in_size was
no longer modified after the previous commit with the security fix
("Don't free the input buffer too early").
So while the bug appears harmless now, it's important to fix it because
the variable was being modified without proper locking. It's trivial
to fix because there is no need to change the value. Only main thread
needs to set the value in (in SEQ_BLOCK_THR_INIT) when starting a new
Block before the worker thread is activated.
Fixes: 4cce3e27f529 ("liblzma: Add threaded .xz decompressor.") Reviewed-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Thanks-to: Sam James <sam@gentoo.org>
(cherry picked from commit 8188048854e8d11071b8a50d093c74f4c030acc9)
Lasse Collin [Thu, 3 Apr 2025 11:34:42 +0000 (14:34 +0300)]
liblzma: mt dec: Don't free the input buffer too early (CVE-2025-31115)
The input buffer must be valid as long as the main thread is writing
to the worker-specific input buffer. Fix it by making the worker
thread not free the buffer on errors and not return the worker thread to
the pool. The input buffer will be freed when threads_end() is called.
With invalid input, the bug could at least result in a crash. The
effects include heap use after free and writing to an address based
on the null pointer plus an offset.
The bug has been there since the first committed version of the threaded
decoder and thus affects versions from 5.3.3alpha to 5.8.0.
As the commit message in 4cce3e27f529 says, I had made significant
changes on top of Sebastian's patch. This bug was indeed introduced
by my changes; it wasn't in Sebastian's version.
Thanks to Harri K. Koskinen for discovering and reporting this issue.
Fixes: 4cce3e27f529 ("liblzma: Add threaded .xz decompressor.") Reported-by: Harri K. Koskinen <x64nop@nannu.org> Reviewed-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Thanks-to: Sam James <sam@gentoo.org>
(cherry picked from commit d5a2ffe41bb77b918a8c96084885d4dbe4bf6480)
Lasse Collin [Thu, 3 Apr 2025 11:34:42 +0000 (14:34 +0300)]
liblzma: mt dec: Simplify by removing the THR_STOP state
The main thread can directly set THR_IDLE in threads_stop() which is
called when errors are detected. threads_stop() won't return the stopped
threads to the pool or free the memory pointed by thr->in anymore, but
it doesn't matter because the existing workers won't be reused after
an error. The resources will be cleaned up when threads_end() is
called (reinitializing the decoder always calls threads_end()).
Reviewed-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Thanks-to: Sam James <sam@gentoo.org>
(cherry picked from commit c0c835964dfaeb2513a3c0bdb642105152fe9f34)
Lasse Collin [Thu, 3 Apr 2025 11:34:42 +0000 (14:34 +0300)]
liblzma: mt dec: Fix a comment
Reviewed-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Thanks-to: Sam James <sam@gentoo.org>
(cherry picked from commit 831b55b971cf579ee16a854f177c36b20d3c6999)
Lasse Collin [Wed, 22 Jan 2025 13:03:55 +0000 (15:03 +0200)]
Windows: Disable MinGW-w64's stdio functions in size-optimized builds
This only affects builds with UCRT. With legacy MSVCRT, the replacement
functions are always enabled.
Omitting the MinGW-w64 replacements saves over 20 KiB per executable.
The downside is that --enable-small or XZ_SMALL=ON disables thousand
separator support in xz messages. If someone is OK with the slower
speed of slightly smaller builds, lack of thousand separators won't
matter.
Don't override __USE_MINGW_ANSI_STDIO if it is already defined (via
CPPFLAGS or such method).
Lasse Collin [Fri, 10 Jan 2025 11:11:40 +0000 (13:11 +0200)]
Translations: Update Serbian translation
I rewrapped a few overlong lines. Those edits aren't in the
Translation Project. Automatic wrapping in the master branch
means that these strings need to be updated soon anyway.
Lasse Collin [Sun, 5 Jan 2025 10:10:05 +0000 (12:10 +0200)]
liblzma: Always validate the first digit of a preset string
lzma_str_to_filters() may call parse_lzma12_preset() in two ways. The
call from str_to_filters() detects the string type from the first
character(s) and as a side-effect it validates the first digit of
the preset string. So this change makes no difference there.
However, the call from parse_options() doesn't pre-validate the string.
parse_lzma12_preset() will return an invalid value which is passed to
lzma_lzma_preset() which safely rejects it. The bug still affects the
the error message:
Lasse Collin [Sun, 5 Jan 2025 09:40:34 +0000 (11:40 +0200)]
xz: Fix getopt_long argument type in --filters*
Forgetting the argument (or not using = to separate the option from
the argument) resulted in lzma_str_to_filters() being called with NULL
as input string argument. The function handles it fine but xz passes
the NULL to printf() too:
$ xz --filters
xz: Error in --filters=FILTERS option:
xz: (null)
xz: ^
xz: Unexpected NULL pointer argument(s) to lzma_str_to_filters()
Now it's correct:
$ xz --filters
xz: option '--filters' requires an argument
The --filters-help option doesn't take any arguments.
Lasse Collin [Thu, 2 Jan 2025 11:35:48 +0000 (13:35 +0200)]
Windows: Make NLS require UCRT and gettext-runtime >= 0.23.1
Also remove the recently-added workaround from tuklib_gettext.h.
Requiring a new enough gettext-runtime is cleaner. I guess it's
mostly MSYS2 where xz is built with translation support, so once
MSYS2 has Gettext >= 0.23.1, this requirement shouldn't be a problem
in practice.
Lasse Collin [Mon, 30 Dec 2024 08:51:26 +0000 (10:51 +0200)]
xz man page: Describe the source file deletion in -z and -d options
The DESCRIPTION section always explained it, and the OPTIONS section
only described the differences to the default behavior. However, new
users in a hurry may skip reading DESCRIPTION. The default behavior
is a bit dangerous, thus it's good to repeat in --compress and
--decompress docs that source file is removed after successful operation.
liblzma and xz can't be compiled as a unity/jumbo build because of
redeclarations and type name reuse. The CMake documentation recommends
setting UNITY_BUILD to false in this case.
This is especially important if we're compiled as a subproject and the
consumer wants to use CMAKE_UNITY_BUILD=ON for the rest of their code
base.
Lasse Collin [Thu, 19 Dec 2024 16:31:09 +0000 (18:31 +0200)]
xzdec: Use setlocale() instead of tuklib_gettext_setlocale()
xzdec isn't translated and doesn't need libintl on Windows even
when NLS is enabled, thus libintl_setlocale() cannot interfere
with the locale settings. Thus, standard setlocale() works perfectly.
In the commit 78868b6e, the explanation in the commit message is wrong.
Lasse Collin [Thu, 19 Dec 2024 17:36:15 +0000 (19:36 +0200)]
Windows: Revert the setlocale(LC_ALL, ".UTF8") documentation
Only leave the FindFileFirstA() notes from 20dfca81, reverting
the incorrect setlocale() notes. On Windows, Gettext's <libintl.h>
overrides setlocale() with libintl_setlocale() wrapper. I hadn't
noticed this, and thus my conclusions were wrong.
Lasse Collin [Mon, 16 Dec 2024 18:06:07 +0000 (20:06 +0200)]
tuklib_mbstr_width: Change the behavior when wcwidth() is not available
If wcwidth() isn't available (Windows), previously it was assumed
that one byte == one column in the terminal. Now it is assumed that
one multibyte character == one column. This works better with UTF-8.
Languages that only use single-width characters without any combining
characters should work correctly with this.
In xz, none of po/*.po contain combining characters and only ko.po,
zh_CN.po, and zh_TW.po contain fullwidth characters. Thus, "only"
those three translations in xz are broken on Windows with the
UTF-8 code page. Broken means that column headings in xz -lvv and
(only in the master branch) strings in --long-help are misaligned,
so it's not a huge problem. I don't know if those three languages
displayed perfectly before the UTF-8 change because I hadn't tested
translations with native Windows builds before.
Lasse Collin [Wed, 18 Dec 2024 12:23:13 +0000 (14:23 +0200)]
xzdec: Use setlocale() via tuklib_gettext_setlocale()
xzdec isn't translated and didn't have locale-specific behavior
in the past. On Windows with UTF-8 in the application manifest,
setting the locale makes a difference though:
- Without any setlocale() call, non-ASCII filenames don't display
properly in Command Prompt unless one first uses "chcp 65001"
to set the console code page to UTF-8.
- setlocale(LC_ALL, "") is enough to make non-ASCII filenames
print correctly in Command Prompt without using "chcp 65001",
assuming that the non-UTF-8 code page (like 850) supports
those non-ASCII characters.
- setlocale(LC_ALL, ".UTF8") is even better because then mbrtowc() and
such functions use an UTF-8 locale instead of a legacy code page.
The tuklib_gettext_setlocale() macro takes care of this (without
enabling any translations).
Lasse Collin [Tue, 17 Dec 2024 12:59:37 +0000 (14:59 +0200)]
Windows: Use UTF-8 locale when active code page is UTF-8
XZ Utils 5.6.3 set the active code page to UTF-8 to fix CVE-2024-47611.
This wasn't paired with UCRT-specific setlocale(LC_ALL, ".UTF8"), thus
non-ASCII characters from translations became mojibake.
Lasse Collin [Sun, 15 Dec 2024 17:08:32 +0000 (19:08 +0200)]
CMake: Bump maximum policy version to 3.31
With CMake 3.31, there were a few warnings from
CMP0177 "install() DESTINATION paths are normalized".
These occurred because the install(FILES) command in
my_install_man_lang() is called with a DESTINATION path
that contains two consecutive slashes, for example,
"share/man//man1". Such a path is for the English man pages.
With translated man pages, the language code goes between
the slashes. The warning was probably triggered because the
extra slash gets removed by the normalization.
Lasse Collin [Tue, 1 Oct 2024 09:10:23 +0000 (12:10 +0300)]
Windows: Embed an application manifest in the EXE files
IMPORTANT: This includes a security fix to command line tool
argument handling.
Some toolchains embed an application manifest by default to declare
UAC-compliance. Some also declare compatibility with Vista/8/8.1/10/11
to let the app access features newer than those of Vista.
We want all the above but also two more things:
- Declare that the app is long path aware to support paths longer
than 259 characters (this may also require a registry change).
- Force the code page to UTF-8. This allows the command line tools
to access files whose names contain characters that don't exist
in the current legacy code page (except unpaired surrogates).
The UTF-8 code page also fixes security issues in command line
argument handling which can be exploited with malicious filenames.
See the new file w32_application.manifest.comments.txt.
Thanks to Orange Tsai and splitline from DEVCORE Research Team
for discovering this issue.
Thanks to Vijay Sarvepalli for reporting the issue to me.
Thanks to Kelvin Lee for testing with MSVC and helping with
the required build system fixes.
Lasse Collin [Sun, 29 Sep 2024 11:46:52 +0000 (14:46 +0300)]
Windows: Set DLL name accurately in StringFileInfo on Cygwin and MSYS2
Now the information in the "Details" tab in the file properties
dialog matches the naming convention of Cygwin and MSYS2. This
is only a cosmetic change.
Lasse Collin [Sat, 28 Sep 2024 17:09:50 +0000 (20:09 +0300)]
CMake: Add the resource files to the Cygwin and MSYS2 builds
Autotools-based build has always done this so this is for consistency.
However, the CMake build won't create the DEF file when building
for Cygwin or MSYS2 because in that context it should be useless.
(If Cygwin or MSYS2 is used to host building of normal Windows
binaries then the DEF file is still created.)
"xzdec -M123" exited with exit status 1 without printing
any messages. The "M:" entry should have been removed when
the memory usage limiter support was removed from xzdec.
Yifeng Li [Thu, 22 Aug 2024 02:18:49 +0000 (02:18 +0000)]
liblzma: Fix x86-64 movzw compatibility in range_decoder.h
Support for instruction "movzw" without suffix in "GNU as" was
added in commit [1] and stabilized in binutils 2.27, released
in August 2016. Earlier systems don't accept this instruction
without a suffix, making range_decoder.h's inline assembly
unable to build on old systems such as Ubuntu 16.04, creating
error messages like:
lzma_decoder.c: Assembler messages:
lzma_decoder.c:371: Error: no such instruction: `movzw 2(%r11),%esi'
lzma_decoder.c:373: Error: no such instruction: `movzw 4(%r11),%edi'
lzma_decoder.c:388: Error: no such instruction: `movzw 6(%r11),%edx'
lzma_decoder.c:398: Error: no such instruction: `movzw (%r11,%r14,4),%esi'
Lasse Collin [Sat, 6 Jul 2024 11:04:48 +0000 (14:04 +0300)]
xz: Remove the TODO comment about --recursive
It won't be implemented. find + xargs is more flexible, for example,
it allows compressing small files in parallel. An example for that
has been included in the xz man page since 2010.
Lasse Collin [Wed, 3 Jul 2024 17:45:48 +0000 (20:45 +0300)]
CMake: Link xz against Threads::Threads if using pthreads
The liblzma target was recently changed to link against Threads::Threads
with the PRIVATE keyword. I had forgotten that xz itself depends on
pthreads too due to pthread_sigmask(). Thus, the build broke when
building shared liblzma and pthread_sigmask() wasn't in libc.
Thanks to Peter Seiderer for the bug report.
Fixes: ac05f1b0d7cda1e7ae79775a8dfecc54601d7f1c Fixes: https://github.com/tukaani-project/xz/issues/129#issuecomment-2204522994
(cherry picked from commit b3e53122f42796aaebd767bab920cf7bedf69966)
Sam James [Fri, 28 Jun 2024 11:18:35 +0000 (14:18 +0300)]
CI: Speed up Valgrind job by using --trace-children-skip-by-arg=...
This addresses the issue I mentioned in 6c095a98fbec70b790253a663173ecdb669108c4 and speeds up the Valgrind
job a bit, because non-xz tools aren't run unnecessarily with
Valgrind by the script tests.
Lasse Collin [Tue, 25 Jun 2024 13:00:22 +0000 (16:00 +0300)]
Build: Prepend, not append, PTHREAD_CFLAGS to LIBS
It shouldn't make any difference because LIBS should be empty
at that point in configure. But prepending is the correct way
because in general the libraries being added might require other
libraries that come later on the command line.
Lasse Collin [Tue, 25 Jun 2024 11:24:29 +0000 (14:24 +0300)]
Build: Use AC_LINK_IFELSE to handle implicit function declarations
It's more robust in case the compiler allows pre-C99 implicit function
declarations. If an x86 intrinsic is missing and gets treated as
implicit function, the linking step will very probably fail. This
isn't the only way to workaround implicit function declarations but
it might be the simplest and cleanest.
The problem hasn't been observed in the wild.
There are a couple more AC_COMPILE_IFELSE uses in configure.ac.
Of these, Landlock check calls prctl() and in theory could have
the same problem. In practice it doesn't as the check program
looks for several other things too. However, it was changed to
AC_LINK_IFELSE still to look more correct.
Similarly, m4/tuklib_cpucores.m4 and m4/tuklib_physmem.m4 were
updated although they haven't given any trouble either. They
have worked all these years because those check programs rely
on specific headers and types: if headers or types are missing,
compilation will fail. Using the linker makes these checks more
similar to the ones in cmake/tuklib_*.cmake which always link.
Lasse Collin [Mon, 24 Jun 2024 20:35:59 +0000 (23:35 +0300)]
Build: Use AC_LINK_IFELSE instead of -Werror
AC_COMPILE_IFELSE needed -Werror because Clang <= 14 would merely
warn about the unsupported attribute and implicit function declaration.
Changing to AC_LINK_IFELSE handles the implicit declaration because
the symbol __crc32d is unlikely to exist in libc.
Note that the other part of the check is that #include <arm_acle.h>
must work. If the header is missing, most compilers give an error
and the linking step won't be attempted.
Avoiding -Werror makes the check more robust in case CFLAGS contains
warning flags that break -Werror anyway (but this isn't the only check
in configure.ac that has this problem). Using AC_LINK_IFELSE also makes
the check more similar to how it is done in CMakeLists.txt.
Lasse Collin [Tue, 25 Jun 2024 12:51:48 +0000 (15:51 +0300)]
CMake: Always add pthread flags into CMAKE_REQUIRED_LIBRARIES
It was weird to add CMAKE_THREAD_LIBS_INIT in CMAKE_REQUIRED_LIBRARIES
only if CLOCK_MONOTONIC is available. Alternative would be to remove
the thread libs from CMAKE_REQUIRED_LIBRARIES after the check for
pthread_condattr_setclock() but keeping the libs should be fine too.
Then it's ready in case more pthread functions were wanted some day.
Lasse Collin [Mon, 24 Jun 2024 19:41:10 +0000 (22:41 +0300)]
CMake: Fix three checks if building with -flto
In CMake, check_c_source_compiles() always links too. With
link-time optimization, unused functions may get omitted if
main() doesn't depend on them. Consider the following which
tries to check if somefunction() is available when <someheader.h>
has been included:
#include <someheader.h>
int foo(void) { return somefunction(); }
int main(void) { return 0; }
LTO may omit foo() completely because the program as a whole doesn't
need it and then the program will link even if the symbol somefunction
isn't available in libc or other library being linked in, and then
the test may pass when it shouldn't.
What happens if <someheader.h> doesn't declare somefunction()?
Shouldn't the test fail in the compilation phase already? It should
but many compilers don't follow the C99 and later standards that
prohibit implicit function declarations. Instead such compilers
assume that somefunction() exists, compilation succeeds (with a
warning), and then linker with LTO omits the call to somefunction().
Change the tests so that they are part of main(). If compiler accepts
implicitly declared functions, LTO cannot omit them because it has to
assume that they might have side effects and thus linking will fail.
On the other hand, if the functions/intrinsics being used are supported,
they might get optimized away but in that case it's fine because they
really are supported.
It is fine to use __attribute__((target(...))) for main(). At least
it works with GCC 4.9 to 14.1 on x86-64.
Lasse Collin [Mon, 24 Jun 2024 14:39:54 +0000 (17:39 +0300)]
CI: Workaround buggy config.guess on Ubuntu 22.04LTS and 24.04LTS
Check for the wrong triplet from config.guess and override it with
the --build option on the configure command line. Then i386 assembly
autodetection will work.
These Ubuntu versions (and as of writing, also Debian unstable)
ship config.guess version 2022-01-09 which contains a bug that
was fixed in version 2022-05-08. It results in a wrong configure
triplet when using CC="gcc -m32" to build i386 binaries.
Lasse Collin [Sat, 15 Jun 2024 20:34:29 +0000 (23:34 +0300)]
CMake: Link Threads::Threads as PRIVATE to liblzma
This way pthread options aren't passed to the linker when linking
against shared liblzma but they are still passed when linking against
static liblzma. (Also, one never needs the include path of the
threading library to use liblzma since liblzma's API headers
don't #include <pthread.h>. But <pthread.h> tends to be in the
default include path so here this change makes no difference.)
One cannot mix target_link_libraries() calls that use the scope
(PRIVATE, PUBLIC, or INTERFACE) keyword and calls that don't use it.
The calls without the keyword are like PUBLIC except perhaps when
they aren't, or something like that... It seems best to always
specify a scope keyword as the meanings of those three keywords
at least are clear.
Lasse Collin [Sun, 16 Jun 2024 16:37:36 +0000 (19:37 +0300)]
CMake: Use CMAKE_THREAD_LIBS_INIT in liblzma.pc only with pthreads
This shouldn't make much difference in practice as on Windows
no flags are needed anyway and unitialized variable (when threading
is disabled) expands to empty. But it's clearer this way.
Lasse Collin [Sun, 16 Jun 2024 16:18:56 +0000 (19:18 +0300)]
CMake: Use relative paths in liblzma.pc if possible
Now liblzma.pc can be relocatable only if using CMake >= 3.20
but that should be OK as now we shouldn't get broken liblzma.pc
if CMAKE_INSTALL_LIBDIR or CMAKE_INSTALL_INCLUDEDIR contain an
absolute path.
Lasse Collin [Sun, 16 Jun 2024 10:39:37 +0000 (13:39 +0300)]
liblzma: CRC CLMUL: Omit is_arch_extension_supported() when not needed
On E2K the function compiles only due to compiler emulation but the
function is never used. It's cleaner to omit the function when it's
not needed even though it's a "static inline" function.
Lasse Collin [Wed, 12 Jun 2024 11:26:44 +0000 (14:26 +0300)]
CMake: Prefer C11 with a fallback to C99
There is no need to make a similar change in configure.ac.
With Autoconf 2.72, the deprecated macro AC_PROG_CC_C99
is an alias for AC_PROG_CC which prefers a C11 compiler.
projects / thirdparty / xz.git / log
