Tim Kientzle [Sun, 5 Jul 2026 17:12:33 +0000 (10:12 -0700)]
[Read core] Allow format readers to FAIL a header read
While investigating improvements to Zip's header reading,
I found an overlooked case in the read core: When a format
returned FAILED for a header read (indicating that the entry
could not be read but that the archive as a whole is still valid),
the core silently converted that to FATAL, ending the archive
at that point.
The change to the read core adds a new internal state "DATA_RECOVERY"
indicating that we're post-header (like the existing DATA state) but
that client requests to actually obtain data are not valid.
Tim Kientzle [Sun, 5 Jul 2026 17:11:50 +0000 (10:11 -0700)]
[Read core] Test handling of FAILED header reads
While investigating improvements to Zip's header reading,
I found an overlooked case in the read core: When a format
returned FAILED for a header read (indicating that the entry
could not be read but that the archive as a whole is still valid),
the core silently converted that to FATAL, ending the archive
at that point.
Two formats already returned FAILED for certain header issues:
Zip and LHA both indicated unreadable symlink entries in this way,
which should allow the rest of the archive to continue being read.
New tests here verify that we can now continue reading past
such entries, and that clients who ignore the FAILED result
get the same FATAL handling as other API misuse.
data [Sun, 5 Jul 2026 16:31:53 +0000 (00:31 +0800)]
warc: track remaining entry bytes when writing data
The WARC writer capped each data write against the full declared entry
size, but never reduced that value after accepting data. As a result,
repeated write calls could emit more bytes than the WARC Content-Length.
Track the remaining byte count for the current regular-file entry and
decrement it after each successful write. Further writes are then
clamped to the remaining length and return zero once the entry is full.
acl: fix over-read on empty field in archive_acl_from_text_nl
archive_acl_from_text_nl() parses ACL text from a pointer and a length
and does not require a NUL terminator; the tar reader passes a
__archive_read_ahead() window of exactly value_length bytes for
SCHILY.acl.access/.default/.ace. When an entry is empty (a
whitespace-only value, or trailing whitespace after a valid entry)
next_field() consumes the rest of the buffer as whitespace and leaves
field[0].start one past the end, which the '#'-comment probe then
dereferences. Capture the buffer end and skip the entry when its first
field starts there. The wide archive_acl_from_text_w is NUL terminated
and unaffected.
acl: add test_acl_from_text_overread regression test
Drives archive_acl_from_text_nl() with exact-length, non-NUL-terminated
buffers whose first field is only whitespace, covering the empty-entry
and trailing-whitespace cases. Allow the test to include
archive_acl_private.h via __LIBARCHIVE_TEST.
Read the fixed metadata prefix of the PKWARE Unix extra field 0x000d.
The fixed portion contains atime, mtime, uid, and gid values. Require
the full 12-byte prefix before reading it, and leave the variable-length
data field unsupported for now.
Add a test for the PKWARE Unix extra field 0x000d.
The fixture stores a regular file with the 12-byte fixed metadata
prefix in both the local header and central directory. The test checks
that the ZIP reader exposes mtime, atime, uid, and gid metadata for
seekable and streaming reads.
WARC records require a known Content-Length. Reject regular file
entries without a valid size instead of treating them as zero.
Avoid casting archive_entry_size() through size_t before storing it in
the uint64_t WARC Content-Length field, since that can truncate large
sizes on 32-bit platforms.
If enough bytes have to be skipped, a signed integer overflow could
occur (most realistically on 32 bit systems). Use an unsigned type,
which could still overflow but has only negative impact on diagnostic
warning message.
archive_write_pax_header() set hardlink entries to size zero for the
restricted pax format, then immediately set all hardlink entries to size
zero again.
Remove the narrower check because it is fully subsumed by the following
hardlink check.
header_common() parsed the tar mode field once when setting the file
type and again when setting permissions. Cache the parsed value and reuse
it for both setters.
archive_entry_set_filetype() and archive_entry_set_perm() already mask
the relevant bits, so passing the full header mode preserves behavior.
The permission-set guard is unchanged.
Tim Kientzle [Fri, 3 Jul 2026 20:05:41 +0000 (13:05 -0700)]
[Write] Truncate write-to-fd at the declared entry size
This is the fix for....
Update the common archive_read_data_into_fd() to truncate
output to the actual entry size from the most recent
archive_read_header result.
This addresses a hole in handling of certain subtly-malformed archives
that can have more data than declared in the header. Before this,
bsdtar explicitly truncates the entry data to the actual declared size
when extracting to a file, but did not do so when extracting to
stdout.
Tim Kientzle [Fri, 3 Jul 2026 19:57:07 +0000 (12:57 -0700)]
[Write] Truncate write-to-fd at the declared entry size
This is the test for....
Update the common archive_read_data_into_fd() to truncate
output to the actual entry size from the most recent
archive_read_header result.
This addresses a hole in handling of certain subtly-malformed archives
that can have more data than declared in the header. Before this,
bsdtar explicitly truncates the entry data to the actual declared size
when extracting to a file, but did not do so when extracting to
stdout.
Tim Kientzle [Sat, 4 Jul 2026 04:25:54 +0000 (21:25 -0700)]
[Zip] Reject archives if the 32-bit size field is not correct
Reverse an old experiment: When the 32-bit size field is provided, it
now must match the actual size exactly. We no longer permit it to
only match the least-significant 32 bits of the actual size.
--begin-history--
When developing the earliest versions of libarchive's Zip support, I
experimented with the idea of supporting greater-than-4GiB entries in
32-bit Zip archives. This proved to be technically feasible as long
as the _compressed_ data was small enough to keep the resulting
archive under 4GiB. But this also required us to interpret
the 32-bit uncompressed size field as representing only the
least-significant 32 bits of the size.
--end-history--
Tim Kientzle [Sat, 4 Jul 2026 04:22:48 +0000 (21:22 -0700)]
[Zip] Be strict about the 32-bit uncompressed size
Test: If the uncompressed size stored in the archive differs
from the actual uncompressed size by a multiple of 2^32,
we should report some kind of error.
This currently goes undiagnosed because of an old attempt
to support larger-than-4GiB entries in plain 32-bit Zip
archives without necessarily requiring Zip64 extensions.
While this is technically feasible, it requires treating
the 32-bit size field ambiguously, which can lead to size
confusion from readers.
ArchiveProperties entries have a property ID, a size, and payload.
The reader parsed the ID and size, but did not consume the payload.
Payload bytes could then be read as the next property ID, which
misaligned the rest of header parsing.
Skip the payload in chunks so parsing stays aligned with 7-Zip/7zz.
If the raw parser ever treats a basically endless stream which
eventually overflows INT64_MAX, fail instead of overflowing
offset + size calculation, leading to subsequent issues in callers.
Tim Kientzle [Fri, 3 Jul 2026 20:14:27 +0000 (13:14 -0700)]
[Zip] Reject entries with overlong bodies
If an entry declares an uncompressed size, we should not return any
data beyond that point. This doesn't impact bsdtar and friends since
they independently truncate to the declared entry size, but other
clients may not be so careful.
Apparently discovered (but not reported) by @bikini
Reported by @dkgkdfg65
Tim Kientzle [Fri, 3 Jul 2026 20:13:31 +0000 (13:13 -0700)]
[Zip] Reject entries with overlong bodies
Tests for...
If an entry declares an uncompressed size, we should not return any
data beyond that point. This doesn't impact bsdtar and friends since
they independently truncate to the declared entry size, but other
clients may not be so careful.
Apparently discovered (but not reported) by @bikini
Reported by @dkgkdfg65
zip: avoid signed overflow in encrypted size checks
Avoid adding the encryption overhead directly to entry sizes when deciding
whether Zip64 is needed or when updating the stored compressed size.
For the Zip64 decision, compare against ZIP_4GB_MAX - additional_size. For
stored encrypted entries, use archive_ckd_add_i64() so the size update and
overflow check happen together.
cab: Translate E8 calls only in first 1 GB of data
The cab specification states that E8 translation shall be performed only
within the first 1 GB range of an uncompressed file, i.e. for the first
0x8000 CFDATA entries in a folder.
win32: repair symlink mode 'H' (hybrid) in wildcard search
The Windows disk reader supports wildcards by converting a path like
"foo/*" into a single tree rooted at "foo/". This does not play nicely
with symlink mode `H`, which treats the first item in each tree as a
dereferenced link (L) and all subsequent items as physical links (P). On
Windows, we take the first _match_ in L mode and all subsequent
descendants and matches in P mode.
This creates a behavioral difference between `bsdtar -c -H *` and
`bsdtar -c -H one two three`.
This commit changes how we handle H->L/P transitions when doing wildcard
matches. The transition from H to P is now done when we descend into our
first directory, and we switch back to H mode when we ascend out of
it[^1].
Closes #3222
[^1]: This does not matter in practice because tree is breadth-first
rather than depth-first, but I've done it in the interest of
completeness and not leaving ourselves caltrops to step on later.
Pluto Yang [Wed, 1 Jul 2026 12:57:14 +0000 (20:57 +0800)]
test: fix incorrect hole count reporting on loongarch64
On LoongArch64 with 16KB pages, Linux tmpfs often uses Transparent
Huge Pages (THP) sized at 32MiB. When a test writes data followed
by a 32MiB hole, the kernel may allocate a single huge page folio
to back the entire range in RAM. Because the range is physically
backed by memory, SEEK_HOLE correctly (from the kernel's perspective)
identifies the range as DATA, failing the test's assertion. Increasing
MIN_HOLE to 64MiB ensures the hole exceeds the huge page size on
loongarch64, preserving the sparse nature of the file for the test.
Move the macro definitions up to the field where they are used, which is
done for other such fields as well.
This in turn allows to avoid the magic value 0 in initial assignment.
Use ST_RD_TRANSLATION instead for better documentation where the finite
state machine starts (and that this state cannot be reached again).
datauwu [Tue, 30 Jun 2026 13:46:55 +0000 (21:46 +0800)]
7zip: fix SFX boundary checks
The SFX scanner missed a valid 7-Zip header when exactly 32 bytes
remained in the lookahead buffer.
The ELF SFX path also missed ".data" when it was at the last valid
position in the section string table. It also allowed e_shstrndx to be
equal to e_shnum, even though that is one past the section table.
Tighten these checks so valid edge cases are handled correctly.
datauwu [Tue, 30 Jun 2026 13:46:54 +0000 (21:46 +0800)]
7zip: add SFX boundary tests
Add PE and ELF SFX fixtures for the SFX scanner boundary cases.
The PE fixture places an empty 7z archive at the final exact
32-byte scan window. The ELF fixture places ".data\0" at the final
valid section string-table position.
datauwu [Tue, 30 Jun 2026 09:12:11 +0000 (17:12 +0800)]
iso9660: fix compress test names
Several ISO tests used a bz2 suffix even though they read .Z files
and verify ARCHIVE_FILTER_COMPRESS. Rename them to use a compress
suffix so the test names match the filter.
datauwu [Tue, 30 Jun 2026 09:12:10 +0000 (17:12 +0800)]
archive_write: group option setter tests
The write option setter checks used several tiny source files with the
same basic shape.
Keep the same DEFINE_TEST cases. Put the write-side option setter
checks in test_archive_write_set_options.c. Remove the extra source
files from the build lists.
datauwu [Tue, 30 Jun 2026 09:12:10 +0000 (17:12 +0800)]
archive_read: group option setter tests
The read option setter checks used several tiny source files with the
same basic shape.
Keep the same DEFINE_TEST cases. Put the read-side option setter
checks in test_archive_read_set_options.c. Remove the extra source
files from the build lists.
datauwu [Tue, 30 Jun 2026 09:12:09 +0000 (17:12 +0800)]
archive_read: group close-twice tests
The close-twice read checks used three tiny source files.
Keep the same DEFINE_TEST cases. Put the fresh-object, fd-backed,
and filename-backed checks in one source file. Remove the two extra
files from the build lists.
Commit e1f890dc62f7931d971e2483efd8a3d30a98b1f4 fixed the underlying
root cause of these OOB checks: If a header is invalid and ARCHIVE_FATAL
is returned, do not allow data reads anymore.
The checks while reading CFDATA already looked rather off, since the
check exists in header parser and the value is never updated, but
prevent worst case scenario.
Keep the tests as regression tests if archive_read.c is modified again.
projects / thirdparty / libarchive.git / log
