Joel Rosdahl [Sun, 18 Jan 2026 13:48:06 +0000 (14:48 +0100)]
feat: Add support for remote storage helpers
This commit adds support for communicating with a remote storage server
using a long-lived local helper process, started by ccache on demand.
The helper process can keep connections alive (thus amortizing the
session setup cost), and knowledge about remote storage protocols can be
kept out of the ccache code base, making it possible to develop and
release support for different storage server protocols independently.
The storage helper listens to a Unix-domain socket on POSIX systems and
a named pipe on Windows systems. Ccache communicates with it using a
custom IPC protocol, described in doc/remote_storage_helper_spec.md. The
helper is named ccache-storage-<scheme> (e.g. ccache-storage-https) and
can be placed in $PATH, next to the ccache executable or in ccache's
libexec directory. Storage helpers time terminate after a while on
inactivity.
Built-in support for http and redis is kept for now but will likely be
removed in a future ccache release.
- The syntax of the remote_storage/CCACHE_REMOTE_STORAGE configuration
option has been improved (in a backward compatible way): What used to
be called "attributes" are now split into "properties" and "custom
attributes", where properties are things that ccache knows about and
acts on (e.g. read-only and shards) while attributes are custom
key-value pairs sent to the helper without ccache knowing or caring
about them. Syntax: A property is "key=value", a custom attribute is
"@key=value".
- Properties/attributes are now separated by whitespace instead of pipe
characters.
New remote storage properties:
- data-timeout: Timeout for send/receive data transfer. Resets whenever
data is received or can be sent. Default: 1s.
- request-timeout: Timeout for the whole request. Default: 10s.
- idle-timeout: Timeout for the helper process to wait before exiting
after client inactivity (0s: stay up indefinitely). Default: 10m.
- helper: Override which storage helper to spawn (either a filename or a
full path). This is mainly useful when developing or testing new
helpers.
The new command line option "--stop-storage-helpers" can be used to ask
spawned storage helpers to stop immediately.
A new "libexec_dirs" configuration option is available for overriding
the default libexec determined at build time.
(${n} is file number 0-9, ${nn} is file number in two-digit hex form.)
The original reason for adding R and M suffixes was to simplify
implementation of local cleanup and recompression somewhat (and also
discoverability in tests), but I think that consistency trumps this.
Joel Rosdahl [Sat, 25 Oct 2025 17:21:09 +0000 (19:21 +0200)]
feat: Use base16 encoding for all hash digest keys
This reverts the mixed encoding scheme introduced in commit 47cb00f7 ("Encode
hash digests as 4 base16 digits + 29 base32hex digits") back to pure base16
encoding.
Background: The mixed base16/base32hex encoding was introduced to reduce
filename lengths slightly, thereby reducing the number of system calls when
scanning local cache directories. At the time, "the effect is very small but
there is no real downside".
However, with the introduction of remote storage helpers implemented outside
ccache's code base, a downside has emerged: key encoding inconsistency. Remote
storage helpers must now either:
1. reimplement ccache's custom encoding algorithm to maintain key consistency
between the remote storage and ccache's local storage (and logs), or
2. use standard base16 encoding, resulting in different key representations
between ccache and the remote storage.
By standardizing on base16 encoding throughout, we ensure that:
- remote storage helpers can use simple, standard base16 encoding
- keys are represented identically in ccache's local storage and remote storage
- external implementations don't need to replicate custom encoding logic
- debugging and key correlation across systems is straightforward
The minor filesystem performance benefit of shorter filenames is outweighed by
the ecosystem benefits of a simple, standard encoding scheme.
Local storage will for now be backward compatible: to avoid invalidating
existing cache entries, ccache will write in the new format but will try
both the new and the legacy format on cache lookup. The legacy format is
also still used for hashes that are part of the input hash. When some
other input hash material is changed (which invalidates cache entries
anyway) in the future, we'll drop the legacy format completely.
Using MSVC's source dependency file for extracting header files didn't
work out:
- The header filenames are always absolute paths, so cache entries
cannot be shared between directories even when using relative paths on
the command line.
- The header filenames are normalized to lowercase, making it harder to
match them against things like base_dir.
Joel Rosdahl [Mon, 15 Dec 2025 17:19:27 +0000 (18:19 +0100)]
test: Fix -march=native test to work on hybrid CPU systems
On a hybrid CPU system (e.g. Intel with performance and efficient
cores), -march=native expansion can be different on different core types
and thus fail the "-march=native, no CWD in input hash" test if the
first and second compilation end up on different core types. Fix this by
just verifying that CWD doesn't appear in the expansion.
Joel Rosdahl [Sat, 22 Nov 2025 13:44:00 +0000 (14:44 +0100)]
feat: Improve compiler check on macOS
On macOS, the /usr/bin/clang(++) executables are shims that redirect to
different compilers based on $DEVELOPER_DIR or xcode-select
configuration, so ccache's default compiler check (hash mtime+size) is
done on those shims instead of the real compilers.
Improve this by trying to look up the real compiler via $DEVELOPER_DIR
or xcode-select configuration when hashing compiler data.
Joel Rosdahl [Fri, 31 Oct 2025 20:19:40 +0000 (21:19 +0100)]
feat(msvc): Use /sourceDependencies to extract include file paths
Extract include file paths from MSVC's /sourceDependencies output
instead of parsing preprocessed output. This eliminates the need for the
/utf-8 flag and msvc_utf8 config option, allowing ccache to work
correctly with both Unicode and "ANSI" (non-Unicode) codebases without
user configuration.
This resolves the encoding issues described in issues #1619 and #1650 by
using a more robust approach that doesn't depend on source file
encoding.
The change applies to both preprocessor mode and depend mode. For MSVC
versions older than 2017 15.7 that don't support /sourceDependencies,
ccache falls back to parsing preprocessed output. Other MSVC-like
compilers (clang-cl, nvcc, etc.) continue using /showIncludes.
Joel Rosdahl [Tue, 18 Nov 2025 18:15:38 +0000 (19:15 +0100)]
test: Add basic MSVC integration test suite
We need some basic MSVC integration/smoke tests. Since I think it's a
dead end to try to adapt the bash-based test suite for MSVC, this commit
adds a separate Python-based test suite instead. It's not a full
solution but rather something quick to cover the most basic testing
needs.
J. Neuschäfer [Tue, 28 Oct 2025 18:48:23 +0000 (19:48 +0100)]
build(blake3): Assume aarch64 to be little-endian on CMake 3.18/3.19 (#1652)
The recently introduced detection of big-endian AArch64 uses the
CMAKE_C_BYTE_ORDER variable, which was introduced in 3.20, but ccache
currently only requires CMake 3.18.
To avoid using an undefined variable, simply assume AArch64 to be
little-endian on CMake versions that don't define the aforementioned
variable. This is okay because big-endian AArch64 has few users and the
subset of them that use CMake 3.18/3.19 is likely negligible.
projects / thirdparty / ccache.git / log
