Joel Rosdahl [Mon, 7 Feb 2022 19:36:47 +0000 (20:36 +0100)]
feat: Remove automatic enabling of ccache for ccache build
When enabling ccache using the standard recommended method (via PATH or
via masquerading symlinks), UseCcache.cmake will introduce double
invocations of ccache. I find this a bit annoying and I also think that
it should be up to the user to enable or not enable ccache and which
settings to use.
Joel Rosdahl [Mon, 7 Feb 2022 19:20:08 +0000 (20:20 +0100)]
fix: Ensure that raw filenames are static in length
PrimaryStorage::evict assumes that the entry number is encoded in the
filename as one character, so make sure that this precondition holds in
Result::{Reader,Writer} as well.
Joel Rosdahl [Wed, 17 Nov 2021 15:47:41 +0000 (16:47 +0100)]
fix: Correct entry_size field for result entries
The entry_size field in result entries created by ccache 4.5 are off by
one. Fix this by properly accounting for the new format_ver field in
result payload data.
Note: This bug only affects the recompression feature (-X/--recompress)
and not other ccache usage.
Joel Rosdahl [Mon, 8 Nov 2021 18:38:35 +0000 (19:38 +0100)]
feat: Add namespace support
If a namespace configured, the namespace string will be added to the
hashed data for each compilation. This will make the associated cache
entries logically separate from cache entries with other namespaces, but
they will still share the same storage space. Cache entries can also be
selectively removed from the primary cache with the command line option
--evict-namespace, potentially in combination with --evict-older-than.
Joel Rosdahl [Thu, 4 Nov 2021 07:20:33 +0000 (08:20 +0100)]
feat: Improve cache entry format
Cache entry header changes:
* Changed magic bytes to represent a generic “ccache entry” instead of a
specific result/manifest entry.
* Added a payload type field (currently manifest or result) separate
from the magic bytes. This allows code that only operates on the
“container” (e.g. recompression) to be ignorant of payload types and
changes to payload formats.
* Added a creation timestamp field. This can be useful when debugging,
for instance to match the timestamp with a build start/end.
* Added a field for the ccache version that created the cache entry.
This can be useful for debugging, and it also makes it possible for
future ccache versions to refuse using results from previous buggy
ccache versions.
* Added a tag field to be used by a future feature.
Cache entry epilogue changes:
* Use 128-bit XXH3 instead of 64-bit XXH3 for the checksum since those
extra bits are computed by the algorithm anyway.
Manifest/result entry changes:
* Added manifest/result format version fields. This means that the
payload version can be stepped without stepping the outer format
version, thereby making it possible for older ccache versions to for
instance recompress the cache even though the payload format has
changed.
Other improvements:
* Added generic Reader and Writer interfaces to decouple code from using
FILE* directly.
* Refactored checksum handling into ChecksummingReader and
ChecksummingWriter.
After ignoring SIGPIPE (b2a88e7) keep-alive support has been
re-enabled for HTTP Storage. Unfortunately it's still broken
but does not crash ccache, anymore.
This is the tcpdump of a (cache miss) compilation that takes longer
than the server keep-alive timeout:
```
10:51:17.214678 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [S], seq 1018344313, win 62727, options [mss 8961,sackOK,TS val 1638189791 ecr 0,nop,wscale 7], length 0
10:51:17.216386 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [S.], seq 2695216843, ack 1018344314, win 26847, options [mss 8645,sackOK,TS val 1598789083 ecr 1638189791,nop,wscale 9], length 0
10:51:17.216424 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 1, win 491, options [nop,nop,TS val 1638189793 ecr 1598789083], length 0
10:51:17.216534 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [P.], seq 1:143, ack 1, win 491, options [nop,nop,TS val 1638189793 ecr 1598789083], length 142: HTTP: GET /cache/ff87vlgke3poou7uq1ma2m3hh000gckmq HTTP/1.1
10:51:17.217393 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [.], ack 143, win 55, options [nop,nop,TS val 1598789084 ecr 1638189793], length 0
10:51:17.217433 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [P.], seq 1:309, ack 143, win 55, options [nop,nop,TS val 1598789085 ecr 1638189793], length 308: HTTP: HTTP/1.1 404 Not Found
10:51:17.217456 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 309, win 489, options [nop,nop,TS val 1638189794 ecr 1598789085], length 0
10:51:17.538819 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [P.], seq 143:285, ack 309, win 489, options [nop,nop,TS val 1638190115 ecr 1598789085], length 142: HTTP: GET /cache/135di89r02vv5fgopbtsopfjllo01b8lg HTTP/1.1
10:51:17.539838 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [P.], seq 309:617, ack 285, win 57, options [nop,nop,TS val 1598789407 ecr 1638190115], length 308: HTTP: HTTP/1.1 404 Not Found
10:51:17.539876 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 617, win 487, options [nop,nop,TS val 1638190117 ecr 1598789407], length 0
10:52:32.536322 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [F.], seq 617, ack 285, win 57, options [nop,nop,TS val 1598864404 ecr 1638190117], length 0
10:52:32.577893 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 618, win 487, options [nop,nop,TS val 1638265155 ecr 1598864404], length 0
The PUT error due to the closed socket is also logged:
```
[2021-09-14T10:52:42.068112 2089 ] Storing result file modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o
[2021-09-14T10:52:42.068118 2089 ] Storing embedded file #0 .o (36847216 bytes) from modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o
[2021-09-14T10:52:42.154930 2089 ] Storing result file modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o.d
[2021-09-14T10:52:42.154951 2089 ] Storing embedded file #1 .d (122626 bytes) from modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o.d
[2021-09-14T10:52:42.155260 2089 ] Stored 135di89r02vv5fgopbtsopfjllo01b8lg in primary storage (/cache/ccache/1/3/5di89r02vv5fgopbtsopfjllo01b8lgR)
[2021-09-14T10:52:42.175445 2089 ] Failed to put /cache/135di89r02vv5fgopbtsopfjllo01b8lg to http storage: Unknown (1)
[2021-09-14T10:52:42.175999 2089 ] Adding result key to /cache/ccache/f/f/87vlgke3poou7uq1ma2m3hh000gckmqM
[2021-09-14T10:52:42.179375 2089 ] Using default compression level 1
[2021-09-14T10:52:42.180306 2089 ] Stored ff87vlgke3poou7uq1ma2m3hh000gckmq in primary storage (/cache/ccache/f/f/87vlgke3poou7uq1ma2m3hh000gckmqM)
[2021-09-14T10:52:42.180341 2089 ] Not putting in http://jenkins3-bazelcache.ec2.getgotools.net:8080/cache since it failed earlier
```
I believe a 90s compilation is worth caching and more important than
skipping the TCP three-way handshake.
Joel Rosdahl [Tue, 28 Sep 2021 18:36:17 +0000 (20:36 +0200)]
fix: Don’t update manifest on preprocessed hit with disabled direct mode
After c7c0837a23fe9dc79613bf3dd4ddd8d91c58d541, update_manifest_file can
be called erroneously if the direct mode is disabled when there has been
a preprocessed hit.
Fix this by asserting that update_manifest_file is only called when
direct mode is enabled, fixing the call site and adding the test case
that should already have existed.
projects / thirdparty / ccache.git / log
