Joel Rosdahl [Wed, 17 Nov 2021 15:47:41 +0000 (16:47 +0100)]
fix: Correct entry_size field for result entries
The entry_size field in result entries created by ccache 4.5 are off by
one. Fix this by properly accounting for the new format_ver field in
result payload data.
Note: This bug only affects the recompression feature (-X/--recompress)
and not other ccache usage.
Joel Rosdahl [Mon, 8 Nov 2021 18:38:35 +0000 (19:38 +0100)]
feat: Add namespace support
If a namespace configured, the namespace string will be added to the
hashed data for each compilation. This will make the associated cache
entries logically separate from cache entries with other namespaces, but
they will still share the same storage space. Cache entries can also be
selectively removed from the primary cache with the command line option
--evict-namespace, potentially in combination with --evict-older-than.
Joel Rosdahl [Thu, 4 Nov 2021 07:20:33 +0000 (08:20 +0100)]
feat: Improve cache entry format
Cache entry header changes:
* Changed magic bytes to represent a generic “ccache entry” instead of a
specific result/manifest entry.
* Added a payload type field (currently manifest or result) separate
from the magic bytes. This allows code that only operates on the
“container” (e.g. recompression) to be ignorant of payload types and
changes to payload formats.
* Added a creation timestamp field. This can be useful when debugging,
for instance to match the timestamp with a build start/end.
* Added a field for the ccache version that created the cache entry.
This can be useful for debugging, and it also makes it possible for
future ccache versions to refuse using results from previous buggy
ccache versions.
* Added a tag field to be used by a future feature.
Cache entry epilogue changes:
* Use 128-bit XXH3 instead of 64-bit XXH3 for the checksum since those
extra bits are computed by the algorithm anyway.
Manifest/result entry changes:
* Added manifest/result format version fields. This means that the
payload version can be stepped without stepping the outer format
version, thereby making it possible for older ccache versions to for
instance recompress the cache even though the payload format has
changed.
Other improvements:
* Added generic Reader and Writer interfaces to decouple code from using
FILE* directly.
* Refactored checksum handling into ChecksummingReader and
ChecksummingWriter.
After ignoring SIGPIPE (b2a88e7) keep-alive support has been
re-enabled for HTTP Storage. Unfortunately it's still broken
but does not crash ccache, anymore.
This is the tcpdump of a (cache miss) compilation that takes longer
than the server keep-alive timeout:
```
10:51:17.214678 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [S], seq 1018344313, win 62727, options [mss 8961,sackOK,TS val 1638189791 ecr 0,nop,wscale 7], length 0
10:51:17.216386 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [S.], seq 2695216843, ack 1018344314, win 26847, options [mss 8645,sackOK,TS val 1598789083 ecr 1638189791,nop,wscale 9], length 0
10:51:17.216424 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 1, win 491, options [nop,nop,TS val 1638189793 ecr 1598789083], length 0
10:51:17.216534 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [P.], seq 1:143, ack 1, win 491, options [nop,nop,TS val 1638189793 ecr 1598789083], length 142: HTTP: GET /cache/ff87vlgke3poou7uq1ma2m3hh000gckmq HTTP/1.1
10:51:17.217393 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [.], ack 143, win 55, options [nop,nop,TS val 1598789084 ecr 1638189793], length 0
10:51:17.217433 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [P.], seq 1:309, ack 143, win 55, options [nop,nop,TS val 1598789085 ecr 1638189793], length 308: HTTP: HTTP/1.1 404 Not Found
10:51:17.217456 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 309, win 489, options [nop,nop,TS val 1638189794 ecr 1598789085], length 0
10:51:17.538819 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [P.], seq 143:285, ack 309, win 489, options [nop,nop,TS val 1638190115 ecr 1598789085], length 142: HTTP: GET /cache/135di89r02vv5fgopbtsopfjllo01b8lg HTTP/1.1
10:51:17.539838 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [P.], seq 309:617, ack 285, win 57, options [nop,nop,TS val 1598789407 ecr 1638190115], length 308: HTTP: HTTP/1.1 404 Not Found
10:51:17.539876 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 617, win 487, options [nop,nop,TS val 1638190117 ecr 1598789407], length 0
10:52:32.536322 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [F.], seq 617, ack 285, win 57, options [nop,nop,TS val 1598864404 ecr 1638190117], length 0
10:52:32.577893 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 618, win 487, options [nop,nop,TS val 1638265155 ecr 1598864404], length 0
The PUT error due to the closed socket is also logged:
```
[2021-09-14T10:52:42.068112 2089 ] Storing result file modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o
[2021-09-14T10:52:42.068118 2089 ] Storing embedded file #0 .o (36847216 bytes) from modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o
[2021-09-14T10:52:42.154930 2089 ] Storing result file modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o.d
[2021-09-14T10:52:42.154951 2089 ] Storing embedded file #1 .d (122626 bytes) from modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o.d
[2021-09-14T10:52:42.155260 2089 ] Stored 135di89r02vv5fgopbtsopfjllo01b8lg in primary storage (/cache/ccache/1/3/5di89r02vv5fgopbtsopfjllo01b8lgR)
[2021-09-14T10:52:42.175445 2089 ] Failed to put /cache/135di89r02vv5fgopbtsopfjllo01b8lg to http storage: Unknown (1)
[2021-09-14T10:52:42.175999 2089 ] Adding result key to /cache/ccache/f/f/87vlgke3poou7uq1ma2m3hh000gckmqM
[2021-09-14T10:52:42.179375 2089 ] Using default compression level 1
[2021-09-14T10:52:42.180306 2089 ] Stored ff87vlgke3poou7uq1ma2m3hh000gckmq in primary storage (/cache/ccache/f/f/87vlgke3poou7uq1ma2m3hh000gckmqM)
[2021-09-14T10:52:42.180341 2089 ] Not putting in http://jenkins3-bazelcache.ec2.getgotools.net:8080/cache since it failed earlier
```
I believe a 90s compilation is worth caching and more important than
skipping the TCP three-way handshake.
Joel Rosdahl [Tue, 28 Sep 2021 18:36:17 +0000 (20:36 +0200)]
fix: Don’t update manifest on preprocessed hit with disabled direct mode
After c7c0837a23fe9dc79613bf3dd4ddd8d91c58d541, update_manifest_file can
be called erroneously if the direct mode is disabled when there has been
a preprocessed hit.
Fix this by asserting that update_manifest_file is only called when
direct mode is enabled, fixing the call site and adding the test case
that should already have existed.
Joel Rosdahl [Fri, 20 Aug 2021 17:56:17 +0000 (19:56 +0200)]
fix(test): Fix .incbin test with newer binutils
The assembler in binutils 2.37 doesn’t want to include /dev/null with
a .incbin directive:
/tmp/ccbGWiew.s: Assembler messages:
/tmp/ccbGWiew.s:4: Error: unable to include `/dev/null'
incbin.s: Assembler messages:
incbin.s:1: Error: unable to include `/dev/null'
Joel Rosdahl [Thu, 19 Aug 2021 18:34:43 +0000 (20:34 +0200)]
build: Improve detection av Ccache version from source repo
When running CMake outside the source repository a plain git command
naturally won’t find the repository. Fix this by pointing git towards
the repository.
Joel Rosdahl [Mon, 16 Aug 2021 06:10:21 +0000 (08:10 +0200)]
feat: Improve statistics summary
The number of statistics counters has become very large, making the
output “ccache -s” hard to overview.
Improvements:
- The -s/--show-stats option now prints a more condensed overview where
the counters representing “uncacheable calls” are summed as
“uncacheable” and “errors” counters. Note: Scripts should use
--print-stats instead of trying to parse the output of --show-stats.
- Added hit rate for direct/preprocessed hits/misses as well.
- Added a new -v/--verbose option, which makes --show-stats and
--show-log-stats show more details.
Joel Rosdahl [Tue, 10 Aug 2021 13:32:07 +0000 (15:32 +0200)]
feat: Make --config-path and --directory affect whole command line
Command line options are processed strictly in order, which means that
ccache -d dir -s
is not the same as
ccache -s -d dir
which is not very intuitive. There are also planned features where is
doesn’t make sense to have to put “non-command options” before the
“command options”.
Improve this by processing command line options in two passes: first
non-command options and then command options (still in order).
projects / thirdparty / ccache.git / log
