After ignoring SIGPIPE (b2a88e7) keep-alive support has been
re-enabled for HTTP Storage. Unfortunately it's still broken
but does not crash ccache, anymore.
This is the tcpdump of a (cache miss) compilation that takes longer
than the server keep-alive timeout:
```
10:51:17.214678 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [S], seq 1018344313, win 62727, options [mss 8961,sackOK,TS val 1638189791 ecr 0,nop,wscale 7], length 0
10:51:17.216386 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [S.], seq 2695216843, ack 1018344314, win 26847, options [mss 8645,sackOK,TS val 1598789083 ecr 1638189791,nop,wscale 9], length 0
10:51:17.216424 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 1, win 491, options [nop,nop,TS val 1638189793 ecr 1598789083], length 0
10:51:17.216534 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [P.], seq 1:143, ack 1, win 491, options [nop,nop,TS val 1638189793 ecr 1598789083], length 142: HTTP: GET /cache/ff87vlgke3poou7uq1ma2m3hh000gckmq HTTP/1.1
10:51:17.217393 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [.], ack 143, win 55, options [nop,nop,TS val 1598789084 ecr 1638189793], length 0
10:51:17.217433 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [P.], seq 1:309, ack 143, win 55, options [nop,nop,TS val 1598789085 ecr 1638189793], length 308: HTTP: HTTP/1.1 404 Not Found
10:51:17.217456 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 309, win 489, options [nop,nop,TS val 1638189794 ecr 1598789085], length 0
10:51:17.538819 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [P.], seq 143:285, ack 309, win 489, options [nop,nop,TS val 1638190115 ecr 1598789085], length 142: HTTP: GET /cache/135di89r02vv5fgopbtsopfjllo01b8lg HTTP/1.1
10:51:17.539838 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [P.], seq 309:617, ack 285, win 57, options [nop,nop,TS val 1598789407 ecr 1638190115], length 308: HTTP: HTTP/1.1 404 Not Found
10:51:17.539876 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 617, win 487, options [nop,nop,TS val 1638190117 ecr 1598789407], length 0
10:52:32.536322 IP 10.34.132.45.webcache > 10.34.129.62.37468: Flags [F.], seq 617, ack 285, win 57, options [nop,nop,TS val 1598864404 ecr 1638190117], length 0
10:52:32.577893 IP 10.34.129.62.37468 > 10.34.132.45.webcache: Flags [.], ack 618, win 487, options [nop,nop,TS val 1638265155 ecr 1598864404], length 0
The PUT error due to the closed socket is also logged:
```
[2021-09-14T10:52:42.068112 2089 ] Storing result file modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o
[2021-09-14T10:52:42.068118 2089 ] Storing embedded file #0 .o (36847216 bytes) from modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o
[2021-09-14T10:52:42.154930 2089 ] Storing result file modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o.d
[2021-09-14T10:52:42.154951 2089 ] Storing embedded file #1 .d (122626 bytes) from modules/asyncly/Test/Unit/CMakeFiles/test_asyncly.dir/future/FutureTest.cpp.o.d
[2021-09-14T10:52:42.155260 2089 ] Stored 135di89r02vv5fgopbtsopfjllo01b8lg in primary storage (/cache/ccache/1/3/5di89r02vv5fgopbtsopfjllo01b8lgR)
[2021-09-14T10:52:42.175445 2089 ] Failed to put /cache/135di89r02vv5fgopbtsopfjllo01b8lg to http storage: Unknown (1)
[2021-09-14T10:52:42.175999 2089 ] Adding result key to /cache/ccache/f/f/87vlgke3poou7uq1ma2m3hh000gckmqM
[2021-09-14T10:52:42.179375 2089 ] Using default compression level 1
[2021-09-14T10:52:42.180306 2089 ] Stored ff87vlgke3poou7uq1ma2m3hh000gckmq in primary storage (/cache/ccache/f/f/87vlgke3poou7uq1ma2m3hh000gckmqM)
[2021-09-14T10:52:42.180341 2089 ] Not putting in http://jenkins3-bazelcache.ec2.getgotools.net:8080/cache since it failed earlier
```
I believe a 90s compilation is worth caching and more important than
skipping the TCP three-way handshake.
Joel Rosdahl [Tue, 28 Sep 2021 18:36:17 +0000 (20:36 +0200)]
fix: Don’t update manifest on preprocessed hit with disabled direct mode
After c7c0837a23fe9dc79613bf3dd4ddd8d91c58d541, update_manifest_file can
be called erroneously if the direct mode is disabled when there has been
a preprocessed hit.
Fix this by asserting that update_manifest_file is only called when
direct mode is enabled, fixing the call site and adding the test case
that should already have existed.
Joel Rosdahl [Fri, 20 Aug 2021 17:56:17 +0000 (19:56 +0200)]
fix(test): Fix .incbin test with newer binutils
The assembler in binutils 2.37 doesn’t want to include /dev/null with
a .incbin directive:
/tmp/ccbGWiew.s: Assembler messages:
/tmp/ccbGWiew.s:4: Error: unable to include `/dev/null'
incbin.s: Assembler messages:
incbin.s:1: Error: unable to include `/dev/null'
Joel Rosdahl [Thu, 19 Aug 2021 18:34:43 +0000 (20:34 +0200)]
build: Improve detection av Ccache version from source repo
When running CMake outside the source repository a plain git command
naturally won’t find the repository. Fix this by pointing git towards
the repository.
Joel Rosdahl [Mon, 16 Aug 2021 06:10:21 +0000 (08:10 +0200)]
feat: Improve statistics summary
The number of statistics counters has become very large, making the
output “ccache -s” hard to overview.
Improvements:
- The -s/--show-stats option now prints a more condensed overview where
the counters representing “uncacheable calls” are summed as
“uncacheable” and “errors” counters. Note: Scripts should use
--print-stats instead of trying to parse the output of --show-stats.
- Added hit rate for direct/preprocessed hits/misses as well.
- Added a new -v/--verbose option, which makes --show-stats and
--show-log-stats show more details.
Joel Rosdahl [Tue, 10 Aug 2021 13:32:07 +0000 (15:32 +0200)]
feat: Make --config-path and --directory affect whole command line
Command line options are processed strictly in order, which means that
ccache -d dir -s
is not the same as
ccache -s -d dir
which is not very intuitive. There are also planned features where is
doesn’t make sense to have to put “non-command options” before the
“command options”.
Improve this by processing command line options in two passes: first
non-command options and then command options (still in order).
Joel Rosdahl [Sun, 8 Aug 2021 08:40:58 +0000 (10:40 +0200)]
refactor: Improve handling of statistics updates and expected events
Ccache uses exceptions for both truly exceptional events (such as I/O
error) and non-exceptional events (such as unsupported compiler
arguments). Regardless of the nature of the event, at most one
statistics counter can be incremented.
Improve this by:
- Using nonstd::expected instead of exceptions internally in the main
code flow. This makes the code easier to reason about.
- Adding support for communicating several statistics counter updates
both for success and failure code paths. This is in preparation for
future improvements of statistics related to secondary storage.
Joel Rosdahl [Sat, 7 Aug 2021 11:12:56 +0000 (13:12 +0200)]
fix: Don’t read let Context read user configuration in unit tests
accd21e4 inadvertently made unit tests populate Context::config from the
environment and the user configuration. Fix this by moving code for
reading config, setting of logging, etc. into a separate method only to
be called by non-test code.
Joel Rosdahl [Sat, 7 Aug 2021 07:00:17 +0000 (09:00 +0200)]
feat(storage): Add support for cache sharding on secondary storage (#912)
This adds support for a shards attribute with a comma-separated list of
names for sharding (partitioning) the cache entries using Rendezvous
hashing, typically to spread the cache over a server cluster. When set,
the storage URL must contain an asterisk (*), which will be replaced by
one of the shard names to form a real URL. A shard name can optionally
have an appended weight within parentheses to indicate how much of the
key space should be associated with that shard. A shard with weight w
will contain w/S of the cache, where S is the sum of all shard weights.
A weight could for instance be set to represent the available memory for
a memory cache on a specific server. The default weight is 1.
will put 55% (3/5.5) of the cache on redis://cache-a.example.com, 18%
(1/5.5) on redis://cache-b.example.com and 27% (1.5/5.5) on
redis://cache-c.example.com.
projects / thirdparty / ccache.git / log
