]> git.ipfire.org Git - thirdparty/git.git/blob - t/perf/README
clone: allow "--bare" with "-o"
[thirdparty/git.git] / t / perf / README
1 Git performance tests
2 =====================
3
4 This directory holds performance testing scripts for git tools. The
5 first part of this document describes the various ways in which you
6 can run them.
7
8 When fixing the tools or adding enhancements, you are strongly
9 encouraged to add tests in this directory to cover what you are
10 trying to fix or enhance. The later part of this short document
11 describes how your test scripts should be organized.
12
13
14 Running Tests
15 -------------
16
17 The easiest way to run tests is to say "make". This runs all
18 the tests on the current git repository.
19
20 === Running 2 tests in this tree ===
21 [...]
22 Test this tree
23 ---------------------------------------------------------
24 0001.1: rev-list --all 0.54(0.51+0.02)
25 0001.2: rev-list --all --objects 6.14(5.99+0.11)
26 7810.1: grep worktree, cheap regex 0.16(0.16+0.35)
27 7810.2: grep worktree, expensive regex 7.90(29.75+0.37)
28 7810.3: grep --cached, cheap regex 3.07(3.02+0.25)
29 7810.4: grep --cached, expensive regex 9.39(30.57+0.24)
30
31 Output format is in seconds "Elapsed(User + System)"
32
33 You can compare multiple repositories and even git revisions with the
34 'run' script:
35
36 $ ./run . origin/next /path/to/git-tree p0001-rev-list.sh
37
38 where . stands for the current git tree. The full invocation is
39
40 ./run [<revision|directory>...] [--] [<test-script>...]
41
42 A '.' argument is implied if you do not pass any other
43 revisions/directories.
44
45 You can also manually test this or another git build tree, and then
46 call the aggregation script to summarize the results:
47
48 $ ./p0001-rev-list.sh
49 [...]
50 $ ./run /path/to/other/git -- ./p0001-rev-list.sh
51 [...]
52 $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh
53
54 aggregate.perl has the same invocation as 'run', it just does not run
55 anything beforehand.
56
57 You can set the following variables (also in your config.mak):
58
59 GIT_PERF_REPEAT_COUNT
60 Number of times a test should be repeated for best-of-N
61 measurements. Defaults to 3.
62
63 GIT_PERF_MAKE_OPTS
64 Options to use when automatically building a git tree for
65 performance testing. E.g., -j6 would be useful. Passed
66 directly to make as "make $GIT_PERF_MAKE_OPTS".
67
68 GIT_PERF_MAKE_COMMAND
69 An arbitrary command that'll be run in place of the make
70 command, if set the GIT_PERF_MAKE_OPTS variable is
71 ignored. Useful in cases where source tree changes might
72 require issuing a different make command to different
73 revisions.
74
75 This can be (ab)used to monkeypatch or otherwise change the
76 tree about to be built. Note that the build directory can be
77 re-used for subsequent runs so the make command might get
78 executed multiple times on the same tree, but don't count on
79 any of that, that's an implementation detail that might change
80 in the future.
81
82 GIT_PERF_REPO
83 GIT_PERF_LARGE_REPO
84 Repositories to copy for the performance tests. The normal
85 repo should be at least git.git size. The large repo should
86 probably be about linux.git size for optimal results.
87 Both default to the git.git you are running from.
88
89 GIT_PERF_EXTRA
90 Boolean to enable additional tests. Most test scripts are
91 written to detect regressions between two versions of Git, and
92 the output will compare timings for individual tests between
93 those versions. Some scripts have additional tests which are not
94 run by default, that show patterns within a single version of
95 Git (e.g., performance of index-pack as the number of threads
96 changes). These can be enabled with GIT_PERF_EXTRA.
97
98 You can also pass the options taken by ordinary git tests; the most
99 useful one is:
100
101 --root=<directory>::
102 Create "trash" directories used to store all temporary data during
103 testing under <directory>, instead of the t/ directory.
104 Using this option with a RAM-based filesystem (such as tmpfs)
105 can massively speed up the test suite.
106
107
108 Naming Tests
109 ------------
110
111 The performance test files are named as:
112
113 pNNNN-commandname-details.sh
114
115 where N is a decimal digit. The same conventions for choosing NNNN as
116 for normal tests apply.
117
118
119 Writing Tests
120 -------------
121
122 The perf script starts much like a normal test script, except it
123 sources perf-lib.sh:
124
125 #!/bin/sh
126 #
127 # Copyright (c) 2005 Junio C Hamano
128 #
129
130 test_description='xxx performance test'
131 . ./perf-lib.sh
132
133 After that you will want to use some of the following:
134
135 test_perf_fresh_repo # sets up an empty repository
136 test_perf_default_repo # sets up a "normal" repository
137 test_perf_large_repo # sets up a "large" repository
138
139 test_perf_default_repo sub # ditto, in a subdir "sub"
140
141 test_checkout_worktree # if you need the worktree too
142
143 At least one of the first two is required!
144
145 You can use test_expect_success as usual. In both test_expect_success
146 and in test_perf, running "git" points to the version that is being
147 perf-tested. The $MODERN_GIT variable points to the git wrapper for the
148 currently checked-out version (i.e., the one that matches the t/perf
149 scripts you are running). This is useful if your setup uses commands
150 that only work with newer versions of git than what you might want to
151 test (but obviously your new commands must still create a state that can
152 be used by the older version of git you are testing).
153
154 For actual performance tests, use
155
156 test_perf 'descriptive string' '
157 command1 &&
158 command2
159 '
160
161 test_perf spawns a subshell, for lack of better options. This means
162 that
163
164 * you _must_ export all variables that you need in the subshell
165
166 * you _must_ flag all variables that you want to persist from the
167 subshell with 'test_export':
168
169 test_perf 'descriptive string' '
170 foo=$(git rev-parse HEAD) &&
171 test_export foo
172 '
173
174 The so-exported variables are automatically marked for export in the
175 shell executing the perf test. For your convenience, test_export is
176 the same as export in the main shell.
177
178 This feature relies on a bit of magic using 'set' and 'source'.
179 While we have tried to make sure that it can cope with embedded
180 whitespace and other special characters, it will not work with
181 multi-line data.
182
183 Rather than tracking the performance by run-time as `test_perf` does, you
184 may also track output size by using `test_size`. The stdout of the
185 function should be a single numeric value, which will be captured and
186 shown in the aggregated output. For example:
187
188 test_perf 'time foo' '
189 ./foo >foo.out
190 '
191
192 test_size 'output size'
193 wc -c <foo.out
194 '
195
196 might produce output like:
197
198 Test origin HEAD
199 -------------------------------------------------------------
200 1234.1 time foo 0.37(0.79+0.02) 0.26(0.51+0.02) -29.7%
201 1234.2 output size 4.3M 3.6M -14.7%
202
203 The item being measured (and its units) is up to the test; the context
204 and the test title should make it clear to the user whether bigger or
205 smaller numbers are better. Unlike test_perf, the test code will only be
206 run once, since output sizes tend to be more deterministic than timings.