From: Philippe Waroquiers Date: Sun, 8 Apr 2012 19:52:38 +0000 (+0000) Subject: outer/inner setup: new perf/vg_perf options to run perf tests + support translation... X-Git-Tag: svn/VALGRIND_3_8_0~364 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=2bd926a725b8c0e3ff0ace9f0ccd9357b650afcf;p=thirdparty%2Fvalgrind.git outer/inner setup: new perf/vg_perf options to run perf tests + support translation chaining in inner. * perf/vg_perf: Similarly to tests/vg_regtest, perf/vg_perf now accepts the 3 optional arguments: --outer-valgrind --outer-tool --outer-args This allows easy analysis or comparison of performance between different Valgrind versions (e.g. using callgrind, or cachegrind/cg_diff). * See README_DEVELOPERS for more details. * vg_regtest modified so as to use the 'in-place' build of inner, rather than the installed version. * added option --smc-check=all-non-file to vg_perf and vg_regtest outer default arguments (needed when evaluating a Valgrind which does translation chaining). git-svn-id: svn://svn.valgrind.org/valgrind/trunk@12496 --- diff --git a/README_DEVELOPERS b/README_DEVELOPERS index d09917a2a3..e6a005e66c 100644 --- a/README_DEVELOPERS +++ b/README_DEVELOPERS @@ -136,7 +136,15 @@ A different and possibly easier way is as follows: Self-hosting ~~~~~~~~~~~~ -To run Valgrind under Valgrind: +This section explains : + (A) How to configure Valgrind to run under Valgrind. + Such a setup is called self hosting, or outer/inner setup. + (B) How to run Valgrind regression tests in a 'self-hosting' mode, + e.g. to verify Valgrind has no bugs such as memory leaks. + (C) How to run Valgrind performance tests in a 'self-hosting' mode, + to analyse and optimise the performance of Valgrind and its tools. + +(A) How to configure Valgrind to run under Valgrind: (1) Check out 2 trees, "Inner" and "Outer". Inner runs the app directly. Outer runs Inner. @@ -148,6 +156,7 @@ To run Valgrind under Valgrind: (4) Choose a very simple program (date) and try outer/.../bin/valgrind --sim-hints=enable-outer --trace-children=yes \ + --smc-check=all-non-file \ --run-libc-freeres=no --tool=cachegrind -v \ inner/.../bin/valgrind --vgdb-prefix=./inner --tool=none -v prog @@ -156,6 +165,10 @@ program, not its stage2. Outer needs --run-libc-freeres=no, as otherwise it will try to find and run __libc_freeres in the inner, while libc is not used by the inner. Inner needs --vgdb-prefix=./inner to avoid inner gdbserver colliding with outer gdbserver. +Currently, inner does *not* use the client request +VALGRIND_DISCARD_TRANSLATIONS for the JITted code or the code patched for +translation chaining. So the outer needs --smc-check=all-non-file to +detect the modified code. Debugging the whole thing might imply to use up to 3 GDB: * a GDB attached to the Outer valgrind, allowing @@ -186,7 +199,8 @@ All this has not been tested much, so don't be surprised if you hit problems. When using self-hosting with an outer Callgrind tool, use '--pop-on-jump' (on the outer). Otherwise, Callgrind has much higher memory requirements. -Regression tests in an outer/inner setup: +(B) Regression tests in an outer/inner setup: + To run all the regression tests with an outer memcheck, do : perl test/vg_regtest --outer-valgrind=../outer/.../bin/valgrind \ --all @@ -197,7 +211,7 @@ Regression tests in an outer/inner setup: To run regression tests with another outer tool: perl tests/vg_regtest --outer-valgrind=../outer/.../bin/valgrind \ - --outer-tool=helgrind " --all + --outer-tool=helgrind --all --outer-args allows to give specific arguments to the outer tool, replacing the default one provided by vg_regtest. @@ -211,6 +225,48 @@ it contains the detected race conditions. The file tests/outer_inner.supp contains suppressions for the irrelevant or benign errors found in the inner. +(C) Performance tests in an outer/inner setup: + + To run all the performance tests with an outer cachegrind, do : + perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind perf + + To run a specific perf test (e.g. bz2) in this setup, do : + perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind perf/bz2 + + To run all the performance tests with an outer callgrind, do : + perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind \ + --outer-tool=callgrind perf + + To compare the performance of multiple Valgrind versions, do : + perl perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind \ + --vg=../inner_xxxx --vg=../inner_yyyy perf + (where inner_xxxx and inner_yyyy are the versions to compare). + Cachegrind and cg_diff are particularly handy to obtain a delta + between the two versions. + +When the outer tool is callgrind or cachegrind, the following +output files will be created for each test: + .out.... + .outer.log.... + (where tt is the two letters abbreviation for the inner tool(s) run). + +For example, the command + perl perf/vg_perf \ + --outer-valgrind=../outer_trunk/install/bin/valgrind \ + --outer-tool=callgrind \ + --vg=../inner_tchain --vg=../inner_trunk perf/many-loss-records + +produces the files + callgrind.out.inner_tchain.no.many-loss-records.18465 + callgrind.outer.log.inner_tchain.no.many-loss-records.18465 + callgrind.out.inner_tchain.me.many-loss-records.21899 + callgrind.outer.log.inner_tchain.me.many-loss-records.21899 + callgrind.out.inner_trunk.no.many-loss-records.21224 + callgrind.outer.log.inner_trunk.no.many-loss-records.21224 + callgrind.out.inner_trunk.me.many-loss-records.22916 + callgrind.outer.log.inner_trunk.me.many-loss-records.22916 + + Printing out problematic blocks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you want to print out a disassembly of a particular block that diff --git a/perf/vg_perf.in b/perf/vg_perf.in index 2188a31ed6..9be023e18e 100644 --- a/perf/vg_perf.in +++ b/perf/vg_perf.in @@ -58,9 +58,14 @@ usage: vg_perf [options] [files or dirs] -h --help show this message --reps= number of repeats for each program [1] --tools= tools to run [Nulgrind and Memcheck] - --vg Valgrind(s) to measure (can be specified multiple - times). The "in-place" build is used. - [Valgrind in the current directory] + --vg Valgrind(s) to measure [Valgrind in the current directory] + (can be specified multiple times). + The "in-place" build is used. + + --outer-valgrind: run these Valgrind(s) under the given outer valgrind. + These Valgrind(s) must be configured with --enable-inner. + --outer-tool: tool to use by the outer valgrind (default cachegrind). + --outer-args: use this as outer tool args. Any tools named in --tools must be present in all directories specified with --vg. (This is not checked.) @@ -79,6 +84,14 @@ my $n_reps = 1; # Run each test $n_reps times and choose the best one. my @vgdirs; # Dirs of the various Valgrinds being measured. my @tools = ("none", "memcheck"); # tools being measured +# Outer valgrind to use, and args to use for it. +# If this is set, --valgrind should be set to the installed inner valgrind, +# and --valgrind-lib will be ignore +my $outer_valgrind; +my $outer_tool = "cachegrind"; +my $outer_args; + + my $num_tests_done = 0; my $num_timings_done = 0; @@ -119,7 +132,6 @@ sub add_vgdir($) { my ($vgdir) = @_; if ($vgdir !~ /^\//) { $vgdir = "$tests_dir/$vgdir"; } - validate_program($vgdir, "./coregrind/valgrind", 1, 1); push(@vgdirs, $vgdir); } @@ -137,6 +149,12 @@ sub process_command_line() add_vgdir($1); } elsif ($arg =~ /^--tools=(.+)$/) { @tools = split(/,/, $1); + } elsif ($arg =~ /^--outer-valgrind=(.*)$/) { + $outer_valgrind = $1; + } elsif ($arg =~ /^--outer-tool=(.*)$/) { + $outer_tool = $1; + } elsif ($arg =~ /^--outer-args=(.*)$/) { + $outer_args = $1; } else { die $usage; } @@ -256,6 +274,17 @@ sub do_one_test($$) my $cmd = "$timecmd $prog $args"; my $tNative = time_prog($cmd, $n_reps); + if (defined $outer_valgrind) { + $outer_valgrind = validate_program($tests_dir, $outer_valgrind, 1, 1); + foreach my $vgdir (@vgdirs) { + validate_program($vgdir, "./coregrind/valgrind", 1, 1); + } + } else { + foreach my $vgdir (@vgdirs) { + validate_program($vgdir, "./coregrind/valgrind", 1, 1); + } + } + foreach my $vgdir (@vgdirs) { # Benchmark name printf("%-8s ", $name); @@ -272,20 +301,50 @@ sub do_one_test($$) # First two chars of toolname for abbreviation my $tool_abbrev = $tool; $tool_abbrev =~ s/(..).*/$1/; - - # Do the tool run(s). Set both VALGRIND_LIB and VALGRIND_LIB_INNER - # in case this Valgrind was configured with --enable-inner. And - # also VALGRINDLIB, which was the old name for the variable, to - # allow comparison against old Valgrind versions (eg. 2.4.X). printf(" %s:", $tool_abbrev); - my $vgsetup = "VALGRINDLIB=$vgdir/.in_place " - . "VALGRIND_LIB=$vgdir/.in_place " - . "VALGRIND_LIB_INNER=$vgdir/.in_place "; + my $run_outer_args = ""; + if (not defined $outer_args) { + $run_outer_args = + " -v --command-line-only=yes" + . " --run-libc-freeres=no --sim-hints=enable-outer" + . " --smc-check=all-non-file" + . " --vgdb=no --trace-children=yes --read-var-info=no" + . " --suppressions=../tests/outer_inner.supp" + . " --memcheck:leak-check=full --memcheck:show-reachable=no" + . " --cachegrind:cache-sim=yes --cachegrind:branch-sim=yes" + . " --cachegrind:cachegrind-out-file=cachegrind.out.$vgdirname.$tool_abbrev.$name.%p" + . " --callgrind:cache-sim=yes --callgrind:branch-sim=yes" + . " --callgrind:dump-instr=yes --callgrind:collect-jumps=yes" + . " --callgrind:callgrind-out-file=callgrind.out.$vgdirname.$tool_abbrev.$name.%p" + . " "; + } else { + $run_outer_args = $outer_args; + } + + my $vgsetup = ""; my $vgcmd = "$vgdir/coregrind/valgrind " . "--command-line-only=yes --tool=$tool -q " . "--memcheck:leak-check=no " . "--trace-children=yes " . "$vgopts "; + # Do the tool run(s). + if (defined $outer_valgrind ) { + # in an outer-inner setup, only set VALGRIND_LIB_INNER + $vgsetup = "VALGRIND_LIB_INNER=$vgdir/.in_place "; + $vgcmd = "$outer_valgrind " + . "--tool=" . $outer_tool . " " + . "$run_outer_args " + . "--log-file=" . "$outer_tool.outer.log.$vgdirname.$tool_abbrev.$name.%p " + . $vgcmd; + } else { + # Set both VALGRIND_LIB and VALGRIND_LIB_INNER + # in case this Valgrind was configured with --enable-inner. And + # also VALGRINDLIB, which was the old name for the variable, to + # allow comparison against old Valgrind versions (eg. 2.4.X). + $vgsetup = "VALGRINDLIB=$vgdir/.in_place " + . "VALGRIND_LIB=$vgdir/.in_place " + . "VALGRIND_LIB_INNER=$vgdir/.in_place "; + } my $cmd = "$vgsetup $timecmd $vgcmd $prog $args"; my $tTool = time_prog($cmd, $n_reps); printf("%4.1fs (%4.1fx,", $tTool, $tTool/$tNative); diff --git a/tests/vg_regtest.in b/tests/vg_regtest.in index 9ea1d05d06..224385fb63 100755 --- a/tests/vg_regtest.in +++ b/tests/vg_regtest.in @@ -164,8 +164,6 @@ my $valgrind = "./coregrind/valgrind"; chomp(my $tests_dir = `pwd`); # Outer valgrind to use, and args to use for it. -# If this is set, --valgrind should be set to the installed inner valgrind, -# and --valgrind-lib will be ignore my $outer_valgrind; my $outer_tool = "memcheck"; my $outer_args; @@ -238,11 +236,12 @@ sub process_command_line() $valgrind = validate_program($tests_dir, $valgrind, 1, 0); if (defined $outer_valgrind) { - $outer_valgrind = validate_program($tests_dir, $outer_valgrind, 1, 0); + $outer_valgrind = validate_program($tests_dir, $outer_valgrind, 1, 1); if (not defined $outer_args) { $outer_args = - " --command-line-only=yes" + " --command-line-only=yes" . " --run-libc-freeres=no --sim-hints=enable-outer" + . " --smc-check=all-non-file" . " --vgdb=no --trace-children=yes --read-var-info=no" . " --suppressions=" . validate_program($tests_dir,"./tests/outer_inner.supp",1,0) @@ -457,12 +456,12 @@ sub do_one_test($$) } # Pass the appropriate --tool option for the directory (can be overridden - # by an "args:" line, though). Set both VALGRIND_LIB and - # VALGRIND_LIB_INNER in case this Valgrind was configured with - # --enable-inner. + # by an "args:" line, though). my $tool=determine_tool(); if (defined $outer_valgrind ) { - mysystem("$outer_valgrind " + # in an outer-inner setup, only set VALGRIND_LIB_INNER + mysystem( "VALGRIND_LIB_INNER=$valgrind_lib " + . "$outer_valgrind " . "--tool=" . $outer_tool . " " . "$outer_args " . "--log-file=" . "$name.outer.log " @@ -471,7 +470,9 @@ sub do_one_test($$) . "--tool=$tool $extraopts $vgopts " . "$prog $args > $name.stdout.out 2> $name.stderr.out"); } else { - mysystem("VALGRIND_LIB=$valgrind_lib VALGRIND_LIB_INNER=$valgrind_lib " + # Set both VALGRIND_LIB and VALGRIND_LIB_INNER in case this Valgrind + # was configured with --enable-inner. + mysystem( "VALGRIND_LIB=$valgrind_lib VALGRIND_LIB_INNER=$valgrind_lib " . "$valgrind --command-line-only=yes --memcheck:leak-check=no " . "--tool=$tool $extraopts $vgopts " . "$prog $args > $name.stdout.out 2> $name.stderr.out");