Slightly improve x86 unwind intensive workload.
e.g. perf/memrw is improved by 2% to 3% with this patch.
The unwinding code on x86 is trying to unwind using
either the %ebp-chain or CFI unwinding.
If these 2 techniques fail, then it tries to unwind
using FPO (PDB) debug info.
However, unless running wine or similar, there will never be
such FPO/PDB info.
The function VG_(use_FPO_info) is thus called for nothing
for each 'end of stack'. This function scans all the loaded di
to find a debug info that has some FP, to not find anything.
With this patch, the unwind code on x86 will only call VG_(use_FPO_info) if
some FPO/PDB info was loaded.
The fact that FPO/PDB info was loaded is cached and updated similarly to
cfi cache : each time new debug info is loaded, the cache value is refreshed
using the debuginfo generation.
The patch also changes the name of VG_(CF_info_generation)
to VG_(debuginfo_generation), as this generation is changed for
any kind of load or unload of debug info, not only for CFI based debug
info
Some platforms such as x86 and amd64 have efficient unaligned access.
On these platforms, implement read_/write_<type> by doing a direct
access, rather than calling a function that will read or write
'byte per byte'.
For platforms that do not have efficient unaligned access,
or that do not support at all unaligned access, call function
readUAS_/writeUAS_<type> that works as before.
Currently, direct acecss is activated only for x86 and amd64.
Unclear what other platforms support (efficiently) unaligned access.
On unwind intensive code (such as perf/memrw on amd64), this patch
gives up to 5% improvement.
This patch decreases significantly the memory needed for OldRef and
slightly increases the performance. It also moderately improves
the nr of cases where helgrind can provide the stack trace of the old
access (when using the same amount of memory for the OldRef entries).
The patch also provides a new helgrind monitor command to show
the recorded accesses for an address+len, and adds an optional argument
lock_address to the monitor command 'info locks', to show the info
about just this lock.
Currently, oldref are maintained in a sparse WA, that points to N
entries, as specified by --conflict-cache-size=N.
For each entry (associated to an address), we have the last 5 accesses.
Old entries are recycled in an exact LRU order.
But inside an entry, we could have a recent access, and 4 very
old accesses that are kept 'alive' by a single thread accessing
repetitively the address shared with the 4 other old entries.
The attached patch replaces the sparse WA that maintains the OldREf
by an hash table.
Each OldRef now also only maintains one single access for an address.
As an OldRef now maintains only one access, all the entries are now
strictly in LRU mode.
Memory used for OldRef
-----------------------
For the trunk, an OldRef has a size of 72 bytes (on 32 bits archs)
maintaining up to 5 accesses to the same address.
On 64 bits arch, an OldRef is 104 bytes.
With the patch, an OldRef has a size of 32 bytes (on 32 bits archs)
or 56 bytes (on 64 bits archs).
So, for one single access, the new code needs (on 32 bits)
32 bytes, while the trunk needs only 14.4 bytes.
However, that is the worst case, assuming that the 5 entries in the
accs array are all used.
Looking on 2 big apps (one of them being firefox), we see that
we have very few OldRef entries that have the 5 entries occupied.
On a firefox startup, of the 5x1,000,000 accesses, we only have
1,406,939 accesses that are used.
So, in average, the trunk uses in reality around 52 bytes per access.
The default value for --conflict-cache-size has been doubled to 2000000.
This ensures that the memory used for the OldRef is more or less the
same as the trunk (104Mb for OldRef entries).
Memory used for sparseWA versus hashtable
-----------------------------------------
Looking on 2 big apps (one of them being firefox), we see that
there are big variations on the size of the WA : it can go in a few
seconds from 10MB to 250MB, or can decrease back to 10 MB.
This all depends where the last N accesses were done: if well localised,
the WA will be small.
If the last N accesses were distributed over a big address space,
then the WA will be big: the last level of WA (the biggest memory consumer)
uses slightly more than 1KB (2KB on 64 bits) for each '256 bytes' memory
zone where there is an oldref. So, in the worst case, on 32 bits, we
need > 1_000_000_000 sparseWA memory to keep 1_000_000 OldRef.
The hash table has between 1 to 2 Word overhead per OldRef
(as the chain array is +- doubled each time the hash table is full).
So, unless the OldRef are extremely localised, the overhead of the
hash table will be significantly less.
With the patch, the core arena total alloc is: 5299535/1201448632 totalloc-blocks/bytes
The trunk is 6693111/3959050280 totalloc-blocks/bytes
(so, around 1.20Gb versus 3.95Gb).
This big difference is due to the fact that the sparseWA repetitively
allocates then frees Level0 or LevelN when OldRef in the region covered
by the Level0/N have all been recycled.
In terms of CPU
---------------
With the patch, on amd64, a firefox startup seems slightly faster (around 1%).
The peak memory mmaped/used decreases by 200Mb.
For a libreoffice test, the memory decreases by 230Mb. CPU also decreases
slightly (1%).
In terms of correctness:
-----------------------
The trunk could potentially show not the most recent access
to the memory of a race : the first OldRef entry matching the raced upon
address was used, while we could have a more recent access in a following
OldRef entry. In other words, the trunk only guaranteed to find the
most recent access in an OldRef, but not between the several OldRef that
could cover the raced upon address.
So, assuming it is important to show the most recent access, this patch
ensures we really show the most recent access, even in presence of overlapping
accesses.
Mark Wielaard [Fri, 22 May 2015 09:20:03 +0000 (09:20 +0000)]
Add procfs-non-linux.stderr.exp variants to EXTRA_DIST.
For bz#344936 procfs-non-linux.stderr.exp was renamed and split into
procfs-non-linux.stderr.exp-with-readlinkat and
procfs-non-linux.stderr.exp-without-readlinkat add both to EXTRA_DIST.
Fixes make post-regtest-checks.
Improve presentation of first line of --profile-heap=yes
(i.e. use
-------- Arena "client": 4,194,304/4,194,304 max/cu...
instead of
-------- Arena "client": 4194304/4194304 max/cu....
Rhys Kidd [Wed, 20 May 2015 11:31:35 +0000 (11:31 +0000)]
Improve documentation of syscall: unix: 44 profil() which was deprecated around OS X 10.6 and removed from the xnu kernel shipped with OS X 10.7. See unresolved bz#264253.
Carl Love [Tue, 19 May 2015 16:08:05 +0000 (16:08 +0000)]
Fix for the HWCAP2 aux vector.
The support assumed that if HWCAP2 is present that the system also supports
ISA2.07. That assumption is not correct as we have found a few systems (OS)
where the HWCAP2 entry is present but the ISA2.07 bit is not set. This patch
fixes the assertion test to specifically check the ISA2.07 support bit setting
in the HWCAP2 and vex_archinfo->hwcaps variable. The setting for the
ISA2.07 support must be the same in both variables if the HWCAP2 entry exists.
This patch reduces the memory needed for the linesF.
Currently, each SecMap has an array of linesF, referenced by the linesZ
of the secmap that needs a lineF, via an index stored in dict[1].
When the array is full, its size is doubled.
The linesF array of a secmap is freed when the SecMap is GC-ed.
The above strategy has the following consequences:
A. in average, 25% of the LinesF are unused.
B. if a SecMap has 'temporarily' a need for linesF, but afterwards,
these linesF are converted to normal lineZ representation, the linesF
will not be recuperated unless the SecMap is GC-ed (i.e. fully marked
no access).
The patch replaces the linesF array private per SecMap
by a pool allocator of LinesF shared between all SecMap.
A lineZ that needs a lineF will directly point to its lineF (using a pointer
stored in dict[1]), instead of having in dict[1] the index in the SecMap
linesF array.
When a lineZ needs a lineF, it is allocated from the pool allocator.
When a lineZ does not need anymore a lineF, it is returned back to the
pool allocator.
On a firefox startup, the above strategy reduces the memory for linesF
by about 42Mb. It seems that the more firefox is used (e.g. to visit
a few websites), the bigger the memory gain.
After opening the home page of valgrind, wikipedia and google, the memory
gain is about 94Mb:
trunk:
linesF: 392,181 allocd ( 203,934,120 bytes occupied) ( 173,279 used)
patch:
linesF: 212,966 allocd ( 109,038,592 bytes occupied) ( 170,252 used)
There is also less alloc/free operations in core arena with the patch:
trunk:
core : 810,680,320/ 802,291,712 max/curr mmap'd, 17/19 unsplit/split sb unmmap'd, 759,441,224/ 703,191,896 max/curr, 40631760/16376828248 totalloc-blocks/bytes, 188015696 searches 8 rzB
patch:
core : 701,628,416/ 690,753,536 max/curr mmap'd, 12/29 unsplit/split sb unmmap'd, 643,041,944/ 577,793,712 max/curr, 32050040/14056017712 totalloc-blocks/bytes, 174097728 searches 8 rzB
In terms of performance, no CPU impact detected on Firefox startup.
Note we have no representative reproducible (and preferrably small)
perf test that uses extensively linesF. Firefox is a good heavy lineF
user but is far to be reproducible, and is very far to be small.
Theoretically, in terms of CPU performance, the patch might have some
small benefits here and there for read operations, as the lineF pointer
is directly retrieved from the lineZ, rather than retrieved via an indirection
in the linesF array.
For write operations, the patch might need a little bit more CPU,
as we replace an
assignment to lineF inUse boolean to False (and then probably back to True
when the cacheline is written back)
by
a call to pool allocator VG_(freeEltPA) (and then probably a call to
VG_(allocEltPA) when the cacheline is written back).
These PA functions are small, so cost should be ok.
We might however still maintain in clear_LineF_of_Z the last cleared lineF
and re-use it in alloc_LineF_for_Z. Not sure how many calls to the PA functions
would be avoided by this '1 elt cache' (and the needed 'if elt == NULL'
check in both clear_LineF_of_Z and alloc_LineF_for_Z.
This possible optimisationwill be looked at later.
Carl Love [Fri, 15 May 2015 20:09:05 +0000 (20:09 +0000)]
Patch 5 in a revised series of cleanup patches from Will Schmidt
Add a .exp for the pth_cond_destroy_busy for PPC64 big endian.
This is specifically to cover the last line of output as
seen on ppc64BE, which is "ERROR SUMMARY: X errors from 3 contexts",
where X is 6, versus 3 as seen on other architectures.
The additional errors show up on BE during the "Thread #1: pthread_cond
_destroy: destruction of condition variable being waited upon."
Signed-off-by: Will Schmidt <will_schmidt@vnet.ibm.com>
This patch fixes Vagrind bugzilla 347686
Carl Love [Fri, 15 May 2015 16:50:06 +0000 (16:50 +0000)]
Patch 6 in a revised series of cleanup patches from Will Schmidt
Fix multipleinheritance heuristic for ppc64LE (leak_cpp_interior test).
Adjust the PPC64 #ifdiffery to indicate that ppc64BE uses a thunk table,
but ppc64LE (in particular, the ELF ABIV2) does not. In this case, thunk
table == function descriptors.
Signed-off-by: Will Schmidt <will_schmidt@vnet.ibm.com>
--
This patch replaces the previously posted "[6/7] add leak_cpp_interior
test .exp results ....."
This patch (re-)gains performance in helgrind, following revision 15207, that
reduced memory use doing SecMap GC, but was slowing down some workloads
(typically, workloads doing a lot of malloc/free).
A significant part of the slowdown came from the clear of the filter,
that was not optimised for big ranges : the filter was working byte
per byte till an 8 alignment. Then working per 8 bytes at a time.
With the patch, the filter clear is done the following way:
* all the bytes till 8 alignement are done together
* then 8 bytes at a time till filter_line alignment (32 bytes)
* then 32 bytes at a time.
Moreover, as the filter cache is small (1024 lines of 32 bytes),
clearing filter for ranges bigger than 32Kb was uselessly checking
several times the same entry. This is now avoided by using a range
check rather than a tag equality check.
As the new filter clear is significanly more complex than the previous simple
algorithm, the old algorithm is kept and used to check the new algorithm
when CHECK_ZSM is defined as 1.
The patch also contains a few micro optimisations and
disables
// VG_(track_die_mem_stack) ( evh__die_mem );
as this had no effect and was somewhat costly.
With this patch, we have almost reached for all perf tests the same
performance as we had before revision 15207. Some tests are still
slightly slower than before the SecMap GC (max 2% difference).
Some tests are now significantly faster (e.g. sarp).
For almost all tests, we are now faster than valgrind 3.10.1.
Details below.
Regtested on x86/amd64/ppc64 (and regtested with all compile time
checks set).
I have also regtested with libreoffice and firefox.
(with firefox, also with CHECK_ZSM set to 1).
Details about performance:
hgtrace = this patch
trunk_untouched = trunk
base_secmap = trunk before secmap GC
valgrind 3.10.1 included for comparison
Measured on core i5 2.53GHz
Carl Love [Thu, 14 May 2015 21:52:59 +0000 (21:52 +0000)]
Patch 4 in a revised series of cleanup patches from Will Schmidt
Add a suppression to handle a "Jump to the invalid address..." message
that gets generated on power. This is a variation of the existing
suppressions.
While here, I also updated the "prog:" line in the vgtest file to reference
the supp_unknown executable, versus the badjump executable. They share the
same source code, so I think this is effectively cosmetic.
Add the lwpid in the scheduler status information
E.g. we now have:
Thread 1: status = VgTs_Runnable (lwpid 15782)
==15782== at 0x8048EB5: main (sleepers.c:188)
client stack range: [0xBE836000 0xBE839FFF] client SP: 0xBE838F80
valgrind stack top usage: 10264 of 1048576
Thread 2: status = VgTs_WaitSys (lwpid 15828)
==15782== at 0x2E9451: ??? (syscall-template.S:82)
==15782== by 0x8048AD3: sleeper_or_burner (sleepers.c:84)
==15782== by 0x39B924: start_thread (pthread_create.c:297)
==15782== by 0x2F107D: clone (clone.S:130)
client stack range: [0x442F000 0x4E2EFFF] client SP: 0x4E2E338
valgrind stack top usage: 2288 of 1048576
This allows to attach with GDB to the good lwpid in case
you want to examine the valgrind state rather than the guest state.
(it is needed to attach to the specific lwpid as valgrind is not
linked with lib pthread, so GDB cannot discover the threads
of the process).
Implement 'qXfer:exec-file:read' packet in Valgrind gdbserver.
Thanks to this packet, with recent GDB (>= 7.9.50.20150514-cvs), the
command 'target remote' will automatically load the executable file of
the process running under Valgrind. This means you do not need to
specify the executable file yourself, GDB will discover it itself.
See GDB documentation about 'qXfer:exec-file:read' packet for more
info.
Carl Love [Wed, 13 May 2015 21:46:47 +0000 (21:46 +0000)]
Patch 2 in a revised series of cleanup patches from Will Schmidt
Add a deep-D test .exp values for ppc64.
Depending on the system and the systems endianness, there are variances
in the library reference, and to the specific line number in the library.
I was able to add and modify existing filters to cover most of the variations,
but did need to add a .exp to cover the additional call stack entry as seen
on power.
This change allows the ppc64 targets to pass the massif/deep-D test.
Carl Love [Wed, 13 May 2015 21:10:12 +0000 (21:10 +0000)]
Patch 1 in a revised series of cleanup patches from Will Schmidt
Update the massif/big-alloc test for ppc64*.
In comparison to the existing .exp files, the time,total,extra-heap
values generated on ppc64* vary from the other architectures.
This .exp allows the ppc64 targets to pass the test.
* avoid indirection via function pointers to call SVal__rcinc and SVal__rcdec
* declare these functions inlined
* transform 2 asserts on hot path in conditionally compiled checks
on CHECK_ZSM
This slightly optimises some perf tests with helgrind
Improves the way arena statistics are shown
The mmap'd max/curr and max/curr nr of bytes will be shown e.g. as
11,440,408/ 4,508,968
instead of 11440656/ 4509200
So, using more space, but more readable (in particular when the
nr exceeds the width, and so are not aligned anymore)
This patch decreases the memory used by the helgrind SecMap,
by implementing a Garbage Collection for the SecMap.
The basic change is that freed memory is marked as noaccess
(while before, it kept the previous marking, on the basis that
non buggy applications are not accessing freed memory in any case).
Keeping the previous marking avoids the CPU/memory changes needed
to mark noaccess.
However, marking freed memory noaccess and GC the secmap reduces
the memory on big apps.
For example, a firefox test needs 220Mb less (on about 2.06 Gb).
Similar reduction for libreoffice batch (260 MB less on 1.09 Gb).
On such applications, the performance with the patch is similar to the trunk.
There is a performance decrease for applications that are doing
a lot of malloc/free repetitively: e.g. on some perf tests, an increase
in cpu of up to 15% has been observed.
Several performance optimisations can be done afterwards to not loose
too much performance. The decrease of memory is expected to produce
in any case significant benefit in memory constrained environments
(e.g. android phones).
So, after discussion with Julian, it was decided to commit as-is
and (re-)gain (part of) performance in follow-up commits.
Add some cfi directives in the code doing syscall (by Valgrind).
This allows to attach to Valgrind when VAlgrind is blocked in a syscall
and have GDB producing a stacktrace, rather than being unable
to unwind.
I.e. instead of having:
(gdb) bt
#0 0x380460f2 in do_syscall_WRK ()
(gdb)
with the directives, we obtain:
(gdb) bt
#0 vgPlain_mk_SysRes_x86_linux (val=1) at m_syscall.c:65
#1 vgPlain_do_syscall (sysno=168, a1=944907996, a2=1, a3=4294967295, a4=0, a5=0, a6=0, a7=0, a8=0) at m_syscall.c:791
#2 0x38031986 in vgPlain_poll (fds=0x385226dc <remote_desc_pollfdread_activity>, nfds=1, timeout=-1) at m_libcfile.c:535
#3 0x3807479f in vgPlain_poll_no_eintr (fds=0x385226dc <remote_desc_pollfdread_activity>, nfds=1, timeout=-1)
at m_gdbserver/remote-utils.c:86
#4 0x380752f0 in readchar (single=4096) at m_gdbserver/remote-utils.c:938
#5 0x38075ae3 in getpkt (buf=0x61f35020 "") at m_gdbserver/remote-utils.c:997
#6 0x38076fcb in server_main () at m_gdbserver/server.c:1048
#7 0x38072af2 in call_gdbserver (tid=1, reason=init_reason) at m_gdbserver/m_gdbserver.c:721
#8 0x380735ba in vgPlain_gdbserver (tid=1) at m_gdbserver/m_gdbserver.c:788
#9 0x3802c6ef in do_actions_on_error (allow_db_attach=<optimized out>, err=<optimized out>) at m_errormgr.c:532
#10 pp_Error (err=0x61f580e0, allow_db_attach=1 '\001', xml=1 '\001') at m_errormgr.c:644
#11 0x3802cc34 in vgPlain_maybe_record_error (tid=1643479264, ekind=8, a=2271560481, s=0x0, extra=0x62937f1c)
at m_errormgr.c:851
#12 0x38028821 in vgMemCheck_record_free_error (tid=1, a=2271560481) at mc_errors.c:836
#13 0x38007b65 in vgMemCheck_free (tid=1, p=0x87654321) at mc_malloc_wrappers.c:496
#14 0x3807e261 in do_client_request (tid=1) at m_scheduler/scheduler.c:1840
#15 vgPlain_scheduler (tid=1) at m_scheduler/scheduler.c:1406
#16 0x3808b6b2 in thread_wrapper (tidW=<optimized out>) at m_syswrap/syswrap-linux.c:102
#17 run_a_thread_NORETURN (tidW=1) at m_syswrap/syswrap-linux.c:155
#18 0x00000000 in ?? ()
(gdb)
Carl Love [Wed, 6 May 2015 21:11:35 +0000 (21:11 +0000)]
Patch 8 in a series of cleanup patches from Will Schmidt
Add a helper script to determine if the platform is ppc64le.
This is specifically used to help exclude the 32-bit tests from being
run on a ppc64LE (ABIV2) platform. The 32-bit targets, specifically ppc32/*
is not built on LE.
Carl Love [Wed, 6 May 2015 19:44:14 +0000 (19:44 +0000)]
Patch 2 in a series of cleanup patches from Will Schmidt
Adjust the badjump2 test for ppc64le/ABIV2. Under the ABIV2 there
is no function descriptor, so the fn[] setup does not apply.
This fixes the badjump2 test failure as seen on ppc64le.
* Out of memory message was using 'bytes have already been allocated.'
while this nr is in fact the total anonymously mmap-ed.
Change the message so as to reflect the shown number.
* Show also the total anonymous mmaped in non OOM memory statistics
This patch reduces the memory needed for a VtsTE by 25% (one word)
on 32 bits platforms. No memory reduction on 64 bits platforms,
due to alignment.
The patch also shows the vts stats when showing the helgrind stats.
The perf/memrw.c perf test gets also some new additional features
allowing e.g. to control the size of the read or written blocks.
This patch adds a function that allows to directly properly size an xarray
when the size is known in advance.
3 places identified where this function can be used trivially.
The result is a reduction of 'realloc' operations in core
arena, and a small reduction in ttaux arena
(it is the nr of operations that decreases, the memory usage itself
stays the same (ignoring some 'rounding' effects).
E.g. for perf/bigcode 0, we change from
core 1085742/ 216745904 totalloc-blocks/bytes, 1085733 searches
ttaux 5348/ 6732560 totalloc-blocks/bytes, 5326 searches
to
core 712666/ 190998592 totalloc-blocks/bytes, 712657 searches
ttaux 5319/ 6731808 totalloc-blocks/bytes, 5296 searches
For bz2, we switch from
core 50285/ 32383664 totalloc-blocks/bytes, 50256 searches
ttaux 670/ 245160 totalloc-blocks/bytes, 669 searches
to
core 32564/ 29971984 totalloc-blocks/bytes, 32535 searches
ttaux 605/ 243280 totalloc-blocks/bytes, 604 searches
Performance wise, on amd64, this improves memcheck performance
on perf tests by 0.0, 0.1 or 0.2 seconds depending on the test.
projects / thirdparty / valgrind.git / log
