This patch changes the policy that does the GC of OldRef and RCEC
conflict cache size.
The current policy is:
A 'more or less' LRU policy is implemented by giving
to each OldRef a generation nr in which it was last touched.
A new generation is created every 50000 new access.
GC is done when the nr of OldRef reaches --conflict-cache-size.
The GC consists in removing enough generations to free
half of the entries.
After GC of OldRef, the RCEC (Ref Counted Exe Contexts)
not referenced anymore are GC-ed.
The new policy is:
An exact LRU policy is implemented using a doubly linked list
of OldRef.
When reaching --conflict-cache-size, the LRU entry is re-used.
The not referenced RCEC are GC-ed when less than 75% of the RCEC
are referenced, and the nr of RCEC is 'big' (at least half the
size of the contextTab, and at least the max nr of RCEC reached
previously).
(note: tried to directly recover a unref'ed RCEC when recovering
the LRU oldref, but that gives a lot of re-creation of RCEC).
new policy has the following advantages/disadvantages:
1. It is faster (at least for big applications)
On a firefox startup/exit, we gain about 1m30 second on 11m.
Similar 5..10% speed up encountered on other big applications
or on the new perf/memrw test.
The speed increase depends on the amount of memory
touched by the application. For applications with a
working set fitting in conflict-cache-size, the new policy
might be marginally slower than previous policy on platforms
having a small cache : the current policy only sets a generation
nr when an address is re-accessed, while the new policy
has to unchain and rechain the OldRef access in the LRU
doubly linked list.
2. It uses less memory (at least for big applications)
Firefox startup/exit "core" arena max use decreases from
1175MB mmap-ed/1060MB alloc-ed
to
994MB mmap-ed/913MB alloc-ed
The decrease in memory is the result of having a lot less RCEC:
The current policy let the nr of RCEC grow till the conflict
cache size is GC-ed.
The new policy limits the nr of RCEC to 133% of the RCEC
really referenced. So, we end up with a max nr of RCEC
a lot smaller with the new policy : max RCEC 191000
versus 1317000, for a total nr of discard RCEC operations
almost the same: 33M versus 32M.
Also, the current policy allocates a big temporary array
to do the GC of OldRef.
With the new policy, size of an OldRef increases because
we need 2 pointers for the LRU doubly linked list, and
we need the accessed address.
In total, the OldRef increase is limited to one Word,
as we do not need anymore the gen, and the 'magic'
for sanity check was removed (the check somewhat
becomes less needed, because an OldRef is never freed
anymore. Also, we do a new cross-check between
the ga in the OldRef and the sparseWA key).
For applications using small memory and having
a small nr of different stack traces accessing memory,
the new policy causes an increase in memory (one Word
per OldRef).
3. Functionally, the new policy gives better past information:
once the steady state is reached (i.e. the conflict cache
is full), the new policy has always --conflict-cache-size
entries of past information.
The current policy has a nr of past information varying
between --conflict-cache-size/2 and --conflict-cache-size
(so in average, 75% of conflict-cache-size).
4. The new code is a little bit smaller/simpler:
The generation based GC is replaced by a simpler LRU policy.
So, in summary, this patch should allow big applications
to use less cpu/memory, while having very little
or no impact on memory/cpu of small applications.
Note that the OldRef data structure LRU policy
is not really explicitely tested by a regtest.
Not easy at first sight to make such a test portable
between platforms/OS/compilers/....
For ppc64, use the endianess of the running program, rather
than an harcoded endness.
(this is because ppc64 supports 2 endness, decided at runtime)
For mips, use BE if running on a non mips system, otherwise
use the endness of the running program
(this is because mips supports 2 endness, but decided at compile time).
fix 346307 fuse filesystem syscall deadlocks
Mark 2 additional syscalls as 'mayblock' when fuse-compatible hint
is given.
Patch from aozgovde@ralota.com
Factor out the 'extend' function. We only need one version for Linux and
one for Darwin. Down from 11.
Carve out a new function 'track_frame_memory' that communicates to the
tool the allocation of a new stack frame. This was slightly different on
Linux and Darwin but should be the same on both platforms.
New files: priv_sigframe.h and sigframe-common.c
Add support for the lbarx, lharx, stbcx and sthcs instructions.
The instructions are part of the ISA 2.06 but were not implemented
in all versions of hardware. The four instructions are all supported
in ISA 2.07. The instructions were put under the ISA 2.07 category
of supported instructions in this patch.
Carl Love [Fri, 17 Apr 2015 23:43:36 +0000 (23:43 +0000)]
Add support for the lbarx, lharx, stbcx and sthcs instructions.
The instructions are part of the ISA 2.06 but were not implemented
in all versions of hardware. The four instructions are all supported
in ISA 2.07. The instructions were put under the ISA 2.07 category
of supported instructions in this patch.
Followup to r14974. That revision oversimplified a condition, part
of which was presumed to be redundant but wasn't. This caused code
to hang due to an infinite signal-delivery loop. Observed and
tracked down by Austin English.
Add 2 tests none/tests/libvex_test and libvexmultiarch_test
The objective of libvex_test is to verify that the VEX lib
can be used in 'single arch mode' (host == guest).
The objective of libvexmultiarch_test is to verify that the VEX lib
can be used in 'multi arch mode' (freely choose host and guest).
(but not many combinations are working: if wordsize or endianess
differs, then libVEX quickly asserts somewhere).
libvex_test.c is somewhat bizarre, as it uses the architecture
for which we have compiled as the guest, and use a 'foreign' arch
as the host.
That allows to avoid having to define in the test a bunch
of arch specific asm instructions : the test just decode a part
of its own code, and translate it to other archs.
By default, only the combination host == guest is run.
Arguments must be given to run other combinations.
See libvex_test.c for a description on how to specify which combinations
to run.
LibVEX host != guest does not (yet?) work when endianess or word size differs
between host and guest.
Also, currently, TILEGX host is not working properly (unless guest is also
TILEGX), as the evcheck instructions generated differs according to
the offset of the host_EvC_{FAILADDR,COUNTER}.
So, using TILEGX as host is only done when guest is also TILEGX.
Note that it is possible to specify a specific host arch to use.
For example, to force TILEGX to be used, do:
./none/tests/libvexmultiarch_test 1034
(where 1034 is the decimal value corresponding to the enum VexArchTILEGX.
This currently aborts with:
...
------------------------ Assembly ------------------------
vex: priv/host_tilegx_defs.c:2353 (emit_TILEGXInstr): Assertion `evCheckSzB_TILEGX() == (UChar*)p - (UChar*)p0' failed.
//// failure exit called by libVEX
Whe TILEGX is fixed, we can remove the specific condition that avoids using
TILEGX as host.
Small changes have been done on VEX to allow more combinations
to work:
* host_mips_defs.c : when not compiled on mips,
a lot of mips specific code is not compiled at all, because
one of _MIPSEL or _MIPSEB must be defined to have either the
little endian code or big endian code.
emit32 function must however work to use mips as host.
So, for this function, if _MIPSEL is not defined, then
the big endian code is compiled in by default.
(the mips endianess should probably be handled like the ppc
endianess, for which the endianness to use is decided at runtime).
* host_arm64_isel.c : addition of a 'do not emit anything' for
ABI HINT (avoid an assert e.g. for amd64 guest, arm64 host)
* libvex_guest_amd64.h : when I was still hoping mixing amd64 and x86,
a first assert was firing up due to size/alignment
of VexGuestAMD64State when compiled in 32 bits.
=> addition of pad elements to ensure the size and alignment
of VexGuestAMD64State stays the same when compiled in 32 and
64 bits (the 64 bits layout is unchanged).
The new tests have been run on x86/amd64/ppc64/s390x.
It is very well possible that the tests will fail on untested archs
(ppc32 or mips* or arm* or tilegx)
(e.g. because the hardcoded hwcaps in libvex_test.c are not ok).
It should be relatively trivial to fix these hwcaps problems.
Some other problems might be less easy to understand and fix
(e.g. similar to the TILEGX evcheck or mips emit32 problem).
Remove useless arguments in sparsewa, that were inheritated from WordFM
These arguments are not needed for sparsewa, as they can only
return the key given in input.
Have the event map GC use the same approach as the other GC
done from libhb_maybe_GC, i.e. check the condition in
libhb_maybe_GC, and call the (non inlined) GC only if
a GC is needed.
Carl Love [Thu, 9 Apr 2015 16:23:20 +0000 (16:23 +0000)]
ADD AT_DCACHEBSIZE and AT_HWCAP2 support for POWER PC
Valgrind currently does not support the following AUX vector entries:
AT_DCACHEBSIZE, and AT_HWCAP2. By default these entries are suppressed by
Valgrind. The attached patch adds the needed support so the user level programs
can correctly determine that hardware level they are running on. Specifically
that the ISA 2.07 for Power 8 is supported.
Bugzilla 345695
This fix adds the needed support. It makes a minor change to allow the
VEX settings of the host platform to be passed down so they can be checked
against the HWCAP values.
The files touched are:
coregrind/m_initimg/initimg-linux.c
coregrind/pub_core_initimg.h
coregrind/m_main.c
Assorted cleanups: remove magic constants and unneeded header file. Update
a few comments. Exit with code 127 in bash emulation mode when file was
not found.
Certain kernels on s390 provide extra read permissions on executable
segments. See discussion here: https://bugs.kde.org/show_bug.cgi?id=345824#c4
Making sure that rx and x compare equal.
Followup to r14898 which changes the storage of segment names by
putting them into a string table.
This patch adds reference counting to segment names and frees them
when they are no longer used. The so freed memory can be reclaimed to
store future segment names.
New file coregrind/m_aspacemgr/aspacemgr-segnames.c which has all the
code dealing with segment names. Carved out of aspacemgr-linux.c
Detailled comments in the code.
Fixes BZ 344559.
The linux launcher showed some odd behaviour. When given a shell script
named 'now' with this contents:
#!
/bin/date
the platform selection logic does this:
--11196:1:launcher no tool requested, defaulting to 'memcheck'
--11196:2:launcher selecting platform for './now'
--11196:2:launcher selecting platform for './now'
--11196:2:launcher opened './now'
--11196:2:launcher read 13 bytes from './now'
--11196:2:launcher selecting platform for ''
--11196:2:launcher selecting platform for '/home/florian/bin/'
--11196:2:launcher opened '/home/florian/bin/'
--11196:2:launcher selected platform 'unknown'
--11196:1:launcher no platform detected, defaulting platform to 'amd64-linux'
That is not quite right. Instead the platform should be determined by
examining the default shell.
Additionally, define VKI_BINPRM_BUF_SIZE because on linux only that many
characters are considered on a #! line. C.f. <linux>/fs/binfmt_script.c
m_ume/* needs to be adapted as well but that is a different patch.
Add testcase for BZ 231357.
To do that a small enhancement to vg_regtest was needed:
(1) New declaration to allow specifying an environemnt variable
that is set prior to invoking valgrind.
eg: env: VAR=VAL
There can be more than one such declaration
(2) prog-asis: program_name
This is like prog: except the program name is not prefixed with
the testdir.
Further reduction of the size of the sector TTE tables
For default memcheck configuration, 32 bits) this patch
decreases by 13.6 MB ie. from 89945856 to 76317696.
Note that the type EClassNo is introduced only for readibility
purpose (and avoid some cast). That does not change the size
of the TTEntry.
The TTEntry size is reduced by using unions and/or Bool on 1 bit.
No performance impact detected (outer callgrind/inner memcheck bz2
on x86 shows a small improvement).
Revision 14976 causes a regression : stacktrace produced when the
stack has not yet been extended to cover SP will only contain one
element, as the stack limits are considered to be the limits of
the resvn segment.
This patch fixes that, by taking Resvn/SmUpper segment into
account to properly compute the limits.
It also contains a new regtest that fails with the trunk
(only one function in the stacktrace)
and succeeds with this patch (the 2 expected functions).
The hint given by Valgrind gdbserver when enabling host visibility
in gdbserver was wrongly giving the file load address,
instead of the text segment address start.
This means that GDB was then showing wrong symbols for an address
(typically, symbols slightly before the address being printed).
This patch ensures the hint given is using the text start address.
Helgrind optimisation:
* do VTS pruning only if new threads were declared
very dead since the last pruning round.
* When doing pruning, use the new list of threads very dead
to do the pruning : this decreases the cost of the dichotomic search
in VTS__substract
Florian Krohm [Fri, 27 Mar 2015 08:47:22 +0000 (08:47 +0000)]
Change the minimum allowable value of aspacem_minAddr to
be VKI_PAGE_SIZE. That follows from the requirement that
the address ought to be page aligned and > 0.
Improve --stats=yes:
* give the avg nr of IPs per execontext
* use the newly introduced %f in m_transtab.c ratio
and in the avg nr of execontext per list
Florian Krohm [Thu, 26 Mar 2015 21:55:00 +0000 (21:55 +0000)]
Add function VG_(am_is_valid_for_aspacem_minAddr) so that the parser
for command line options does not need to know what addresses are valid
for aspacem_minAddr.
That information should be hidden in the address space manager.
Have the very detailed gdbsrv debuglog (e.g. exchange of packets
between GDB and V gdbsrv, fetching/setting registers, ...) done
at debuglog level 3 instead of 1.
This allows to do gdbsrv commands at debuglog level 2
without seeing too much trace.
This patch further reduces the memory used by TT/TC (by about 15Mb
on 32 bits memcheck default nr of sectors).
Memory is reduced by using UShort typedef-s for Sector no and TTE no.
Note that for TTE no, we had a mixture of UShort, UInt and Int used
depending on the place (a TTE no was in any case constrained to be an UShort).
The bss memory/startup space is also reduced by allocating the htt on demand
(like tt and tc), using mmap the first time a sector is initialised.
Changes:
* pub_core_transtab.h :
* 2 typedef to identify a sector and a tt entry (these 2 types are UShort)
* add 2 #define 'invalid values' for these types
* change the interface to use these types rather than UInt
* m_transtab.c
* use wherever relevant these 2 new types rather than UInt or UShort
* replace the use of -1 by INV_SNO or INV_TTE
* remove now useless typecast from Int/UInt to UShort for tte
* schedule.c: use the new types
Florian Krohm [Mon, 23 Mar 2015 17:13:04 +0000 (17:13 +0000)]
Add VG_(am_is_bogus_client_stack_pointer)(Addr).
The function is used in VG_(client_syscall) to avoid extending the stack
when it is clear that the current value of the stack pointer does not
point into a segment that looks like a stack segment.
See the comments in the code there.
As a side effect of this we can now revert r15018 which increased
the stack size of the alternate stack in memcheck/tests/sigaltstack.c.
The reason is that the belief at the time: "alternate stack is too small"
was not correct. What instead happened was that VG_(client_syscall) called
VG_(extend_stack) without need (the syscall was tgkill) and the new stack
pointer happened to be in a file segment.
In other words: the current stack pointer was still within the alternate
stack, i.e. the alternate stack was (barely) large enough.
Tom Hughes [Sun, 22 Mar 2015 11:01:58 +0000 (11:01 +0000)]
Include the platform name in the unhandled system call message
We often get bug reports for an unhandled system call which don't
make it clear what platform is in use, which makes it impossible
to know which system call it is.
343902 --vgdb=yes doesn't break when --xml=yes is used
Changes ensures that gdbserver is called also when xml is yes.
When gdbserver is set to yes, we have to temporarily reset
xml output to no, as gdbserver output (e.g. print the last error)
has to be printed to gdb.
Florian Krohm [Sat, 21 Mar 2015 10:58:37 +0000 (10:58 +0000)]
Change the GCC demangler to not use VLA. The rationale is that these VLAs
are allocated on the stack and they can become quite large - in particular
when the client is a C++ application using the Boost library.
In combination with the demanglers recursive nature this can quickly lead
to exhaustion of valgrind's per-thread stack (which cannot be dynamically
grown). Additionally, due to the large VLAs (I've seen a 32k array) we
could run out of stack space without issuing a prior warning and instead
just segfault.
Therefore this patch allocates these arrays on the heap and frees them
later. Basically this is a respin of Joseph's r10385.
Change TT/TC hashing data structure (decreases memory by 50MB for memcheck 32bits)
This patch changes the way the transtab entries hash table is done.
Currently, the hash table is an open hash table considered full at 65%.
This means that in average, 1 entry on 3 is unused.
(all the hash table memory will be 'active' for big applications,
as the active entries are normally reasonably distributed over the hash table).
The size of a transtab entry is significant (about 150 Bytes).
To avoid having 35% of the entries unused, the translation table
is split in 2:
An hash table, that will contain an index pointing at the transtab entries.
With this technique, we are adding a small hash table,
but we spare 35% of the translation table.
Performance measurements have shown no degradation,
and some platforms have better performance. Not too clear why,
probably this helps platforms with small caches ?).
Florian Krohm [Mon, 16 Mar 2015 22:03:42 +0000 (22:03 +0000)]
Increase the size of the alternate stack. It was too small.
This was found by accident and there is no known way to detect
an overflow of an alternate stack in the general case.
New Option --avg-transtab-entry-size=<number> can be used to tune
the size of the translation table sectors, either to gain memory
or to avoid too many retranslations.
Fix the following errors detected by makefile check
memcheck/tests/Makefile.am:1: error: wrap8.stderr.exp-ppc64 is missing in EXTRA_DIST
memcheck/tests/Makefile.am:1: error: wrap8.stdout.exp-ppc64 is missing in EXTRA_DIST
memcheck/tests/Makefile.am:1: error: wrap8.stdout.exp2 is in EXTRA_DIST but doesn't exist
memcheck/tests/Makefile.am:1: error: wrap8.stderr.exp2 is in EXTRA_DIST but doesn't exist
projects / thirdparty / valgrind.git / log
