include/pub_tool_deduppoolalloc.h
coregrind/pub_core_deduppoolalloc.h
coregrind/m_deduppoolalloc.c
and uses it (currently only) for the strings in m_debuginfo/storage.c
The idea is that such ddup pool allocator will also be used for other
highly duplicated information (e.g. the DiCFSI information), where
significant gains can also be achieved.
The dedup pool for strings also decreases significantly the memory
needed by the read inline information (patch still to be committed,
see bug 278972).
When testing with a big executable (tacot_process),
this reduces the size of the dinfo arena from
trunk:
158941184/
109760512 max/curr mmap'd,
156775944/
107882728 max/curr,
to
ddup:
157892608/
106614784 max/curr mmap'd,
156362160/
101414712 max/curr
(so 3Mb less mmap-ed once debug info is read, 1Mb less mmap-ed in peak,
6Mb less allocated once debug info is read).
This is all gained due to the string which changes from:
trunk: 17,434,704 in 266: di.storage.addStr.1
to
ddup: 10,966,608 in 750: di.storage.addStr.1
(6.5Mb less memory used by strings)
The gain in mmap-ed memory is smaller due to fragmentation.
Probably one could decrease the fragmentation by using bigger
size for the dedup pool, but then we would lose memory on the last
allocated pool (and for small libraries, we often do not use much
of a big pool block).
Solution might be to increase the pool size but have a "shrink_block"
operation. To be looked at in the future.
In terms of performance, startup of a big executable (on an old pentium)
is not influenced significantly (something like 0.1 seconds on 15 seconds
startup for a big executable, on a slow pentium).
The dedup pool uses a hash table. The hash function used currently
is the VG_(adler32) check sum. It is reported (and visible also here)
that this checksum is not a very good hash function (many collisions).
To have statistics about collisions, use --stats -v -v -v
As an example of the collisions, on the strings in debug info of memcheck tool on x86,
one obtain:
--4789-- dedupPA:di.storage.addStr.1 9983 allocs (8174 uniq) 11 pools (4820 bytes free in last pool)
--4789-- nr occurences of chains of len N, N-plicated keys, N-plicated elts
--4789-- N: 0 : nr chain 6975, nr keys 0, nr elts 0
--4789-- N: 1 : nr chain 3670, nr keys 6410, nr elts 8174
--4789-- N: 2 : nr chain 1070, nr keys 226, nr elts 0
--4789-- N: 3 : nr chain 304, nr keys 100, nr elts 0
--4789-- N: 4 : nr chain 104, nr keys 84, nr elts 0
--4789-- N: 5 : nr chain 72, nr keys 42, nr elts 0
--4789-- N: 6 : nr chain 44, nr keys 34, nr elts 0
--4789-- N: 7 : nr chain 18, nr keys 13, nr elts 0
--4789-- N: 8 : nr chain 17, nr keys 8, nr elts 0
--4789-- N: 9 : nr chain 4, nr keys 6, nr elts 0
--4789-- N:10 : nr chain 9, nr keys 4, nr elts 0
--4789-- N:11 : nr chain 1, nr keys 0, nr elts 0
--4789-- N:13 : nr chain 1, nr keys 1, nr elts 0
--4789-- total nr of unique chains: 12289, keys 6928, elts 8174
which shows that on 8174 different strings, we have only 6410 strings which have
a unique hash value. As other examples, N:13 line shows we have 13 strings
mapping to the same key. N:14 line shows we have 4 groups of 10 strings mapping to the
same key, etc.
So, adler32 is definitely a bad hash function.
Trials have been done with another hash function, giving a much lower
collision rate. So, a better (but still fast) hash function would probably
be beneficial. To be looked at ...
git-svn-id: svn://svn.valgrind.org/valgrind/trunk@14029
pub_core_commandline.h \
pub_core_coredump.h \
pub_core_cpuid.h \
+ pub_core_deduppoolalloc.h \
pub_core_debuginfo.h \
pub_core_debugger.h \
pub_core_debuglog.h \
m_commandline.c \
m_clientstate.c \
m_cpuid.S \
+ m_deduppoolalloc.c \
m_debugger.c \
m_debuglog.c \
m_errormgr.c \
static void free_DebugInfo ( DebugInfo* di )
{
Word i, j, n;
- struct strchunk *chunk, *next;
TyEnt* ent;
GExpr* gexpr;
ML_(dinfo_free)(di->symtab);
}
- for (chunk = di->strchunks; chunk != NULL; chunk = next) {
- next = chunk->next;
- ML_(dinfo_free)(chunk);
- }
+ VG_(deleteDedupPA) (di->strpool);
/* Delete the two admin arrays. These lists exist primarily so
that we can visit each object exactly once when we need to
vg_assert(var);
/* Nothing to free in var: all the pointer fields refer
to stuff either on an admin list, or in
- .strchunks */
+ .strpool */
}
VG_(deleteXA)(arange->vars);
/* Don't free arange itself, as OSetGen_Destroy does
#include "pub_core_basics.h" // Addr
#include "pub_core_xarray.h" // XArray
+#include "pub_core_deduppoolalloc.h" // DedupPoolAlloc
#include "priv_d3basics.h" // GExpr et al.
#include "priv_image.h" // DiCursor
::sec_names is NULL. If there are other names, these are stored in
::sec_names, which is a NULL terminated vector holding the names.
The vector is allocated in VG_AR_DINFO, the names themselves live
- in DebugInfo::strchunks.
+ in DebugInfo::strpool.
From the point of view of ELF, the primary vs secondary distinction
is artificial: they are all just names associated with the address,
typedef
struct {
- HChar* name; /* in DebugInfo.strchunks */
+ HChar* name; /* in DebugInfo.strpool */
UWord typeR; /* a cuOff */
GExpr* gexpr; /* on DebugInfo.gexprs list */
GExpr* fbGX; /* SHARED. */
HChar* fileName; /* where declared; may be NULL. in
- DebugInfo.strchunks */
+ DebugInfo.strpool */
Int lineNo; /* where declared; may be zero. */
}
DiVariable;
};
-/* To do with the string table in struct _DebugInfo (::strchunks) */
-#define SEGINFO_STRCHUNKSIZE (64*1024)
+/* To do with the string table in struct _DebugInfo (::strpool) */
+#define SEGINFO_STRPOOLSIZE (16*1024)
/* We may encounter more than one .eh_frame section in an object --
Addr fpo_maxavma;
Addr fpo_base_avma;
- /* Expandable arrays of characters -- the string table. Pointers
- into this are stable (the arrays are not reallocated). */
- struct strchunk {
- UInt strtab_used;
- struct strchunk* next;
- HChar strtab[SEGINFO_STRCHUNKSIZE];
- } *strchunks;
+ /* Pool of strings -- the string table. Pointers
+ into this are stable (the memory is not reallocated). */
+ DedupPoolAlloc *strpool;
/* Variable scope information, as harvested from Dwarf3 files.
typedef
struct _TempVar {
- HChar* name; /* in DebugInfo's .strchunks */
+ HChar* name; /* in DebugInfo's .strpool */
/* Represent ranges economically. nRanges is the number of
ranges. Cases:
0: .rngOneMin .rngOneMax .manyRanges are all zero
GExpr* fbGX[N_D3_VAR_STACK]; /* if isFunc, contains the FB
expr, else NULL */
/* The file name table. Is a mapping from integer index to the
- (permanent) copy of the string in in DebugInfo's .strchunks. */
+ (permanent) copy of the string in in DebugInfo's .strpool. */
XArray* /* of UChar* */ filenameTable;
}
D3VarParser;
vg_assert(!di->loctab);
vg_assert(!di->cfsi);
vg_assert(!di->cfsi_exprs);
- vg_assert(!di->strchunks);
+ vg_assert(!di->strpool);
vg_assert(!di->soname);
{
#include "pub_core_libcprint.h"
#include "pub_core_xarray.h"
#include "pub_core_oset.h"
+#include "pub_core_deduppoolalloc.h"
#include "priv_misc.h" /* dinfo_zalloc/free/strdup */
#include "priv_image.h"
*/
HChar* ML_(addStr) ( struct _DebugInfo* di, const HChar* str, Int len )
{
- struct strchunk *chunk;
- Int space_needed;
HChar* p;
if (len == -1) {
} else {
vg_assert(len >= 0);
}
-
- space_needed = 1 + len;
-
- // Allocate a new strtab chunk if necessary
- if (di->strchunks == NULL ||
- (di->strchunks->strtab_used
- + space_needed) > SEGINFO_STRCHUNKSIZE) {
- chunk = ML_(dinfo_zalloc)("di.storage.addStr.1", sizeof(*chunk));
- chunk->strtab_used = 0;
- chunk->next = di->strchunks;
- di->strchunks = chunk;
- }
- chunk = di->strchunks;
-
- p = &chunk->strtab[chunk->strtab_used];
- VG_(memcpy)(p, str, len);
- chunk->strtab[chunk->strtab_used+len] = '\0';
- chunk->strtab_used += space_needed;
-
+ if (UNLIKELY(di->strpool == NULL))
+ di->strpool = VG_(newDedupPA)(SEGINFO_STRPOOLSIZE,
+ 1,
+ ML_(dinfo_zalloc),
+ "di.storage.addStr.1",
+ ML_(dinfo_free));
+ p = VG_(allocEltDedupPA) (di->strpool, len+1, str);
return p;
}
Int level,
Addr aMin,
Addr aMax,
- HChar* name, /* in di's .strchunks */
+ HChar* name, /* in di's .strpool */
UWord typeR, /* a cuOff */
GExpr* gexpr,
GExpr* fbGX,
HChar* fileName, /* where decl'd - may be NULL.
- in di's .strchunks */
+ in di's .strpool */
Int lineNo, /* where decl'd - may be zero */
Bool show )
{
vg_assert(a_name);
vg_assert(b_name);
- vg_assert(a_name != b_name);
+ // vg_assert(a_name != b_name);
+ // ???? now the pointers can be equal but is that
+ // ???? not the indication of a latent bug ????
vlena = VG_(strlen)(a_name);
vlenb = VG_(strlen)(b_name);
canonicaliseLoctab ( di );
ML_(canonicaliseCFI) ( di );
canonicaliseVarInfo ( di );
+ if (di->strpool)
+ VG_(freezeDedupPA) (di->strpool);
}
--- /dev/null
+/*--------------------------------------------------------------------*/
+/*--- A pool (memory) allocator that avoids duplicated copies. ---*/
+/*--- m_deduppoolalloc.c ---*/
+/*--------------------------------------------------------------------*/
+/*
+ This file is part of Valgrind, a dynamic binary instrumentation
+ framework.
+
+ Copyright (C) 2014-2014 Philippe Waroquiers philippe.waroquiers@skynet.be
+
+ This program is free software; you can redistribute it and/or
+ modify it under the terms of the GNU General Public License as
+ published by the Free Software Foundation; either version 2 of the
+ License, or (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful, but
+ WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, write to the Free Software
+ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
+ 02111-1307, USA.
+
+ The GNU General Public License is contained in the file COPYING.
+*/
+
+#include "pub_core_basics.h"
+#include "pub_core_libcbase.h"
+#include "pub_core_libcprint.h"
+#include "pub_core_libcassert.h"
+#include "pub_core_xarray.h"
+#include "pub_core_deduppoolalloc.h" /* self */
+#include "pub_core_hashtable.h"
+#include "pub_core_poolalloc.h"
+#include "pub_core_options.h"
+#include "pub_core_mallocfree.h"
+#include "pub_core_debuglog.h"
+
+struct _DedupPoolAlloc {
+ SizeT poolSzB; /* Minimum size of a pool. */
+ SizeT eltAlign;
+ void* (*alloc)(const HChar*, SizeT); /* pool allocator */
+ const HChar* cc; /* pool allocator's cc */
+ void (*free)(void*); /* pool allocator's free-er */
+ /* XArray of void* (pointers to pools). The pools themselves.
+ Each element is a pointer to a block of size at least PoolSzB bytes. */
+ XArray *pools;
+
+ /* hash table of pool elements, used to dedup.
+ If NULL, it means the DedupPoolAlloc is frozen. */
+ VgHashTable ht_elements;
+
+ /* Hash table nodes of pool_elements are allocated with a pool, to
+ decrease memory overhead during insertion in the DedupPoolAlloc. */
+ PoolAlloc *ht_node_pa;
+
+ UChar *curpool_free; /* Pos in current pool to allocate next elt. */
+ UChar *curpool_limit; /* Last pos in current pool. */
+
+ /* Total nr of alloc calls, resulting in (we hope) a lot less
+ real (dedup) elements. */
+ ULong nr_alloc_calls;
+};
+
+typedef
+ struct _ht_node {
+ struct _ht_node *next; // Read/Write by hashtable (pub_tool_hashtable.h)
+ UWord key; // Read by hashtable (pub_tool_hashtable.h)
+ SizeT eltSzB;
+ void *elt;
+ }
+ ht_node;
+
+extern DedupPoolAlloc* VG_(newDedupPA) ( SizeT poolSzB,
+ SizeT eltAlign,
+ void* (*alloc)(const HChar*, SizeT),
+ const HChar* cc,
+ void (*free_fn)(void*) )
+{
+ DedupPoolAlloc* ddpa;
+ vg_assert(poolSzB >= eltAlign);
+ vg_assert(poolSzB >= 100); /* let's say */
+ vg_assert(poolSzB >= 10*eltAlign); /* let's say */
+ vg_assert(alloc);
+ vg_assert(cc);
+ vg_assert(free_fn);
+ ddpa = alloc(cc, sizeof(*ddpa));
+ vg_assert(ddpa);
+ VG_(memset)(ddpa, 0, sizeof(*ddpa));
+ ddpa->poolSzB = poolSzB;
+ ddpa->eltAlign = eltAlign;
+ ddpa->alloc = alloc;
+ ddpa->cc = cc;
+ ddpa->free = free_fn;
+ ddpa->pools = VG_(newXA)( alloc, cc, free_fn, sizeof(void*) );
+
+ ddpa->ht_elements = VG_(HT_construct) (cc);
+ ddpa->ht_node_pa = VG_(newPA) ( sizeof(ht_node),
+ 1000,
+ alloc,
+ cc,
+ free_fn);
+
+ ddpa->curpool_limit = NULL;
+ ddpa->curpool_free = ddpa->curpool_limit + 1;
+ vg_assert(ddpa->pools);
+ return ddpa;
+}
+
+void VG_(deleteDedupPA) ( DedupPoolAlloc* ddpa)
+{
+ Word i;
+ if (ddpa->ht_elements)
+ VG_(freezeDedupPA) (ddpa); // Free data structures used for insertion.
+ for (i = 0; i < VG_(sizeXA) (ddpa->pools); i++)
+ ddpa->free (*(UWord **)VG_(indexXA) ( ddpa->pools, i ));
+ VG_(deleteXA) (ddpa->pools);
+ ddpa->free (ddpa);
+}
+
+static __inline__
+void ddpa_align_curpool_free ( DedupPoolAlloc* ddpa )
+{
+ ddpa->curpool_free = (UChar*)VG_ROUNDUP(ddpa->curpool_free, ddpa->eltAlign);
+}
+
+/* No space. Allocate a new pool. */
+__attribute__((noinline))
+static void ddpa_add_new_pool ( DedupPoolAlloc* ddpa )
+{
+ vg_assert(ddpa);
+ ddpa->curpool_free = ddpa->alloc( ddpa->cc, ddpa->poolSzB);
+ vg_assert(ddpa->curpool_free);
+ ddpa->curpool_limit = ddpa->curpool_free + ddpa->poolSzB - 1;
+ /* add to our collection of pools */
+ VG_(addToXA)( ddpa->pools, &ddpa->curpool_free );
+ ddpa_align_curpool_free (ddpa);
+}
+
+static Word cmp_pool_elt (const void* node1, const void* node2 )
+{
+ const ht_node* hnode1 = node1;
+ const ht_node* hnode2 = node2;
+
+ if (hnode1->key < hnode2->key)
+ return -1;
+ else if (hnode1->key > hnode2->key)
+ return 1;
+ else if (hnode1->eltSzB == hnode2->eltSzB)
+ return VG_(memcmp) (hnode1->elt, hnode2->elt, hnode1->eltSzB);
+ else if (hnode1->eltSzB < hnode2->eltSzB)
+ return -1;
+ else
+ return 1;
+}
+
+/* Print some stats. */
+static void print_stats (DedupPoolAlloc *ddpa)
+{
+ VG_(message)(Vg_DebugMsg,
+ "dedupPA:%s %ld allocs (%d uniq)"
+ " %ld pools (%ld bytes free in last pool)\n",
+ ddpa->cc,
+ (long int) ddpa->nr_alloc_calls,
+ VG_(HT_count_nodes)(ddpa->ht_elements),
+ VG_(sizeXA)(ddpa->pools),
+ (long int) (ddpa->curpool_limit - ddpa->curpool_free + 1));
+ VG_(HT_print_stats) (ddpa->ht_elements, cmp_pool_elt);
+}
+
+/* Dummy free, as the ht elements are allocated in a pool, and
+ we will destroy the pool in one single operation. */
+static void htelem_dummyfree(void* ht_elem)
+{
+}
+
+void VG_(freezeDedupPA) (DedupPoolAlloc *ddpa)
+{
+ if (VG_(clo_stats)
+ && (VG_(clo_verbosity) > 2 || VG_(debugLog_getLevel) () >= 2)) {
+ print_stats(ddpa);
+ }
+ VG_(HT_destruct) ( ddpa->ht_elements, htelem_dummyfree);
+ ddpa->ht_elements = NULL;
+ VG_(deletePA) (ddpa->ht_node_pa);
+ ddpa->ht_node_pa = NULL;
+}
+
+void* VG_(allocEltDedupPA) (DedupPoolAlloc *ddpa, SizeT eltSzB, const void *elt)
+{
+ ht_node ht_elt;
+ void* elt_ins;
+ ht_node *ht_ins;
+ vg_assert(ddpa);
+ vg_assert(ddpa->ht_elements);
+ vg_assert (eltSzB <= ddpa->poolSzB);
+
+ ddpa->nr_alloc_calls++;
+
+ // Currently using adler32 as hash function.
+ // Many references tells adler32 is bad as a hash function.
+ // And effectively, some tests on dwarf debug string shows
+ // a lot of collisions (at least for short elements).
+ // (A lot can be 10% of the elements colliding, even on
+ // small nr of elements such as 10_000).
+ ht_elt.key = VG_(adler32) (0, NULL, 0);
+ ht_elt.key = VG_(adler32) (ht_elt.key, (UChar*)elt, eltSzB);
+
+ ht_elt.eltSzB = eltSzB;
+ ht_elt.elt = (UChar*) elt;
+
+ ht_ins = VG_(HT_gen_lookup) (ddpa->ht_elements, &ht_elt, cmp_pool_elt);
+ if (ht_ins)
+ return ht_ins->elt;
+
+ /* Not found -> we need to allocate a new element from the pool
+ and insert it in the hash table of inserted elements. */
+
+ // Add a new pool if not enough space in the current pool
+ if (UNLIKELY(ddpa->curpool_free + eltSzB - 1 > ddpa->curpool_limit)) {
+ ddpa_add_new_pool(ddpa);
+ }
+
+ elt_ins = ddpa->curpool_free;
+ VG_(memcpy)(elt_ins, elt, eltSzB);
+ ddpa->curpool_free = ddpa->curpool_free + eltSzB;
+ ddpa_align_curpool_free (ddpa);
+
+ ht_ins = VG_(allocEltPA) (ddpa->ht_node_pa);
+ ht_ins->key = ht_elt.key;
+ ht_ins->eltSzB = eltSzB;
+ ht_ins->elt = elt_ins;
+ VG_(HT_add_node)(ddpa->ht_elements, ht_ins);
+ return elt_ins;
+}
#include "pub_core_debuglog.h"
#include "pub_core_hashtable.h"
#include "pub_core_libcassert.h"
+#include "pub_core_libcprint.h"
#include "pub_core_mallocfree.h"
/*--------------------------------------------------------------------*/
table->iterOK = False;
}
-/* Looks up a VgHashNode in the table. Returns NULL if not found. */
+/* Looks up a VgHashNode by key in the table. Returns NULL if not found. */
void* VG_(HT_lookup) ( VgHashTable table, UWord key )
{
VgHashNode* curr = table->chains[ CHAIN_NO(key, table) ];
return NULL;
}
+/* Looks up a VgHashNode by node in the table. Returns NULL if not found.
+ GEN!!! marks the lines that differs from VG_(HT_lookup). */
+void* VG_(HT_gen_lookup) ( VgHashTable table, void* node, HT_Cmp_t cmp )
+{
+ VgHashNode* hnode = (VgHashNode*) node; // GEN!!!
+ VgHashNode* curr = table->chains[ CHAIN_NO(hnode->key, table) ]; // GEN!!!
+
+ while (curr) {
+ if (cmp (hnode, curr) == 0) { // GEN!!!
+ return curr;
+ }
+ curr = curr->next;
+ }
+ return NULL;
+}
+
/* Removes a VgHashNode from the table. Returns NULL if not found. */
void* VG_(HT_remove) ( VgHashTable table, UWord key )
{
return NULL;
}
+/* Removes a VgHashNode by node from the table. Returns NULL if not found.
+ GEN!!! marks the lines that differs from VG_(HT_remove). */
+void* VG_(HT_gen_remove) ( VgHashTable table, void* node, HT_Cmp_t cmp )
+{
+ VgHashNode* hnode = (VgHashNode*) node; // GEN!!!
+ UWord chain = CHAIN_NO(hnode->key, table); // GEN!!!
+ VgHashNode* curr = table->chains[chain];
+ VgHashNode** prev_next_ptr = &(table->chains[chain]);
+
+ /* Table has been modified; hence HT_Next should assert. */
+ table->iterOK = False;
+
+ while (curr) {
+ if (cmp(hnode, curr) == 0) { // GEN!!!
+ *prev_next_ptr = curr->next;
+ table->n_elements--;
+ return curr;
+ }
+ prev_next_ptr = &(curr->next);
+ curr = curr->next;
+ }
+ return NULL;
+}
+
+void VG_(HT_print_stats) ( VgHashTable table, HT_Cmp_t cmp )
+{
+ #define MAXOCCUR 20
+ UInt elt_occurences[MAXOCCUR];
+ UInt key_occurences[MAXOCCUR];
+ UInt cno_occurences[MAXOCCUR];
+ /* Key occurence : how many ht elements have the same key.
+ elt_occurences : how many elements are inserted multiple time.
+ cno_occurences : how many chains have that length.
+ The last entry in these arrays collects all occurences >= MAXOCCUR-1. */
+ #define INCOCCUR(occur,n) (n >= MAXOCCUR ? occur[n-1]++ : occur[n]++)
+ UInt i;
+ UInt nkey, nelt, ncno;
+ VgHashNode *cnode, *node;
+
+ for (i = 0; i < 20; i++) {
+ key_occurences[i] = 0;
+ elt_occurences[i] = 0;
+ cno_occurences[i] = 0;
+ }
+
+ // Note that the below algorithm is quadractic in nr of elements in a chain
+ // but if that happens, the hash table/function is really bad and that
+ // should be fixed.
+ for (i = 0; i < table->n_chains; i++) {
+ ncno = 0;
+ for (cnode = table->chains[i]; cnode != NULL; cnode = cnode->next) {
+ ncno++;
+
+ nkey = 0;
+ // Is the same cnode->key existing before cnode ?
+ for (node = table->chains[i]; node != cnode; node = node->next) {
+ if (node->key == cnode->key)
+ nkey++;
+ }
+ // If cnode->key not in a previous node, count occurences of key.
+ if (nkey == 0) {
+ for (node = cnode; node != NULL; node = node->next) {
+ if (node->key == cnode->key)
+ nkey++;
+ }
+ INCOCCUR(key_occurences, nkey);
+ }
+
+ nelt = 0;
+ // Is the same cnode element existing before cnode ?
+ for (node = table->chains[i]; node != cnode; node = node->next) {
+ if (cmp) {
+ if ((*cmp)(node, cnode) == 0)
+ nelt++;
+ } else
+ if (node->key == cnode->key)
+ nelt++;
+ }
+ // If cnode element not in a previous node, count occurences of elt.
+ if (nelt == 0) {
+ for (node = cnode; node != NULL; node = node->next) {
+ if (cmp) {
+ if ((*cmp)(node, cnode) == 0)
+ nelt++;
+ } else
+ if (node->key == cnode->key)
+ nelt++;
+ }
+ INCOCCUR(elt_occurences, nelt);
+ }
+ }
+ INCOCCUR(cno_occurences, ncno);
+ }
+
+ VG_(message)(Vg_DebugMsg,
+ "nr occurences of"
+ " chains of len N,"
+ " N-plicated keys,"
+ " N-plicated elts\n");
+ nkey = nelt = ncno = 0;
+ for (i = 0; i < MAXOCCUR; i++) {
+ if (elt_occurences[i] > 0 || key_occurences[i] > 0 || cno_occurences[i] > 0)
+ VG_(message)(Vg_DebugMsg,
+ "N:%2d : nr chain %6d, nr keys %6d, nr elts %6d\n",
+ i, cno_occurences[i], key_occurences[i], elt_occurences[i]);
+ nkey += key_occurences[i];
+ nelt += elt_occurences[i];
+ ncno += cno_occurences[i];
+ }
+ VG_(message)(Vg_DebugMsg, "total nr of unique chains: %6d, keys %6d, elts %6d\n",
+ ncno, nkey, nelt);
+}
+
+
/* Allocates a suitably-sized array, copies pointers to all the hashtable
elements into it, then returns both the array and the size of it. The
array must be freed with VG_(free).
--- /dev/null
+
+/*--------------------------------------------------------------------*/
+/*--- A pool (memory) allocator that avoids duplicated copies. ---*/
+/*--- pub_core_deduppoolalloc.h ---*/
+/*--------------------------------------------------------------------*/
+
+/*
+ This file is part of Valgrind, a dynamic binary instrumentation
+ framework.
+
+ Copyright (C) 2014-2014 Philippe Waroquiers philippe.waroquiers@skynet.be
+
+ This program is free software; you can redistribute it and/or
+ modify it under the terms of the GNU General Public License as
+ published by the Free Software Foundation; either version 2 of the
+ License, or (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful, but
+ WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, write to the Free Software
+ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
+ 02111-1307, USA.
+
+ The GNU General Public License is contained in the file COPYING.
+*/
+
+#ifndef __PUB_CORE_DEDUPPOOLALLOC_H
+#define __PUB_CORE_DEDUPPOOLALLOC_H
+
+#include "pub_tool_deduppoolalloc.h"
+
+// No core-only exports; everything in this module is visible to both
+// the core and tools.
+
+#endif // __PUB_CORE_DEDUPPOOLALLOC_H
+
+/*--------------------------------------------------------------------*/
+/*--- end ---*/
+/*--------------------------------------------------------------------*/
pub_tool_aspacemgr.h \
pub_tool_clientstate.h \
pub_tool_clreq.h \
+ pub_tool_deduppoolalloc.h \
pub_tool_debuginfo.h \
pub_tool_errormgr.h \
pub_tool_execontext.h \
--- /dev/null
+
+/*--------------------------------------------------------------------*/
+/*--- A pool (memory) allocator that avoids duplicated copies. ---*/
+/*--- pub_tool_deduppoolalloc.h ---*/
+/*--------------------------------------------------------------------*/
+
+/*
+ This file is part of Valgrind, a dynamic binary instrumentation
+ framework.
+
+ Copyright (C) 2014-2014 Philippe Waroquiers philippe.waroquiers@skynet.be
+
+ This program is free software; you can redistribute it and/or
+ modify it under the terms of the GNU General Public License as
+ published by the Free Software Foundation; either version 2 of the
+ License, or (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful, but
+ WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, write to the Free Software
+ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
+ 02111-1307, USA.
+
+ The GNU General Public License is contained in the file COPYING.
+*/
+
+#ifndef __PUB_TOOL_DEDUPPOOLALLOC_H
+#define __PUB_TOOL_DEDUPPOOLALLOC_H
+
+#include "pub_tool_basics.h" // UWord
+
+//-----------------------------------------------------------------------------
+// PURPOSE: Provides a pool allocator for elements, storing only once identical
+// elements. In other words, this can be considered a "dictionary" of elements.
+//
+// This pool allocator manages elements allocation by allocating "pools" of
+// many elements from a lower level allocator (typically pub_tool_mallocfree.h).
+// Single elements are allocated from these pools.
+// Currently, elements can only be allocated, elements cannot be freed
+// individually.
+// Once allocated, an element must not be modified anymore.
+//
+// A dedup pool allocator has significantly less memory overhead than
+// calling directly pub_tool_mallocfree.h if the deduplication factor
+// is big. However, allocating an element incurs a cost for searching
+// if an identical element is already in the pool.
+//
+// Note: the elements of the pool cannot be freed (at least currently).
+// The only way to free the elements is to delete the pool allocator.
+//--------------------------------------------------------------------
+
+
+typedef struct _DedupPoolAlloc DedupPoolAlloc;
+
+/* Create new DedupPoolAlloc, using given allocation and free function.
+ Alloc fn must not fail (that is, if it returns it must have succeeded.)
+ poolSzB is the (minimum) size in bytes of the pool of elements allocated
+ with alloc.
+ eltAlign is the minimum required alignement for the elements allocated
+ from the DedupPoolAlloc. */
+extern DedupPoolAlloc* VG_(newDedupPA) ( SizeT poolSzB,
+ SizeT eltAlign,
+ void* (*alloc)(const HChar*, SizeT),
+ const HChar* cc,
+ void (*free_fn)(void*) );
+
+/* Allocates a new element from ddpa with eltSzB bytes to store elt. */
+extern void* VG_(allocEltDedupPA) (DedupPoolAlloc *ddpa,
+ SizeT eltSzB, const void *elt);
+
+
+/* The Dedup Pool Allocator must maintain a data structure to avoid
+ duplicates as long as new elements can be allocated from the pool.
+ Once no new elements will be allocated, this dedup data structure
+ can be released using VG_(freezeDedupPA). Once ddpa has been frozen,
+ it is an error to call VG_(allocEltDedupPA). */
+extern void VG_(freezeDedupPA) (DedupPoolAlloc *ddpa);
+
+/* Free all memory associated with a DedupPoolAlloc. */
+extern void VG_(deleteDedupPA) ( DedupPoolAlloc *ddpa);
+
+#endif // __PUB_TOOL_DEDUPPOOLALLOC_
+
+/*--------------------------------------------------------------------*/
+/*--- end pub_tool_deduppoolalloc.h ---*/
+/*--------------------------------------------------------------------*/
/* Add a node to the table. Duplicate keys are permitted. */
extern void VG_(HT_add_node) ( VgHashTable t, void* node );
-/* Looks up a VgHashNode in the table. Returns NULL if not found. If entries
+/* Looks up a VgHashNode by key in the table.
+ * Returns NULL if not found. If entries
* with duplicate keys are present, the most recently-added of the dups will
* be returned, but it's probably better to avoid dups altogether. */
extern void* VG_(HT_lookup) ( VgHashTable table, UWord key );
-/* Removes a VgHashNode from the table. Returns NULL if not found. */
+/* Removes a VgHashNode by key from the table. Returns NULL if not found. */
extern void* VG_(HT_remove) ( VgHashTable table, UWord key );
+typedef Word (*HT_Cmp_t) ( const void* node1, const void* node2 );
+
+/* Same as VG_(HT_lookup) and VG_(HT_remove), but allowing a part of or
+ the full element to be compared for equality, not only the key.
+ The typical use for the below function is to store a hash value of the
+ element in the key, and have the comparison function checking for equality
+ of the full element data.
+ Attention about the comparison function:
+ * It must *not* compare the 'next' pointer.
+ * when comparing the rest of the node, if the node data contains holes
+ between components, either the node memory should be fully initialised
+ (e.g. allocated using VG_(calloc)) or each component should be compared
+ individually. */
+extern void* VG_(HT_gen_lookup) ( VgHashTable table, void* node, HT_Cmp_t cmp );
+extern void* VG_(HT_gen_remove) ( VgHashTable table, void* node, HT_Cmp_t cmp );
+
+/* Output detailed usage/collision statistics.
+ cmp will be used to verify if 2 elements with the same key are equal.
+ Use NULL cmp if the hash table elements are only to be compared by key. */
+extern void VG_(HT_print_stats) ( VgHashTable table, HT_Cmp_t cmp );
+
/* Allocates a suitably-sized array, copies pointers to all the hashtable
elements into it, then returns both the array and the size of it. The
array must be freed with VG_(free). */
/* Create new PoolAlloc, using given allocation and free function, and
for elements of the specified size. Alloc fn must not fail (that
is, if it returns it must have succeeded.) */
-PoolAlloc* VG_(newPA) ( UWord elemSzB,
- UWord nPerPool,
- void* (*alloc)(const HChar*, SizeT),
- const HChar* cc,
- void (*free_fn)(void*) );
+extern PoolAlloc* VG_(newPA) ( UWord elemSzB,
+ UWord nPerPool,
+ void* (*alloc)(const HChar*, SizeT),
+ const HChar* cc,
+ void (*free_fn)(void*) );
/* Free all memory associated with a PoolAlloc. */
projects / thirdparty / valgrind.git / commitdiff
