Prevent shared memory hash tables from growing beyond initial size

Set HASH_FIXED_SIZE on all shared memory hash tables, to prevent them
from growing after the initial allocation. It was always weirdly
indeterministic that if one hash table used up all the unused shared
memory, you could not use that space for other things anymore until
restart. We just got rid of that behavior for the LOCK and PROCLOCK
tables, but it's similarly weird for all other hash tables.

Increase SHMEM_INDEX_SIZE because we were already above the max size,
on that one, and it's now a hard limit.

Some callers of ShmemInitHash() still pass HASH_FIXED_SIZE, but that's
now unnecessary. They should perhaps now be removed, but it doesn't do
any harm either to pass it.

Reviewed-by: Tomas Vondra <tomas@vondra.me>
Discussion: https://www.postgresql.org/message-id/01ab1d41-3eda-4705-8bbd-af898f5007f1@iki.fi

diff --git a/src/backend/storage/ipc/shmem.c b/src/backend/storage/ipc/shmem.c
index 85757a35823a558773b885fbb89a6fb3e670c606..8e002f5c7a616eab5d78c05c201aa0d437bdfa3f 100644 (file)
--- a/src/backend/storage/ipc/shmem.c
+++ b/src/backend/storage/ipc/shmem.c
@@ -24,9 +24,8 @@
  *     available to POSTGRES: fixed-size structures, queues and hash
  *     tables.  Fixed-size structures contain things like global variables
  *     for a module and should never be allocated after the shared memory
- *     initialization phase.  Hash tables have a fixed maximum size, but
- *     their actual size can vary dynamically.  When entries are added
- *     to the table, more space is allocated.  Queues link data structures
+ *     initialization phase.  Hash tables have a fixed maximum size and
+ *     cannot grow beyond that.  Queues link data structures
  *     that have been allocated either within fixed-size structures or as hash
  *     buckets.  Each shared data structure has a string name to identify
  *     it (assigned in the module that declares it).
@@ -56,11 +55,7 @@
  *
  *             (d) memory allocation model: shared memory can never be
  *     freed, once allocated.   Each hash table has its own free list,
- *     so hash buckets can be reused when an item is deleted.  However,
- *     if one hash table grows very large and then shrinks, its space
- *     cannot be redistributed to other tables.  We could build a simple
- *     hash bucket garbage collector if need be.  Right now, it seems
- *     unnecessary.
+ *     so hash buckets can be reused when an item is deleted.
  */
 
 #include "postgres.h"
@@ -187,7 +182,7 @@ InitShmemAllocator(PGShmemHeader *seghdr)
        info.dsize = info.max_dsize = hash_select_dirsize(SHMEM_INDEX_SIZE);
        info.alloc = ShmemHashAlloc;
        info.alloc_arg = NULL;
-       hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
+       hash_flags = HASH_ELEM | HASH_STRINGS | HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE;
        if (!IsUnderPostmaster)
        {
                size = hash_get_shared_size(&info, hash_flags);
@@ -335,7 +330,7 @@ ShmemAddrIsValid(const void *addr)
  * *infoP and hash_flags must specify at least the entry sizes and key
  * comparison semantics (see hash_create()).  Flag bits and values specific
  * to shared-memory hash tables are added here, except that callers may
- * choose to specify HASH_PARTITION and/or HASH_FIXED_SIZE.
+ * choose to specify HASH_PARTITION.
  *
  * Note: before Postgres 9.0, this function returned NULL for some failure
  * cases.  Now, it always throws error instead, so callers need not check
@@ -353,14 +348,15 @@ ShmemInitHash(const char *name,           /* table string name for shmem index */
        /*
         * Hash tables allocated in shared memory have a fixed directory; it can't
         * grow or other backends wouldn't be able to find it. So, make sure we
-        * make it big enough to start with.
+        * make it big enough to start with.  We also allocate all the buckets
+        * upfront.
         *
         * The shared memory allocator must be specified too.
         */
        infoP->dsize = infoP->max_dsize = hash_select_dirsize(nelems);
        infoP->alloc = ShmemHashAlloc;
        infoP->alloc_arg = NULL;
-       hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE;
+       hash_flags |= HASH_SHARED_MEM | HASH_ALLOC | HASH_DIRSIZE | HASH_FIXED_SIZE;
 
        /* look it up in the shmem index */
        location = ShmemInitStruct(name,

diff --git a/src/include/storage/shmem.h b/src/include/storage/shmem.h
index 0ae609aca8b9a5d5bd27ec3f3776f18fa956518e..2fccbdb534c477e118de3a684a141cd7eb87943b 100644 (file)
--- a/src/include/storage/shmem.h
+++ b/src/include/storage/shmem.h
@@ -46,8 +46,8 @@ extern void RequestAddinShmemSpace(Size size);
 /* size constants for the shmem index table */
  /* max size of data structure string name */
 #define SHMEM_INDEX_KEYSIZE             (48)
- /* estimated size of the shmem index table (not a hard limit) */
-#define SHMEM_INDEX_SIZE                (64)
+ /* max number of named shmem structures and hash tables */
+#define SHMEM_INDEX_SIZE                (256)
 
 /* this is a hash bucket in the shmem index table */
 typedef struct