Improve lookup_type_cache() handling on out-of-memory errors

If an error happens during the initialization of the TYPEOID catcache,
as part of lookup_type_cache(), the error handling of that lookup would
cause an assertion failure via finalize_in_progress_typentries(), called
during error recovery, the presence of an in-progress type OID causing
a catcache initialization outside of a transaction context.

The in-progress list is now delayed to happen after the initial entry
lookup.  Alexander Lakhin has found a fancy way to reproduce the
problem, with the injection of probabilistic memory allocation failures.

This problem is unlikely going to show up in practice.  Like the other
changes of this kind, no backpatch is done.

Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/95c64dc2-3abe-4f4e-b285-4c681f565d9f@gmail.com

diff --git a/src/backend/utils/cache/typcache.c b/src/backend/utils/cache/typcache.c
index 650b5d9bad90d338f42bbe462523a96260f1b2d4..00133ba92ef6b3483bf6b4334fd0e35453837ccf 100644 (file)
--- a/src/backend/utils/cache/typcache.c
+++ b/src/backend/utils/cache/typcache.c
@@ -459,13 +459,27 @@ lookup_type_cache(Oid type_id, int flags)
                                                                        allocsize * sizeof(*in_progress_list));
                in_progress_list_maxlen = allocsize;
        }
-       in_progress_offset = in_progress_list_len++;
-       in_progress_list[in_progress_offset] = type_id;
 
        /* Try to look up an existing entry */
        typentry = (TypeCacheEntry *) hash_search(TypeCacheHash,
                                                                                          &type_id,
                                                                                          HASH_FIND, NULL);
+
+       /*
+        * Only mark the new entry as "in progress" after the initial entry
+        * lookup.
+        *
+        * TypeCacheHash uses type_cache_syshash(), potentially triggering the
+        * initialization of the TYPEOID catcache, where an out-of-memory failure
+        * is possible.  If an out-of-memory happens, error recovery would call
+        * finalize_in_progress_typentries(), that could attempt a catcache
+        * initialization again outside a transaction context.
+        *
+        * See also ConditionalCatalogCacheInitializeCache().
+        */
+       in_progress_offset = in_progress_list_len++;
+       in_progress_list[in_progress_offset] = type_id;
+
        if (typentry == NULL)
        {
                /*