Make GetSnapshotData() more resilient on out-of-memory errors

If the allocation of Snapshot->subxip fails, a follow-up call of
GetSnapshotData() would see a partially-initialized snapshot, causing a
NULL dereference on reentry when using "subxip" because only "xip" would
be allocated.  In the event of an out-of-memory error when allocating
"subxip", "xip" is now reset before throwing an ERROR, so as Snapshots
can be allocated and handled gracefully on retry.

This problem is unlikely going to show up in practice, so no backpatch.

Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Matthias van de Meent <boekewurm+postgres@gmail.com>
Discussion: https://postgr.es/m/e77acaac-a1b3-40b3-99ee-5769b4e453e4@gmail.com

diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index f540bb6b23f04ed272c91b7ca1724ee9bc58ab1a..60336b3180364a0d38c29f0a3899c5b093551cc9 100644 (file)
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2158,9 +2158,17 @@ GetSnapshotData(Snapshot snapshot)
                snapshot->subxip = (TransactionId *)
                        malloc(GetMaxSnapshotSubxidCount() * sizeof(TransactionId));
                if (snapshot->subxip == NULL)
+               {
+                       /*
+                        * Clean up the Snapshot state before throwing the error, so that
+                        * a retry does not see a partially-initialized snapshot.
+                        */
+                       free(snapshot->xip);
+                       snapshot->xip = NULL;
                        ereport(ERROR,
                                        (errcode(ERRCODE_OUT_OF_MEMORY),
                                         errmsg("out of memory")));
+               }
        }
 
        /*