Fix race with timeline selection in logical decoding during promotion

During promotion, there is a window where RecoveryInProgress() returns
true but the WAL segments of the old timeline have already been removed.
A logical decoding could pick up the old timeline in this window when
reading a page, failing with the following error:
ERROR: requested WAL segment ... has already been removed

This issue does not lead to any data correctness issue, as retrying to
decode the data works in follow-up decoding attempts.  It impacts
availability, though.  Other WAL page read callbacks have a similar
issue, this commit takes care of what should be the noisiest code path:
logical decoding with START_REPLICATION in a WAL sender.

A TAP test, based on an injection point waiting in the startup process
after the segments have been removed/recycled, is added.  This part is
backpatched down to v17.

This issue has been causing sporadic failures in the buildfarm, and
was reproducible manually.  This issue happens since logical decoding on
standbys exists, down to v16.

Reported-by: Alexander Lakhin <exclusion@gmail.com>
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Xuneng Zhou <xunengzhou@gmail.com>
Discussion: https://postgr.es/m/7daef094-abf3-4672-bc23-3df4763b16a3@gmail.com
Backpatch-through: 16

diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index b37f1d19b270b8138b3a63a58803053a276be755..fcf22b7cd1442b237dc3b63d016b5d5569d2e11c 100644 (file)
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -941,7 +941,29 @@ logical_read_xlog_page(XLogReaderState *state, XLogRecPtr targetPagePtr, int req
        am_cascading_walsender = RecoveryInProgress();
 
        if (am_cascading_walsender)
-               GetXLogReplayRecPtr(&currTLI);
+       {
+               TimeLineID      insertTLI;
+
+               /*
+                * If the insertion timeline has already been set, use it.
+                * InsertTimeLineID is set before the WAL segments of the old timeline
+                * are removed, before SharedRecoveryState switches to
+                * RECOVERY_STATE_DONE.
+                *
+                * There is a window where RecoveryInProgress() still returns true but
+                * the old timeline's WAL segments have already been removed or
+                * recycled.  Using the WAL insertion timeline avoids attempting to
+                * read from those removed segments, improving availability, and is a
+                * safe thing to do as promotion copies the contents in the last
+                * segment of the old timeline to the first segment of the new
+                * timeline, up to the switchpoint.
+                */
+               insertTLI = GetWALInsertionTimeLineIfSet();
+               if (insertTLI != 0)
+                       currTLI = insertTLI;
+               else
+                       GetXLogReplayRecPtr(&currTLI);
+       }
        else
                currTLI = GetWALInsertionTimeLine();