Fix timing-dependent failure in recovery test 004_timeline_switch

The test introduced by 17b2d5ec759c verifies that a WAL receiver
survives across a timeline jump by searching the server logs for
termination messages.  However, it called restart() before the timeline
switch, which kills the WAL receiver and may log the exact message being
checked, hence failing the test.  As TAP tests reuse the same log file
across restarts, a rotate_logfile() is used before the restart so as the
log matching check is not impacted by log entries generated by a
previous shutdown.

Recent changes to file handle inheritance altered I/O timing enough to
make this fail consistently while testing another patch.

While on it, this adds an extra check based on a PID comparison.  This
test may lead to false positives as it could be possible that the WAL
receiver has processed a timeline jump before the initial PID is
grabbed, but it should be good enough in most cases.

Like 17b2d5ec759c, backpatch down to v13.

Author: Bryan Green <dbryan.green@gmail.com>
Co-authored-by: Xuneng Zhou <xunengzhou@gmail.com>
Discussion: https://postgr.es/m/9d00b597-d64a-4f1e-802e-90f9dc394c70@gmail.com
Backpatch-through: 13

diff --git a/src/test/recovery/t/004_timeline_switch.pl b/src/test/recovery/t/004_timeline_switch.pl
index 019dadfdcd5f8f05a358d3b95d1a853d7bb2ced5..adc0ae5fc9423433f2b5619876bedd9af3892f0b 100644 (file)
--- a/src/test/recovery/t/004_timeline_switch.pl
+++ b/src/test/recovery/t/004_timeline_switch.pl
@@ -4,7 +4,7 @@ use warnings;
 use File::Path qw(rmtree);
 use PostgresNode;
 use TestLib;
-use Test::More tests => 4;
+use Test::More tests => 5;
 
 $ENV{PGDATABASE} = 'postgres';
 
@@ -55,8 +55,19 @@ $node_standby_2->append_conf(
        'postgresql.conf', qq(
 primary_conninfo='$connstr_1'
 ));
+
+# Rotate logfile before restarting, for the log checks done below.
+$node_standby_2->rotate_logfile;
 $node_standby_2->restart;
 
+# Wait for walreceiver to reconnect after the restart.  We want to
+# verify that after reconnection, the walreceiver stays alive during
+# the timeline switch.
+$node_standby_2->poll_query_until('postgres',
+       "SELECT EXISTS(SELECT 1 FROM pg_stat_wal_receiver)");
+my $wr_pid_before_switch = $node_standby_2->safe_psql('postgres',
+       "SELECT pid FROM pg_stat_wal_receiver");
+
 # Insert some data in standby 1 and check its presence in standby 2
 # to ensure that the timeline switch has been done.
 $node_standby_1->safe_psql('postgres',
@@ -77,6 +88,14 @@ ok( !$node_standby_2->log_contains(
        ),
        'WAL receiver should not be stopped across timeline jumps');
 
+# Verify that the walreceiver process stayed alive across the timeline
+# switch, check its PID.
+my $wr_pid_after_switch = $node_standby_2->safe_psql('postgres',
+       "SELECT pid FROM pg_stat_wal_receiver");
+
+is($wr_pid_before_switch, $wr_pid_after_switch,
+       'WAL receiver PID matches across timeline jumps');
+
 # Ensure that a standby is able to follow a master on a newer timeline
 # when WAL archiving is enabled.