Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 4914864

Browse filesBrowse files
alvherrealvherre
committed
Add a test framework for recovery
This long-awaited framework is an expansion of the existing PostgresNode stuff to support additional features for recovery testing; the recovery tests included in this commit are a starting point that cover some of the recovery features we have. More scripts are expected to be added later. Author: Michaël Paquier, a bit of help from Amir Rohan Reviewed by: Amir Rohan, Stas Kelvich, Kyotaro Horiguchi, Victor Wagner, Craig Ringer, Álvaro Herrera Discussion: http://www.postgresql.org/message-id/CAB7nPqTf7V6rswrFa=q_rrWeETUWagP=h8LX8XAov2Jcxw0DRg@mail.gmail.com Discussion: http://www.postgresql.org/message-id/trinity-b4a8035d-59af-4c42-a37e-258f0f28e44a-1443795007012@3capp-mailcom-lxa08
1 parent 89ac700 commit 4914864
Copy full SHA for 4914864

File tree

Expand file treeCollapse file tree

12 files changed

+501
-5
lines changed
Filter options
Expand file treeCollapse file tree

12 files changed

+501
-5
lines changed

‎doc/src/sgml/install-windows.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/install-windows.sgml
+3-1Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -440,6 +440,7 @@ $ENV{CONFIG}="Debug";
440440
<userinput>vcregress ecpgcheck</userinput>
441441
<userinput>vcregress isolationcheck</userinput>
442442
<userinput>vcregress bincheck</userinput>
443+
<userinput>vcregress recoverycheck</userinput>
443444
<userinput>vcregress upgradecheck</userinput>
444445
</screen>
445446

@@ -455,7 +456,8 @@ $ENV{CONFIG}="Debug";
455456

456457
<para>
457458
Running the regression tests on client programs, with "vcregress bincheck",
458-
requires an additional Perl module to be installed:
459+
or on recovery tests, with "vcregress recoverycheck" requires an additional
460+
Perl module to be installed:
459461
<variablelist>
460462
<varlistentry>
461463
<term><productname>IPC::Run</productname></term>

‎src/test/Makefile

Copy file name to clipboardExpand all lines: src/test/Makefile
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ subdir = src/test
1212
top_builddir = ../..
1313
include $(top_builddir)/src/Makefile.global
1414

15-
SUBDIRS = regress isolation modules
15+
SUBDIRS = regress isolation modules recovery
1616

1717
# We don't build or execute examples/, locale/, or thread/ by default,
1818
# but we do want "make clean" etc to recurse into them. Likewise for ssl/,

‎src/test/perl/PostgresNode.pm

Copy file name to clipboardExpand all lines: src/test/perl/PostgresNode.pm
+72-2Lines changed: 72 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -346,6 +346,9 @@ On Windows, we use SSPI authentication to ensure the same (by pg_regress
346346
pg_hba.conf is configured to allow replication connections. Pass the keyword
347347
parameter hba_permit_replication => 0 to disable this.
348348
349+
WAL archiving can be enabled on this node by passing the keyword parameter
350+
has_archiving => 1. This is disabled by default.
351+
349352
postgresql.conf can be set up for replication by passing the keyword
350353
parameter allows_streaming => 1. This is disabled by default.
351354
@@ -364,6 +367,7 @@ sub init
364367
$params{hba_permit_replication} = 1
365368
unless defined $params{hba_permit_replication};
366369
$params{allows_streaming} = 0 unless defined $params{allows_streaming};
370+
$params{has_archiving} = 0 unless defined $params{has_archiving};
367371

368372
mkdir $self->backup_dir;
369373
mkdir $self->archive_dir;
@@ -401,6 +405,7 @@ sub init
401405
close $conf;
402406

403407
$self->set_replication_conf if $params{hba_permit_replication};
408+
$self->enable_archiving if $params{has_archiving};
404409
}
405410

406411
=pod
@@ -458,13 +463,20 @@ Initialize a node from a backup, which may come from this node or a different
458463
node. root_node must be a PostgresNode reference, backup_name the string name
459464
of a backup previously created on that node with $node->backup.
460465
461-
Does not start the node after init.
466+
Does not start the node after initializing it.
462467
463468
A recovery.conf is not created.
464469
470+
pg_hba.conf is configured to allow replication connections. Pass the keyword
471+
parameter hba_permit_replication => 0 to disable this.
472+
465473
Streaming replication can be enabled on this node by passing the keyword
466474
parameter has_streaming => 1. This is disabled by default.
467475
476+
Restoring WAL segments from archives using restore_command can be enabled
477+
by passiong the keyword parameter has_restoring => 1. This is disabled by
478+
default.
479+
468480
The backup is copied, leaving the original unmodified. pg_hba.conf is
469481
unconditionally set to enable replication connections.
470482
@@ -479,6 +491,10 @@ sub init_from_backup
479491
my $root_name = $root_node->name;
480492

481493
$params{has_streaming} = 0 unless defined $params{has_streaming};
494+
$params{hba_permit_replication} = 1
495+
unless defined $params{hba_permit_replication};
496+
$params{has_restoring} = 0 unless defined $params{has_restoring};
497+
482498
print
483499
"# Initializing node \"$node_name\" from backup \"$backup_name\" of node \"$root_name\"\n";
484500
die "Backup \"$backup_name\" does not exist at $backup_path"
@@ -498,8 +514,9 @@ sub init_from_backup
498514
qq(
499515
port = $port
500516
));
501-
$self->set_replication_conf;
517+
$self->set_replication_conf if $params{hba_permit_replication};
502518
$self->enable_streaming($root_node) if $params{has_streaming};
519+
$self->enable_restoring($root_node) if $params{has_restoring};
503520
}
504521

505522
=pod
@@ -608,6 +625,59 @@ standby_mode=on
608625
));
609626
}
610627

628+
# Internal routine to enable archive recovery command on a standby node
629+
sub enable_restoring
630+
{
631+
my ($self, $root_node) = @_;
632+
my $path = $root_node->archive_dir;
633+
my $name = $self->name;
634+
635+
print "### Enabling WAL restore for node \"$name\"\n";
636+
637+
# On Windows, the path specified in the restore command needs to use
638+
# double back-slashes to work properly and to be able to detect properly
639+
# the file targeted by the copy command, so the directory value used
640+
# in this routine, using only one back-slash, need to be properly changed
641+
# first. Paths also need to be double-quoted to prevent failures where
642+
# the path contains spaces.
643+
$path =~ s{\\}{\\\\}g if ($TestLib::windows_os);
644+
my $copy_command = $TestLib::windows_os ?
645+
qq{copy "$path\\\\%f" "%p"} :
646+
qq{cp $path/%f %p};
647+
648+
$self->append_conf('recovery.conf', qq(
649+
restore_command = '$copy_command'
650+
standby_mode = on
651+
));
652+
}
653+
654+
# Internal routine to enable archiving
655+
sub enable_archiving
656+
{
657+
my ($self) = @_;
658+
my $path = $self->archive_dir;
659+
my $name = $self->name;
660+
661+
print "### Enabling WAL archiving for node \"$name\"\n";
662+
663+
# On Windows, the path specified in the restore command needs to use
664+
# double back-slashes to work properly and to be able to detect properly
665+
# the file targeted by the copy command, so the directory value used
666+
# in this routine, using only one back-slash, need to be properly changed
667+
# first. Paths also need to be double-quoted to prevent failures where
668+
# the path contains spaces.
669+
$path =~ s{\\}{\\\\}g if ($TestLib::windows_os);
670+
my $copy_command = $TestLib::windows_os ?
671+
qq{copy "%p" "$path\\\\%f"} :
672+
qq{cp %p $path/%f};
673+
674+
# Enable archive_mode and archive_command on node
675+
$self->append_conf('postgresql.conf', qq(
676+
archive_mode = on
677+
archive_command = '$copy_command'
678+
));
679+
}
680+
611681
# Internal method
612682
sub _update_pid
613683
{

‎src/test/recovery/.gitignore

Copy file name to clipboard
+3Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Generated by test suite
2+
/regress_log/
3+
/tmp_check/

‎src/test/recovery/Makefile

Copy file name to clipboard
+17Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#-------------------------------------------------------------------------
2+
#
3+
# Makefile for src/test/recovery
4+
#
5+
# Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
6+
# Portions Copyright (c) 1994, Regents of the University of California
7+
#
8+
# src/test/recovery/Makefile
9+
#
10+
#-------------------------------------------------------------------------
11+
12+
subdir = src/test/recovery
13+
top_builddir = ../../..
14+
include $(top_builddir)/src/Makefile.global
15+
16+
check:
17+
$(prove_check)

‎src/test/recovery/README

Copy file name to clipboard
+21Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
src/test/recovery/README
2+
3+
Regression tests for recovery and replication
4+
=============================================
5+
6+
This directory contains a test suite for recovery and replication,
7+
testing mainly the interactions of recovery.conf with cluster
8+
instances by providing a simple set of routines that can be used
9+
to define a custom cluster for a test, including backup, archiving,
10+
and streaming configuration.
11+
12+
Running the tests
13+
=================
14+
15+
make check
16+
17+
NOTE: This creates a temporary installation, and some tests may
18+
create one or multiple nodes, be they master or standby(s) for the
19+
purpose of the tests.
20+
21+
NOTE: This requires the --enable-tap-tests argument to configure.

‎src/test/recovery/t/001_stream_rep.pl

Copy file name to clipboard
+70Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Minimal test testing streaming replication
2+
use strict;
3+
use warnings;
4+
use PostgresNode;
5+
use TestLib;
6+
use Test::More tests => 4;
7+
8+
# Initialize master node
9+
my $node_master = get_new_node('master');
10+
$node_master->init(allows_streaming => 1);
11+
$node_master->start;
12+
my $backup_name = 'my_backup';
13+
14+
# Take backup
15+
$node_master->backup($backup_name);
16+
17+
# Create streaming standby linking to master
18+
my $node_standby_1 = get_new_node('standby_1');
19+
$node_standby_1->init_from_backup($node_master, $backup_name,
20+
has_streaming => 1);
21+
$node_standby_1->start;
22+
23+
# Take backup of standby 1 (not mandatory, but useful to check if
24+
# pg_basebackup works on a standby).
25+
$node_standby_1->backup($backup_name);
26+
27+
# Create second standby node linking to standby 1
28+
my $node_standby_2 = get_new_node('standby_2');
29+
$node_standby_2->init_from_backup($node_standby_1, $backup_name,
30+
has_streaming => 1);
31+
$node_standby_2->start;
32+
33+
# Create some content on master and check its presence in standby 1
34+
$node_master->psql('postgres',
35+
"CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a");
36+
37+
# Wait for standbys to catch up
38+
my $applname_1 = $node_standby_1->name;
39+
my $applname_2 = $node_standby_2->name;
40+
my $caughtup_query =
41+
"SELECT pg_current_xlog_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_1';";
42+
$node_master->poll_query_until('postgres', $caughtup_query)
43+
or die "Timed out while waiting for standby 1 to catch up";
44+
$caughtup_query =
45+
"SELECT pg_last_xlog_replay_location() = write_location FROM pg_stat_replication WHERE application_name = '$applname_2';";
46+
$node_standby_1->poll_query_until('postgres', $caughtup_query)
47+
or die "Timed out while waiting for standby 2 to catch up";
48+
49+
my $result =
50+
$node_standby_1->psql('postgres', "SELECT count(*) FROM tab_int");
51+
print "standby 1: $result\n";
52+
is($result, qq(1002), 'check streamed content on standby 1');
53+
54+
$result = $node_standby_2->psql('postgres', "SELECT count(*) FROM tab_int");
55+
print "standby 2: $result\n";
56+
is($result, qq(1002), 'check streamed content on standby 2');
57+
58+
# Check that only READ-only queries can run on standbys
59+
$node_standby_1->command_fails(
60+
[ 'psql', '-A',
61+
'-t', '--no-psqlrc',
62+
'-d', $node_standby_1->connstr,
63+
'-c', "INSERT INTO tab_int VALUES (1)" ],
64+
'Read-only queries on standby 1');
65+
$node_standby_2->command_fails(
66+
[ 'psql', '-A',
67+
'-t', '--no-psqlrc',
68+
'-d', $node_standby_2->connstr,
69+
'-c', "INSERT INTO tab_int VALUES (1)" ],
70+
'Read-only queries on standby 2');

‎src/test/recovery/t/002_archiving.pl

Copy file name to clipboard
+52Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# test for archiving with warm standby
2+
use strict;
3+
use warnings;
4+
use PostgresNode;
5+
use TestLib;
6+
use Test::More tests => 1;
7+
use File::Copy;
8+
9+
# Initialize master node, doing archives
10+
my $node_master = get_new_node('master');
11+
$node_master->init(
12+
has_archiving => 1,
13+
allows_streaming => 1);
14+
my $backup_name = 'my_backup';
15+
16+
# Start it
17+
$node_master->start;
18+
19+
# Take backup for slave
20+
$node_master->backup($backup_name);
21+
22+
# Initialize standby node from backup, fetching WAL from archives
23+
my $node_standby = get_new_node('standby');
24+
$node_standby->init_from_backup($node_master, $backup_name,
25+
has_restoring => 1);
26+
$node_standby->append_conf(
27+
'postgresql.conf', qq(
28+
wal_retrieve_retry_interval = '100ms'
29+
));
30+
$node_standby->start;
31+
32+
# Create some content on master
33+
$node_master->psql('postgres',
34+
"CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a");
35+
my $current_lsn =
36+
$node_master->psql('postgres', "SELECT pg_current_xlog_location();");
37+
38+
# Force archiving of WAL file to make it present on master
39+
$node_master->psql('postgres', "SELECT pg_switch_xlog()");
40+
41+
# Add some more content, it should not be present on standby
42+
$node_master->psql('postgres',
43+
"INSERT INTO tab_int VALUES (generate_series(1001,2000))");
44+
45+
# Wait until necessary replay has been done on standby
46+
my $caughtup_query =
47+
"SELECT '$current_lsn'::pg_lsn <= pg_last_xlog_replay_location()";
48+
$node_standby->poll_query_until('postgres', $caughtup_query)
49+
or die "Timed out while waiting for standby to catch up";
50+
51+
my $result = $node_standby->psql('postgres', "SELECT count(*) FROM tab_int");
52+
is($result, qq(1000), 'check content from archives');

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.