Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit d497093

Browse filesBrowse files
committed
pg_waldump: Add --save-fullpage=PATH to save full page images from WAL records
This option extracts (potentially decompressing) full-page images included in WAL records into a given target directory. These images are subject to the same filtering rules as the normal display of the WAL records, hence with --relation one can for example extract only the FPIs issued on the relation defined. By default, the records are printed or their stats computed (--stats), using --quiet would only save the images without any output generated. This is a tool aimed mostly for very experienced users, useful for fixing page-level corruption or just analyzing the past state of a page, and there were no easy way to do that with the in-core tools up to now when looking at WAL. Each block is saved in a separate file, to ease their manipulation, with the file respecting <lsn>.<ts>.<db>.<rel>.<blk>_<fork> with as format. For instance, 00000000-010000C0.1663.1.6117.123_main refers to: - WAL record LSN in hexa format (00000000-010000C0). - Tablespace OID (1663). - Database OID (1). - Relfilenode (6117). - Block number (123). - Fork name of the file this block came from (_main). Author: David Christensen Reviewed-by: Sho Kato, Justin Pryzby, Bharath Rupireddy, Matthias van de Meent Discussion: https://postgr.es/m/CAOxo6XKjQb2bMSBRpePf3ZpzfNTwjQUc4Tafh21=jzjX6bX8CA@mail.gmail.com
1 parent 5de94a0 commit d497093
Copy full SHA for d497093

File tree

4 files changed

+286
-0
lines changed
Filter options

4 files changed

+286
-0
lines changed

‎doc/src/sgml/ref/pg_waldump.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/ref/pg_waldump.sgml
+66Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -240,6 +240,72 @@ PostgreSQL documentation
240240
</listitem>
241241
</varlistentry>
242242

243+
<varlistentry>
244+
<term><option>--save-fullpage=<replaceable>save_path</replaceable></option></term>
245+
<listitem>
246+
<para>
247+
Save full page images found in the WAL records to the
248+
<replaceable>save_path</replaceable> directory. The images saved
249+
are subject to the same filtering and limiting criteria as the
250+
records displayed.
251+
</para>
252+
<para>
253+
The full page images are saved with the following file name format:
254+
<literal><replaceable>LSN</replaceable>.<replaceable>RELTABLESPACE</replaceable>.<replaceable>DATOID</replaceable>.<replaceable>RELNODE</replaceable>.<replaceable>BLKNO</replaceable><replaceable>FORK</replaceable></literal>
255+
256+
The file names are composed of the following parts:
257+
<informaltable>
258+
<tgroup cols="2">
259+
<thead>
260+
<row>
261+
<entry>Component</entry>
262+
<entry>Description</entry>
263+
</row>
264+
</thead>
265+
266+
<tbody>
267+
<row>
268+
<entry>LSN</entry>
269+
<entry>The <acronym>LSN</acronym> of the record with this image,
270+
formatted as two 8-character hexadecimal numbers
271+
<literal>%08X-%08X</literal></entry>
272+
</row>
273+
274+
<row>
275+
<entry>RELTABLESPACE</entry>
276+
<entry>tablespace OID of the block</entry>
277+
</row>
278+
279+
<row>
280+
<entry>DATOID</entry>
281+
<entry>database OID of the block</entry>
282+
</row>
283+
284+
<row>
285+
<entry>RELNODE</entry>
286+
<entry>filenode of the block</entry>
287+
</row>
288+
289+
<row>
290+
<entry>BLKNO</entry>
291+
<entry>block number of the block</entry>
292+
</row>
293+
294+
<row>
295+
<entry>FORK</entry>
296+
<entry>
297+
The name of the fork the full page image came from, as of
298+
<literal>_main</literal>, <literal>_fsm</literal>,
299+
<literal>_vm</literal>, or <literal>_init</literal>.
300+
</entry>
301+
</row>
302+
</tbody>
303+
</tgroup>
304+
</informaltable>
305+
</para>
306+
</listitem>
307+
</varlistentry>
308+
243309
<varlistentry>
244310
<term><option>-x <replaceable>xid</replaceable></option></term>
245311
<term><option>--xid=<replaceable>xid</replaceable></option></term>

‎src/bin/pg_waldump/meson.build

Copy file name to clipboardExpand all lines: src/bin/pg_waldump/meson.build
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ tests += {
3131
'tap': {
3232
'tests': [
3333
't/001_basic.pl',
34+
't/002_save_fullpage.pl',
3435
],
3536
},
3637
}

‎src/bin/pg_waldump/pg_waldump.c

Copy file name to clipboardExpand all lines: src/bin/pg_waldump/pg_waldump.c
+108Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,13 @@
2323
#include "access/xlogrecord.h"
2424
#include "access/xlogstats.h"
2525
#include "common/fe_memutils.h"
26+
#include "common/file_perm.h"
27+
#include "common/file_utils.h"
2628
#include "common/logging.h"
29+
#include "common/relpath.h"
2730
#include "getopt_long.h"
2831
#include "rmgrdesc.h"
32+
#include "storage/bufpage.h"
2933

3034
/*
3135
* NOTE: For any code change or issue fix here, it is highly recommended to
@@ -70,6 +74,9 @@ typedef struct XLogDumpConfig
7074
bool filter_by_relation_block_enabled;
7175
ForkNumber filter_by_relation_forknum;
7276
bool filter_by_fpw;
77+
78+
/* save options */
79+
char *save_fullpage_path;
7380
} XLogDumpConfig;
7481

7582

@@ -112,6 +119,37 @@ verify_directory(const char *directory)
112119
return true;
113120
}
114121

122+
/*
123+
* Create if necessary the directory storing the full-page images extracted
124+
* from the WAL records read.
125+
*/
126+
static void
127+
create_fullpage_directory(char *path)
128+
{
129+
int ret;
130+
131+
switch ((ret = pg_check_dir(path)))
132+
{
133+
case 0:
134+
/* Does not exist, so create it */
135+
if (pg_mkdir_p(path, pg_dir_create_mode) < 0)
136+
pg_fatal("could not create directory \"%s\": %m", path);
137+
break;
138+
case 1:
139+
/* Present and empty, so do nothing */
140+
break;
141+
case 2:
142+
case 3:
143+
case 4:
144+
/* Exists and not empty */
145+
pg_fatal("directory \"%s\" exists but is not empty", path);
146+
break;
147+
default:
148+
/* Trouble accessing directory */
149+
pg_fatal("could not access directory \"%s\": %m", path);
150+
}
151+
}
152+
115153
/*
116154
* Split a pathname as dirname(1) and basename(1) would.
117155
*
@@ -439,6 +477,62 @@ XLogRecordHasFPW(XLogReaderState *record)
439477
return false;
440478
}
441479

480+
/*
481+
* Function to externally save all FPWs stored in the given WAL record.
482+
* Decompression is applied to all the blocks saved, if necessary.
483+
*/
484+
static void
485+
XLogRecordSaveFPWs(XLogReaderState *record, const char *savepath)
486+
{
487+
int block_id;
488+
489+
for (block_id = 0; block_id <= XLogRecMaxBlockId(record); block_id++)
490+
{
491+
PGAlignedBlock buf;
492+
Page page;
493+
char filename[MAXPGPATH];
494+
char forkname[FORKNAMECHARS + 2]; /* _ + terminating zero */
495+
FILE *file;
496+
BlockNumber blk;
497+
RelFileLocator rnode;
498+
ForkNumber fork;
499+
500+
if (!XLogRecHasBlockRef(record, block_id))
501+
continue;
502+
503+
if (!XLogRecHasBlockImage(record, block_id))
504+
continue;
505+
506+
page = (Page) buf.data;
507+
508+
/* Full page exists, so let's save it */
509+
if (!RestoreBlockImage(record, block_id, page))
510+
pg_fatal("%s", record->errormsg_buf);
511+
512+
(void) XLogRecGetBlockTagExtended(record, block_id,
513+
&rnode, &fork, &blk, NULL);
514+
515+
if (fork >= 0 && fork <= MAX_FORKNUM)
516+
sprintf(forkname, "_%s", forkNames[fork]);
517+
else
518+
pg_fatal("invalid fork number: %u", fork);
519+
520+
snprintf(filename, MAXPGPATH, "%s/%08X-%08X.%u.%u.%u.%u%s", savepath,
521+
LSN_FORMAT_ARGS(record->ReadRecPtr),
522+
rnode.spcOid, rnode.dbOid, rnode.relNumber, blk, forkname);
523+
524+
file = fopen(filename, PG_BINARY_W);
525+
if (!file)
526+
pg_fatal("could not open file \"%s\": %m", filename);
527+
528+
if (fwrite(page, BLCKSZ, 1, file) != 1)
529+
pg_fatal("could not write file \"%s\": %m", filename);
530+
531+
if (fclose(file) != 0)
532+
pg_fatal("could not write file \"%s\": %m", filename);
533+
}
534+
}
535+
442536
/*
443537
* Print a record to stdout
444538
*/
@@ -679,6 +773,8 @@ usage(void)
679773
" (default: 1 or the value used in STARTSEG)\n"));
680774
printf(_(" -V, --version output version information, then exit\n"));
681775
printf(_(" -w, --fullpage only show records with a full page write\n"));
776+
printf(_(" --save-fullpage=PATH\n"
777+
" save full page images\n"));
682778
printf(_(" -x, --xid=XID only show records with transaction ID XID\n"));
683779
printf(_(" -z, --stats[=record] show statistics instead of records\n"
684780
" (optionally, show per-record statistics)\n"));
@@ -719,6 +815,7 @@ main(int argc, char **argv)
719815
{"xid", required_argument, NULL, 'x'},
720816
{"version", no_argument, NULL, 'V'},
721817
{"stats", optional_argument, NULL, 'z'},
818+
{"save-fullpage", required_argument, NULL, 1},
722819
{NULL, 0, NULL, 0}
723820
};
724821

@@ -770,6 +867,7 @@ main(int argc, char **argv)
770867
config.filter_by_relation_block_enabled = false;
771868
config.filter_by_relation_forknum = InvalidForkNumber;
772869
config.filter_by_fpw = false;
870+
config.save_fullpage_path = NULL;
773871
config.stats = false;
774872
config.stats_per_record = false;
775873

@@ -942,6 +1040,9 @@ main(int argc, char **argv)
9421040
}
9431041
}
9441042
break;
1043+
case 1:
1044+
config.save_fullpage_path = pg_strdup(optarg);
1045+
break;
9451046
default:
9461047
goto bad_argument;
9471048
}
@@ -972,6 +1073,9 @@ main(int argc, char **argv)
9721073
}
9731074
}
9741075

1076+
if (config.save_fullpage_path != NULL)
1077+
create_fullpage_directory(config.save_fullpage_path);
1078+
9751079
/* parse files as start/end boundaries, extract path if not specified */
9761080
if (optind < argc)
9771081
{
@@ -1154,6 +1258,10 @@ main(int argc, char **argv)
11541258
XLogDumpDisplayRecord(&config, xlogreader_state);
11551259
}
11561260

1261+
/* save full pages if requested */
1262+
if (config.save_fullpage_path != NULL)
1263+
XLogRecordSaveFPWs(xlogreader_state, config.save_fullpage_path);
1264+
11571265
/* check whether we printed enough */
11581266
config.already_displayed_records++;
11591267
if (config.stop_after_records > 0 &&
+111Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
2+
# Copyright (c) 2022, PostgreSQL Global Development Group
3+
4+
use strict;
5+
use warnings;
6+
use File::Basename;
7+
use PostgreSQL::Test::Cluster;
8+
use PostgreSQL::Test::RecursiveCopy;
9+
use PostgreSQL::Test::Utils;
10+
use Test::More;
11+
12+
my ($blocksize, $walfile_name);
13+
14+
# Function to extract the LSN from the given block structure
15+
sub get_block_lsn
16+
{
17+
my $path = shift;
18+
my $blocksize = shift;
19+
my $block;
20+
21+
open my $fh, '<', $path or die "couldn't open file: $path\n";
22+
die "could not read block\n"
23+
if $blocksize != read($fh, $block, $blocksize);
24+
my ($lsn_hi, $lsn_lo) = unpack('LL', $block);
25+
26+
$lsn_hi = sprintf('%08X', $lsn_hi);
27+
$lsn_lo = sprintf('%08X', $lsn_lo);
28+
29+
return ($lsn_hi, $lsn_lo);
30+
}
31+
32+
my $node = PostgreSQL::Test::Cluster->new('main');
33+
$node->init;
34+
$node->append_conf(
35+
'postgresql.conf', q{
36+
wal_level = 'replica'
37+
max_wal_senders = 4
38+
});
39+
$node->start;
40+
41+
# Generate data/WAL to examine that will have full pages in them.
42+
$node->safe_psql(
43+
'postgres',
44+
"SELECT 'init' FROM pg_create_physical_replication_slot('regress_pg_waldump_slot', true, false);
45+
CREATE TABLE test_table AS SELECT generate_series(1,100) a;
46+
-- Force FPWs on the next writes.
47+
CHECKPOINT;
48+
UPDATE test_table SET a = a + 1;
49+
");
50+
51+
($walfile_name, $blocksize) = split '\|' => $node->safe_psql('postgres',
52+
"SELECT pg_walfile_name(pg_switch_wal()), current_setting('block_size')");
53+
54+
# Get the relation node, etc for the new table
55+
my $relation = $node->safe_psql(
56+
'postgres',
57+
q{SELECT format(
58+
'%s/%s/%s',
59+
CASE WHEN reltablespace = 0 THEN dattablespace ELSE reltablespace END,
60+
pg_database.oid,
61+
pg_relation_filenode(pg_class.oid))
62+
FROM pg_class, pg_database
63+
WHERE relname = 'test_table' AND
64+
datname = current_database()}
65+
);
66+
67+
my $walfile = $node->data_dir . '/pg_wal/' . $walfile_name;
68+
my $tmp_folder = PostgreSQL::Test::Utils::tempdir;
69+
70+
ok(-f $walfile, "Got a WAL file");
71+
72+
$node->command_ok(
73+
[
74+
'pg_waldump', '--quiet',
75+
'--save-fullpage', "$tmp_folder/raw",
76+
'--relation', $relation,
77+
$walfile
78+
]);
79+
80+
# This regexp will match filenames formatted as:
81+
# XXXXXXXX-XXXXXXXX.DBOID.TLOID.NODEOID.dd_fork with the components being:
82+
# - WAL LSN in hex format,
83+
# - Tablespace OID (0 for global)
84+
# - Database OID.
85+
# - Relfilenode.
86+
# - Block number.
87+
# - Fork this block came from (vm, init, fsm, or main).
88+
my $file_re =
89+
qr/^([0-9A-F]{8})-([0-9A-F]{8})[.][0-9]+[.][0-9]+[.][0-9]+[.][0-9]+(?:_vm|_init|_fsm|_main)?$/;
90+
91+
my $file_count = 0;
92+
93+
# Verify filename format matches --save-fullpage.
94+
for my $fullpath (glob "$tmp_folder/raw/*")
95+
{
96+
my $file = File::Basename::basename($fullpath);
97+
98+
like($file, $file_re, "verify filename format for file $file");
99+
$file_count++;
100+
101+
my ($hi_lsn_fn, $lo_lsn_fn) = ($file =~ $file_re);
102+
my ($hi_lsn_bk, $lo_lsn_bk) = get_block_lsn($fullpath, $blocksize);
103+
104+
# The LSN on the block comes before the file's LSN.
105+
ok( $hi_lsn_fn . $lo_lsn_fn gt $hi_lsn_bk . $lo_lsn_bk,
106+
'LSN stored in the file precedes the one stored in the block');
107+
}
108+
109+
ok($file_count > 0, 'verify that at least one block has been saved');
110+
111+
done_testing();

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.