Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 5c279a6

Browse filesBrowse files
jeff-davisjeff-davis
committed
Custom WAL Resource Managers.
Allow extensions to specify a new custom resource manager (rmgr), which allows specialized WAL. This is meant to be used by a Table Access Method or Index Access Method. Prior to this commit, only Generic WAL was available, which offers support for recovery and physical replication but not logical replication. Reviewed-by: Julien Rouhaud, Bharath Rupireddy, Andres Freund Discussion: https://postgr.es/m/ed1fb2e22d15d3563ae0eb610f7b61bb15999c0a.camel%40j-davis.com
1 parent a8cfb0c commit 5c279a6
Copy full SHA for 5c279a6

File tree

24 files changed

+526
-70
lines changed
Filter options

24 files changed

+526
-70
lines changed

‎doc/src/sgml/config.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/config.sgml
+2-2Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11189,8 +11189,8 @@ LOG: CleanUpLock: deleting: lock(0xb7acd844) id(24688,24696,0,0,0,1)
1118911189
<literal>heap2</literal>, <literal>btree</literal>, <literal>hash</literal>,
1119011190
<literal>gin</literal>, <literal>gist</literal>, <literal>sequence</literal>,
1119111191
<literal>spgist</literal>, <literal>brin</literal>, and <literal>generic</literal>.
11192-
Only superusers and users with the appropriate <literal>SET</literal>
11193-
privilege can change this setting.
11192+
Extensions may define additional resource managers. Only superusers and users with
11193+
the appropriate <literal>SET</literal> privilege can change this setting.
1119411194
</para>
1119511195
</listitem>
1119611196
</varlistentry>

‎doc/src/sgml/custom-rmgr.sgml

Copy file name to clipboard
+98Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
<!-- doc/src/sgml/custom-rmgr.sgml -->
2+
3+
<chapter id="custom-rmgr">
4+
<title>Custom WAL Resource Managers</title>
5+
6+
<para>
7+
This chapter explains the interface between the core
8+
<productname>PostgreSQL</productname> system and custom WAL resource
9+
managers, which enable extensions to integrate directly with the <link
10+
linkend="wal"><acronym>WAL</acronym></link>.
11+
</para>
12+
<para>
13+
An extension, especially a <link linkend="tableam">Table Access
14+
Method</link> or <link linkend="indexam">Index Access Method</link>, may
15+
need to use WAL for recovery, replication, and/or <link
16+
linkend="logicaldecoding">Logical Decoding</link>. Custom resource managers
17+
are a more flexible alternative to <link linkend="generic-wal">Generic
18+
WAL</link> (which does not support logical decoding), but more complex for
19+
an extension to implement.
20+
</para>
21+
<para>
22+
To create a new custom WAL resouce manager, first define an
23+
<structname>RmgrData</structname> structure with implementations for the
24+
resource manager methods. Refer to
25+
<filename>src/backend/access/transam/README</filename> and
26+
<filename>src/include/access/xlog_internal.h</filename> in the
27+
<productname>PostgreSQL</productname> source.
28+
<programlisting>
29+
/*
30+
* Method table for resource managers.
31+
*
32+
* This struct must be kept in sync with the PG_RMGR definition in
33+
* rmgr.c.
34+
*
35+
* rm_identify must return a name for the record based on xl_info (without
36+
* reference to the rmid). For example, XLOG_BTREE_VACUUM would be named
37+
* "VACUUM". rm_desc can then be called to obtain additional detail for the
38+
* record, if available (e.g. the last block).
39+
*
40+
* rm_mask takes as input a page modified by the resource manager and masks
41+
* out bits that shouldn't be flagged by wal_consistency_checking.
42+
*
43+
* RmgrTable[] is indexed by RmgrId values (see rmgrlist.h). If rm_name is
44+
* NULL, the corresponding RmgrTable entry is considered invalid.
45+
*/
46+
typedef struct RmgrData
47+
{
48+
const char *rm_name;
49+
void (*rm_redo) (XLogReaderState *record);
50+
void (*rm_desc) (StringInfo buf, XLogReaderState *record);
51+
const char *(*rm_identify) (uint8 info);
52+
void (*rm_startup) (void);
53+
void (*rm_cleanup) (void);
54+
void (*rm_mask) (char *pagedata, BlockNumber blkno);
55+
void (*rm_decode) (struct LogicalDecodingContext *ctx,
56+
struct XLogRecordBuffer *buf);
57+
} RmgrData;
58+
</programlisting>
59+
</para>
60+
<para>
61+
Then, register your new resource
62+
manager.
63+
64+
<programlisting>
65+
/*
66+
* Register a new custom WAL resource manager.
67+
*
68+
* Resource manager IDs must be globally unique across all extensions. Refer
69+
* to https://wiki.postgresql.org/wiki/CustomWALResourceManager to reserve a
70+
* unique RmgrId for your extension, to avoid conflicts with other extension
71+
* developers. During development, use RM_EXPERIMENTAL_ID to avoid needlessly
72+
* reserving a new ID.
73+
*/
74+
extern void RegisterCustomRmgr(RmgrId rmid, RmgrData *rmgr);
75+
</programlisting>
76+
<function>RegisterCustomRmgr</function> must be called from the
77+
extension module's <link linkend="xfunc-c-dynload">_PG_init</link> function.
78+
While developing a new extension, use <literal>RM_EXPERIMENTAL_ID</literal>
79+
for <parameter>rmid</parameter>. When you ready to release the extension to
80+
users, reserve a new resource manager ID at the <ulink
81+
url="https://wiki.postgresql.org/wiki/CustomWALResourceManagers">Custom WAL
82+
Resource Manager</ulink> page.
83+
</para>
84+
85+
<para>
86+
Place the extension module implementing the custom resource manager in <xref
87+
linkend="guc-shared-preload-libraries"/> so that it will be loaded early
88+
during <productname>PostgreSQL</productname> startup.
89+
</para>
90+
<note>
91+
<para>
92+
The extension must remain in shared_preload_libraries as long as any
93+
custom WAL records may exist in the system. Otherwise
94+
<productname>PostgreSQL</productname> will not be able to apply or decode
95+
the custom WAL records, which may prevent the server from starting.
96+
</para>
97+
</note>
98+
</chapter>

‎doc/src/sgml/filelist.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/filelist.sgml
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@
105105
<!ENTITY storage SYSTEM "storage.sgml">
106106
<!ENTITY tablesample-method SYSTEM "tablesample-method.sgml">
107107
<!ENTITY generic-wal SYSTEM "generic-wal.sgml">
108+
<!ENTITY custom-rmgr SYSTEM "custom-rmgr.sgml">
108109
<!ENTITY backup-manifest SYSTEM "backup-manifest.sgml">
109110

110111
<!-- contrib information -->

‎doc/src/sgml/func.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/func.sgml
+19Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26001,6 +26001,25 @@ postgres=# SELECT * FROM pg_walfile_name_offset((pg_backup_stop()).lsn);
2600126001
without recovery, the function returns <literal>NULL</literal>.
2600226002
</para></entry>
2600326003
</row>
26004+
26005+
<row>
26006+
<entry role="func_table_entry"><para role="func_signature">
26007+
<indexterm>
26008+
<primary>pg_get_wal_resource_managers</primary>
26009+
</indexterm>
26010+
<function>pg_get_wal_resource_managers</function> ()
26011+
<returnvalue>setof record</returnvalue>
26012+
( <parameter>rm_id</parameter> <type>integer</type>,
26013+
<parameter>rm_name</parameter> <type>text</type>,
26014+
<parameter>rm_builtin</parameter> <type>boolean</type> )
26015+
</para>
26016+
<para>
26017+
Returns the currently-loaded WAL resource managers in the system. The
26018+
column <parameter>rm_builtin</parameter> indicates whether it's a
26019+
built-in resource manager, or a custom resource manager loaded by an
26020+
extension.
26021+
</para></entry>
26022+
</row>
2600426023
</tbody>
2600526024
</tgroup>
2600626025
</table>

‎doc/src/sgml/generic-wal.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/generic-wal.sgml
+17-3Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,25 @@
66
<para>
77
Although all built-in WAL-logged modules have their own types of WAL
88
records, there is also a generic WAL record type, which describes changes
9-
to pages in a generic way. This is useful for extensions that provide
10-
custom access methods, because they cannot register their own WAL redo
11-
routines.
9+
to pages in a generic way. This is useful for extensions that provide
10+
custom access methods.
1211
</para>
1312

13+
<para>
14+
In comparison with <link linkend="custom-rmgr">Custom WAL Resource
15+
Managers</link>, Generic WAL is simpler for an extension to implement and
16+
does not require the extension library to be loaded in order to apply the
17+
records.
18+
</para>
19+
20+
<note>
21+
<para>
22+
Generic WAL records are ignored during <link
23+
linkend="logicaldecoding">Logical Decoding</link>. If logical decoding is
24+
required for your extension, consider a Custom WAL Resource Manager.
25+
</para>
26+
</note>
27+
1428
<para>
1529
The API for constructing generic WAL records is defined in
1630
<filename>access/generic_xlog.h</filename> and implemented

‎doc/src/sgml/postgres.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/postgres.sgml
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -262,6 +262,7 @@ break is not needed in a wider output rendering.
262262
&tableam;
263263
&indexam;
264264
&generic-wal;
265+
&custom-rmgr;
265266
&btree;
266267
&gist;
267268
&spgist;

‎doc/src/sgml/ref/pg_waldump.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/ref/pg_waldump.sgml
+8Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,14 @@ PostgreSQL documentation
173173
If <literal>list</literal> is passed as name, print a list of valid resource manager
174174
names, and exit.
175175
</para>
176+
<para>
177+
Extensions may define custom resource managers, but pg_waldump does
178+
not load the extension module and therefore does not recognize custom
179+
resource managers by name. Instead, you can specify the custom
180+
resource managers as <literal>custom###</literal> where
181+
"<literal>###</literal>" is the three-digit resource manager ID. Names
182+
of this form will always be considered valid.
183+
</para>
176184
</listitem>
177185
</varlistentry>
178186

‎src/backend/access/transam/rmgr.c

Copy file name to clipboardExpand all lines: src/backend/access/transam/rmgr.c
+123-1Lines changed: 123 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,16 +24,138 @@
2424
#include "commands/dbcommands_xlog.h"
2525
#include "commands/sequence.h"
2626
#include "commands/tablespace.h"
27+
#include "fmgr.h"
28+
#include "funcapi.h"
29+
#include "miscadmin.h"
2730
#include "replication/decode.h"
2831
#include "replication/message.h"
2932
#include "replication/origin.h"
3033
#include "storage/standby.h"
34+
#include "utils/builtins.h"
3135
#include "utils/relmapper.h"
3236

3337
/* must be kept in sync with RmgrData definition in xlog_internal.h */
3438
#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,decode) \
3539
{ name, redo, desc, identify, startup, cleanup, mask, decode },
3640

37-
const RmgrData RmgrTable[RM_MAX_ID + 1] = {
41+
RmgrData RmgrTable[RM_MAX_ID + 1] = {
3842
#include "access/rmgrlist.h"
3943
};
44+
45+
/*
46+
* Start up all resource managers.
47+
*/
48+
void
49+
RmgrStartup(void)
50+
{
51+
for (int rmid = 0; rmid <= RM_MAX_ID; rmid++)
52+
{
53+
if (!RmgrIdExists(rmid))
54+
continue;
55+
56+
if (RmgrTable[rmid].rm_startup != NULL)
57+
RmgrTable[rmid].rm_startup();
58+
}
59+
}
60+
61+
/*
62+
* Clean up all resource managers.
63+
*/
64+
void
65+
RmgrCleanup(void)
66+
{
67+
for (int rmid = 0; rmid <= RM_MAX_ID; rmid++)
68+
{
69+
if (!RmgrIdExists(rmid))
70+
continue;
71+
72+
if (RmgrTable[rmid].rm_cleanup != NULL)
73+
RmgrTable[rmid].rm_cleanup();
74+
}
75+
}
76+
77+
/*
78+
* Emit ERROR when we encounter a record with an RmgrId we don't
79+
* recognize.
80+
*/
81+
void
82+
RmgrNotFound(RmgrId rmid)
83+
{
84+
ereport(ERROR, (errmsg("resource manager with ID %d not registered", rmid),
85+
errhint("Include the extension module that implements this resource manager in shared_preload_libraries.")));
86+
}
87+
88+
/*
89+
* Register a new custom WAL resource manager.
90+
*
91+
* Resource manager IDs must be globally unique across all extensions. Refer
92+
* to https://wiki.postgresql.org/wiki/CustomWALResourceManager to reserve a
93+
* unique RmgrId for your extension, to avoid conflicts with other extension
94+
* developers. During development, use RM_EXPERIMENTAL_ID to avoid needlessly
95+
* reserving a new ID.
96+
*/
97+
void
98+
RegisterCustomRmgr(RmgrId rmid, RmgrData *rmgr)
99+
{
100+
if (rmgr->rm_name == NULL || strlen(rmgr->rm_name) == 0)
101+
ereport(ERROR, (errmsg("custom resource manager name is invalid"),
102+
errhint("Provide a non-empty name for the custom resource manager.")));
103+
104+
if (!RMID_IS_CUSTOM(rmid))
105+
ereport(ERROR, (errmsg("custom resource manager ID %d is out of range", rmid),
106+
errhint("Provide a custom resource manager ID between %d and %d.",
107+
RM_MIN_CUSTOM_ID, RM_MAX_CUSTOM_ID)));
108+
109+
if (!process_shared_preload_libraries_in_progress)
110+
ereport(ERROR,
111+
(errmsg("failed to register custom resource manager \"%s\" with ID %d", rmgr->rm_name, rmid),
112+
errdetail("Custom resource manager must be registered while initializing modules in shared_preload_libraries.")));
113+
114+
if (RmgrTable[rmid].rm_name != NULL)
115+
ereport(ERROR,
116+
(errmsg("failed to register custom resource manager \"%s\" with ID %d", rmgr->rm_name, rmid),
117+
errdetail("Custom resource manager \"%s\" already registered with the same ID.",
118+
RmgrTable[rmid].rm_name)));
119+
120+
/* check for existing rmgr with the same name */
121+
for (int existing_rmid = 0; existing_rmid <= RM_MAX_ID; existing_rmid++)
122+
{
123+
if (!RmgrIdExists(existing_rmid))
124+
continue;
125+
126+
if (!pg_strcasecmp(RmgrTable[existing_rmid].rm_name, rmgr->rm_name))
127+
ereport(ERROR,
128+
(errmsg("failed to register custom resource manager \"%s\" with ID %d", rmgr->rm_name, rmid),
129+
errdetail("Existing resource manager with ID %d has the same name.", existing_rmid)));
130+
}
131+
132+
/* register it */
133+
RmgrTable[rmid] = *rmgr;
134+
ereport(LOG,
135+
(errmsg("registered custom resource manager \"%s\" with ID %d",
136+
rmgr->rm_name, rmid)));
137+
}
138+
139+
/* SQL SRF showing loaded resource managers */
140+
Datum
141+
pg_get_wal_resource_managers(PG_FUNCTION_ARGS)
142+
{
143+
#define PG_GET_RESOURCE_MANAGERS_COLS 3
144+
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
145+
Datum values[PG_GET_RESOURCE_MANAGERS_COLS];
146+
bool nulls[PG_GET_RESOURCE_MANAGERS_COLS] = {0};
147+
148+
SetSingleFuncCall(fcinfo, 0);
149+
150+
for (int rmid = 0; rmid <= RM_MAX_ID; rmid++)
151+
{
152+
if (!RmgrIdExists(rmid))
153+
continue;
154+
values[0] = Int32GetDatum(rmid);
155+
values[1] = CStringGetTextDatum(GetRmgr(rmid).rm_name);
156+
values[2] = BoolGetDatum(RMID_IS_BUILTIN(rmid));
157+
tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
158+
}
159+
160+
return (Datum) 0;
161+
}

‎src/backend/access/transam/xlogreader.c

Copy file name to clipboardExpand all lines: src/backend/access/transam/xlogreader.c
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1102,7 +1102,7 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr,
11021102
(uint32) SizeOfXLogRecord, record->xl_tot_len);
11031103
return false;
11041104
}
1105-
if (record->xl_rmid > RM_MAX_ID)
1105+
if (!RMID_IS_VALID(record->xl_rmid))
11061106
{
11071107
report_invalid_record(state,
11081108
"invalid resource manager ID %u at %X/%X",

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.