Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit c655077

Browse filesBrowse files
alvherrealvherre
committed
Allow users to limit storage reserved by replication slots
Replication slots are useful to retain data that may be needed by a replication system. But experience has shown that allowing them to retain excessive data can lead to the primary failing because of running out of space. This new feature allows the user to configure a maximum amount of space to be reserved using the new option max_slot_wal_keep_size. Slots that overrun that space are invalidated at checkpoint time, enabling the storage to be released. Author: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com> Reviewed-by: Jehan-Guillaume de Rorthais <jgdr@dalibo.com> Reviewed-by: Álvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://postgr.es/m/20170228.122736.123383594.horiguchi.kyotaro@lab.ntt.co.jp
1 parent b63c293 commit c655077
Copy full SHA for c655077

File tree

Expand file treeCollapse file tree

17 files changed

+595
-43
lines changed
Filter options
Expand file treeCollapse file tree

17 files changed

+595
-43
lines changed

‎doc/src/sgml/catalogs.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/catalogs.sgml
+38Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9907,6 +9907,44 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
99079907
</entry>
99089908
</row>
99099909

9910+
<row>
9911+
<entry><structfield>wal_status</structfield></entry>
9912+
<entry><type>text</type></entry>
9913+
<entry></entry>
9914+
9915+
<entry>Availability of WAL files claimed by this slot.
9916+
Possible values are:
9917+
<simplelist>
9918+
<member>
9919+
<literal>normal</literal> means that the claimed files
9920+
are within <varname>max_wal_size</varname>
9921+
</member>
9922+
<member>
9923+
<literal>reserved</literal> means that <varname>max_wal_size</varname>
9924+
is exceeded but the files are still held, either by some replication
9925+
slot or by <varname>wal_keep_segments</varname>
9926+
</member>
9927+
<member>
9928+
<literal>lost</literal> means that some WAL files are definitely lost
9929+
and this slot cannot be used to resume replication anymore.
9930+
</member>
9931+
</simplelist>
9932+
The last two states are seen only when
9933+
<xref linkend="guc-max-slot-wal-keep-size"/> is
9934+
non-negative. If <structfield>restart_lsn</structfield> is NULL, this
9935+
field is null.
9936+
</entry>
9937+
</row>
9938+
9939+
<row>
9940+
<entry><structfield>min_safe_lsn</structfield></entry>
9941+
<entry><type>pg_lsn</type></entry>
9942+
<entry></entry>
9943+
<entry>
9944+
The minimum LSN currently available for walsenders.
9945+
</entry>
9946+
</row>
9947+
99109948
</tbody>
99119949
</tgroup>
99129950
</table>

‎doc/src/sgml/config.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/config.sgml
+23Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3777,6 +3777,29 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
37773777
</listitem>
37783778
</varlistentry>
37793779

3780+
<varlistentry id="guc-max-slot-wal-keep-size" xreflabel="max_slot_wal_keep_size">
3781+
<term><varname>max_slot_wal_keep_size</varname> (<type>integer</type>)
3782+
<indexterm>
3783+
<primary><varname>max_slot_wal_keep_size</varname> configuration parameter</primary>
3784+
</indexterm>
3785+
</term>
3786+
<listitem>
3787+
<para>
3788+
Specify the maximum size of WAL files
3789+
that <link linkend="streaming-replication-slots">replication
3790+
slots</link> are allowed to retain in the <filename>pg_wal</filename>
3791+
directory at checkpoint time.
3792+
If <varname>max_slot_wal_keep_size</varname> is -1 (the default),
3793+
replication slots retain unlimited amount of WAL files. If
3794+
restart_lsn of a replication slot gets behind more than that megabytes
3795+
from the current LSN, the standby using the slot may no longer be able
3796+
to continue replication due to removal of required WAL files. You
3797+
can see the WAL availability of replication slots
3798+
in <link linkend="view-pg-replication-slots">pg_replication_slots</link>.
3799+
</para>
3800+
</listitem>
3801+
</varlistentry>
3802+
37803803
<varlistentry id="guc-wal-sender-timeout" xreflabel="wal_sender_timeout">
37813804
<term><varname>wal_sender_timeout</varname> (<type>integer</type>)
37823805
<indexterm>

‎doc/src/sgml/high-availability.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/high-availability.sgml
+5-3Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -925,9 +925,11 @@ primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass'
925925
<xref linkend="guc-archive-command"/>.
926926
However, these methods often result in retaining more WAL segments than
927927
required, whereas replication slots retain only the number of segments
928-
known to be needed. An advantage of these methods is that they bound
929-
the space requirement for <literal>pg_wal</literal>; there is currently no way
930-
to do this using replication slots.
928+
known to be needed. On the other hand, replication slots can retain so
929+
many WAL segments that they fill up the space allocated
930+
for <literal>pg_wal</literal>;
931+
<xref linkend="guc-max-slot-wal-keep-size"/> limits the size of WAL files
932+
retained by replication slots.
931933
</para>
932934
<para>
933935
Similarly, <xref linkend="guc-hot-standby-feedback"/>

‎src/backend/access/transam/xlog.c

Copy file name to clipboardExpand all lines: src/backend/access/transam/xlog.c
+122-23Lines changed: 122 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,7 @@ int wal_level = WAL_LEVEL_MINIMAL;
108108
int CommitDelay = 0; /* precommit delay in microseconds */
109109
int CommitSiblings = 5; /* # concurrent xacts needed to sleep */
110110
int wal_retrieve_retry_interval = 5000;
111+
int max_slot_wal_keep_size_mb = -1;
111112

112113
#ifdef WAL_DEBUG
113114
bool XLOG_DEBUG = false;
@@ -759,7 +760,7 @@ static ControlFileData *ControlFile = NULL;
759760
*/
760761
#define UsableBytesInPage (XLOG_BLCKSZ - SizeOfXLogShortPHD)
761762

762-
/* Convert min_wal_size_mb and max_wal_size_mb to equivalent segment count */
763+
/* Convert values of GUCs measured in megabytes to equiv. segment count */
763764
#define ConvertToXSegs(x, segsize) \
764765
(x / ((segsize) / (1024 * 1024)))
765766

@@ -3963,9 +3964,10 @@ XLogGetLastRemovedSegno(void)
39633964
return lastRemovedSegNo;
39643965
}
39653966

3967+
39663968
/*
3967-
* Update the last removed segno pointer in shared memory, to reflect
3968-
* that the given XLOG file has been removed.
3969+
* Update the last removed segno pointer in shared memory, to reflect that the
3970+
* given XLOG file has been removed.
39693971
*/
39703972
static void
39713973
UpdateLastRemovedPtr(char *filename)
@@ -9043,6 +9045,7 @@ CreateCheckPoint(int flags)
90439045
*/
90449046
XLByteToSeg(RedoRecPtr, _logSegNo, wal_segment_size);
90459047
KeepLogSeg(recptr, &_logSegNo);
9048+
InvalidateObsoleteReplicationSlots(_logSegNo);
90469049
_logSegNo--;
90479050
RemoveOldXlogFiles(_logSegNo, RedoRecPtr, recptr);
90489051

@@ -9377,6 +9380,7 @@ CreateRestartPoint(int flags)
93779380
replayPtr = GetXLogReplayRecPtr(&replayTLI);
93789381
endptr = (receivePtr < replayPtr) ? replayPtr : receivePtr;
93799382
KeepLogSeg(endptr, &_logSegNo);
9383+
InvalidateObsoleteReplicationSlots(_logSegNo);
93809384
_logSegNo--;
93819385

93829386
/*
@@ -9445,48 +9449,143 @@ CreateRestartPoint(int flags)
94459449
return true;
94469450
}
94479451

9452+
/*
9453+
* Report availability of WAL for the given target LSN
9454+
* (typically a slot's restart_lsn)
9455+
*
9456+
* Returns one of the following enum values:
9457+
* * WALAVAIL_NORMAL means targetLSN is available because it is in the range
9458+
* of max_wal_size.
9459+
*
9460+
* * WALAVAIL_PRESERVED means it is still available by preserving extra
9461+
* segments beyond max_wal_size. If max_slot_wal_keep_size is smaller
9462+
* than max_wal_size, this state is not returned.
9463+
*
9464+
* * WALAVAIL_REMOVED means it is definitely lost. A replication stream on
9465+
* a slot with this LSN cannot continue.
9466+
*
9467+
* * WALAVAIL_INVALID_LSN means the slot hasn't been set to reserve WAL.
9468+
*/
9469+
WALAvailability
9470+
GetWALAvailability(XLogRecPtr targetLSN)
9471+
{
9472+
XLogRecPtr currpos; /* current write LSN */
9473+
XLogSegNo currSeg; /* segid of currpos */
9474+
XLogSegNo targetSeg; /* segid of targetLSN */
9475+
XLogSegNo oldestSeg; /* actual oldest segid */
9476+
XLogSegNo oldestSegMaxWalSize; /* oldest segid kept by max_wal_size */
9477+
XLogSegNo oldestSlotSeg = InvalidXLogRecPtr; /* oldest segid kept by
9478+
* slot */
9479+
uint64 keepSegs;
9480+
9481+
/* slot does not reserve WAL. Either deactivated, or has never been active */
9482+
if (XLogRecPtrIsInvalid(targetLSN))
9483+
return WALAVAIL_INVALID_LSN;
9484+
9485+
currpos = GetXLogWriteRecPtr();
9486+
9487+
/* calculate oldest segment currently needed by slots */
9488+
XLByteToSeg(targetLSN, targetSeg, wal_segment_size);
9489+
KeepLogSeg(currpos, &oldestSlotSeg);
9490+
9491+
/*
9492+
* Find the oldest extant segment file. We get 1 until checkpoint removes
9493+
* the first WAL segment file since startup, which causes the status being
9494+
* wrong under certain abnormal conditions but that doesn't actually harm.
9495+
*/
9496+
oldestSeg = XLogGetLastRemovedSegno() + 1;
9497+
9498+
/* calculate oldest segment by max_wal_size and wal_keep_segments */
9499+
XLByteToSeg(currpos, currSeg, wal_segment_size);
9500+
keepSegs = ConvertToXSegs(Max(max_wal_size_mb, wal_keep_segments),
9501+
wal_segment_size) + 1;
9502+
9503+
if (currSeg > keepSegs)
9504+
oldestSegMaxWalSize = currSeg - keepSegs;
9505+
else
9506+
oldestSegMaxWalSize = 1;
9507+
9508+
/*
9509+
* If max_slot_wal_keep_size has changed after the last call, the segment
9510+
* that would been kept by the current setting might have been lost by the
9511+
* previous setting. No point in showing normal or keeping status values
9512+
* if the targetSeg is known to be lost.
9513+
*/
9514+
if (targetSeg >= oldestSeg)
9515+
{
9516+
/*
9517+
* show "normal" when targetSeg is within max_wal_size, even if
9518+
* max_slot_wal_keep_size is smaller than max_wal_size.
9519+
*/
9520+
if ((max_slot_wal_keep_size_mb <= 0 ||
9521+
max_slot_wal_keep_size_mb >= max_wal_size_mb) &&
9522+
oldestSegMaxWalSize <= targetSeg)
9523+
return WALAVAIL_NORMAL;
9524+
9525+
/* being retained by slots */
9526+
if (oldestSlotSeg <= targetSeg)
9527+
return WALAVAIL_RESERVED;
9528+
}
9529+
9530+
/* Definitely lost */
9531+
return WALAVAIL_REMOVED;
9532+
}
9533+
9534+
94489535
/*
94499536
* Retreat *logSegNo to the last segment that we need to retain because of
94509537
* either wal_keep_segments or replication slots.
94519538
*
94529539
* This is calculated by subtracting wal_keep_segments from the given xlog
94539540
* location, recptr and by making sure that that result is below the
9454-
* requirement of replication slots.
9541+
* requirement of replication slots. For the latter criterion we do consider
9542+
* the effects of max_slot_wal_keep_size: reserve at most that much space back
9543+
* from recptr.
94559544
*/
94569545
static void
94579546
KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
94589547
{
9548+
XLogSegNo currSegNo;
94599549
XLogSegNo segno;
94609550
XLogRecPtr keep;
94619551

9462-
XLByteToSeg(recptr, segno, wal_segment_size);
9463-
keep = XLogGetReplicationSlotMinimumLSN();
9552+
XLByteToSeg(recptr, currSegNo, wal_segment_size);
9553+
segno = currSegNo;
94649554

9465-
/* compute limit for wal_keep_segments first */
9466-
if (wal_keep_segments > 0)
9555+
/*
9556+
* Calculate how many segments are kept by slots first, adjusting for
9557+
* max_slot_wal_keep_size.
9558+
*/
9559+
keep = XLogGetReplicationSlotMinimumLSN();
9560+
if (keep != InvalidXLogRecPtr)
94679561
{
9468-
/* avoid underflow, don't go below 1 */
9469-
if (segno <= wal_keep_segments)
9470-
segno = 1;
9471-
else
9472-
segno = segno - wal_keep_segments;
9473-
}
9562+
XLByteToSeg(keep, segno, wal_segment_size);
94749563

9475-
/* then check whether slots limit removal further */
9476-
if (max_replication_slots > 0 && keep != InvalidXLogRecPtr)
9477-
{
9478-
XLogSegNo slotSegNo;
9564+
/* Cap by max_slot_wal_keep_size ... */
9565+
if (max_slot_wal_keep_size_mb >= 0)
9566+
{
9567+
XLogRecPtr slot_keep_segs;
94799568

9480-
XLByteToSeg(keep, slotSegNo, wal_segment_size);
9569+
slot_keep_segs =
9570+
ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
94819571

9482-
if (slotSegNo <= 0)
9572+
if (currSegNo - segno > slot_keep_segs)
9573+
segno = currSegNo - slot_keep_segs;
9574+
}
9575+
}
9576+
9577+
/* but, keep at least wal_keep_segments if that's set */
9578+
if (wal_keep_segments > 0 && currSegNo - segno < wal_keep_segments)
9579+
{
9580+
/* avoid underflow, don't go below 1 */
9581+
if (currSegNo <= wal_keep_segments)
94839582
segno = 1;
9484-
else if (slotSegNo < segno)
9485-
segno = slotSegNo;
9583+
else
9584+
segno = currSegNo - wal_keep_segments;
94869585
}
94879586

94889587
/* don't delete WAL segments newer than the calculated segment */
9489-
if (segno < *logSegNo)
9588+
if (XLogRecPtrIsInvalid(*logSegNo) || segno < *logSegNo)
94909589
*logSegNo = segno;
94919590
}
94929591

‎src/backend/catalog/system_views.sql

Copy file name to clipboardExpand all lines: src/backend/catalog/system_views.sql
+3-1Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -876,7 +876,9 @@ CREATE VIEW pg_replication_slots AS
876876
L.xmin,
877877
L.catalog_xmin,
878878
L.restart_lsn,
879-
L.confirmed_flush_lsn
879+
L.confirmed_flush_lsn,
880+
L.wal_status,
881+
L.min_safe_lsn
880882
FROM pg_get_replication_slots() AS L
881883
LEFT JOIN pg_database D ON (L.datoid = D.oid);
882884

‎src/backend/replication/logical/logicalfuncs.c

Copy file name to clipboardExpand all lines: src/backend/replication/logical/logicalfuncs.c
+1-1Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,7 +225,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
225225
else
226226
end_of_wal = GetXLogReplayRecPtr(&ThisTimeLineID);
227227

228-
ReplicationSlotAcquire(NameStr(*name), true);
228+
(void) ReplicationSlotAcquire(NameStr(*name), SAB_Error);
229229

230230
PG_TRY();
231231
{

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.