Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit cec2edf

Browse filesBrowse files
author
Amit Kapila
committed
Add logical_decoding_work_mem to limit ReorderBuffer memory usage.
Instead of deciding to serialize a transaction merely based on the number of changes in that xact (toplevel or subxact), this makes the decisions based on amount of memory consumed by the changes. The memory limit is defined by a new logical_decoding_work_mem GUC, so for example we can do this SET logical_decoding_work_mem = '128kB' to reduce the memory usage of walsenders or set the higher value to reduce disk writes. The minimum value is 64kB. When adding a change to a transaction, we account for the size in two places. Firstly, in the ReorderBuffer, which is then used to decide if we reached the total memory limit. And secondly in the transaction the change belongs to, so that we can pick the largest transaction to evict (and serialize to disk). We still use max_changes_in_memory when loading changes serialized to disk. The trouble is we can't use the memory limit directly as there might be multiple subxact serialized, we need to read all of them but we don't know how many are there (and which subxact to read first). We do not serialize the ReorderBufferTXN entries, so if there is a transaction with many subxacts, most memory may be in this type of objects. Those records are not included in the memory accounting. We also do not account for INTERNAL_TUPLECID changes, which are kept in a separate list and not evicted from memory. Transactions with many CTID changes may consume significant amounts of memory, but we can't really do much about that. The current eviction algorithm is very simple - the transaction is picked merely by size, while it might be useful to also consider age (LSN) of the changes for example. With the new Generational memory allocator, evicting the oldest changes would make it more likely the memory gets actually pfreed. The logical_decoding_work_mem can be set in postgresql.conf, in which case it serves as the default for all publishers on that instance. Author: Tomas Vondra, with changes by Dilip Kumar and Amit Kapila Reviewed-by: Dilip Kumar and Amit Kapila Tested-By: Vignesh C Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com
1 parent 2110f71 commit cec2edf
Copy full SHA for cec2edf

File tree

Expand file treeCollapse file tree

6 files changed

+333
-12
lines changed
Filter options
Expand file treeCollapse file tree

6 files changed

+333
-12
lines changed

‎contrib/test_decoding/logical.conf

Copy file name to clipboard
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
wal_level = logical
22
max_replication_slots = 4
3+
logical_decoding_work_mem = 64kB

‎doc/src/sgml/config.sgml

Copy file name to clipboardExpand all lines: doc/src/sgml/config.sgml
+21Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1732,6 +1732,27 @@ include_dir 'conf.d'
17321732
</listitem>
17331733
</varlistentry>
17341734

1735+
<varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
1736+
<term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
1737+
<indexterm>
1738+
<primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
1739+
</indexterm>
1740+
</term>
1741+
<listitem>
1742+
<para>
1743+
Specifies the maximum amount of memory to be used by logical decoding,
1744+
before some of the decoded changes are written to local disk. This
1745+
limits the amount of memory used by logical streaming replication
1746+
connections. It defaults to 64 megabytes (<literal>64MB</literal>).
1747+
Since each replication connection only uses a single buffer of this size,
1748+
and an installation normally doesn't have many such connections
1749+
concurrently (as limited by <varname>max_wal_senders</varname>), it's
1750+
safe to set this value significantly higher than <varname>work_mem</varname>,
1751+
reducing the amount of decoded changes written to disk.
1752+
</para>
1753+
</listitem>
1754+
</varlistentry>
1755+
17351756
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
17361757
<term><varname>max_stack_depth</varname> (<type>integer</type>)
17371758
<indexterm>

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.