Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit b383e4d

Browse filesBrowse files
ankaneitaispiegel
ankane
and
itaispiegel
committed
Reduced memory usage for small tables for IVFFlat index builds - resolves #995 and resolves #996
Co-authored-by: Itai Spiegel <itai@mave.com>
1 parent 1d458ad commit b383e4d
Copy full SHA for b383e4d

4 files changed

+54Lines changed: 54 additions & 0 deletions

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎CHANGELOG.md‎

Copy file name to clipboardExpand all lines: CHANGELOG.md
+4Lines changed: 4 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
## 0.8.5 (unreleased)
2+
3+
- Reduced memory usage for small tables for IVFFlat index builds
4+
15
## 0.8.4 (2026-06-30)
26

37
- Fixed `hnsw graph not repaired` error with HNSW vacuuming
Collapse file

‎src/ivfbuild.c‎

Copy file name to clipboardExpand all lines: src/ivfbuild.c
+7Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -447,6 +447,13 @@ ComputeCenters(IvfflatBuildState * buildstate)
447447
/* Skip samples for unlogged table */
448448
if (buildstate->heap == NULL)
449449
numSamples = 1;
450+
else
451+
{
452+
int64 maxTuples = (int64) RelationGetNumberOfBlocks(buildstate->heap) * MaxHeapTuplesPerPage;
453+
454+
/* Save memory since will not have more than max tuples */
455+
numSamples = Max(Min(numSamples, maxTuples), 1);
456+
}
450457

451458
/* Sample rows */
452459
buildstate->memoryUsed += VECTOR_ARRAY_SIZE(numSamples, buildstate->itemsize);
Collapse file

‎test/expected/ivfflat_vector.out‎

Copy file name to clipboardExpand all lines: test/expected/ivfflat_vector.out
+24Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,3 +173,27 @@ ERROR: 0 is outside the valid range for parameter "ivfflat.max_probes" (1 .. 32
173173
SET ivfflat.max_probes = 32769;
174174
ERROR: 32769 is outside the valid range for parameter "ivfflat.max_probes" (1 .. 32768)
175175
DROP TABLE t;
176+
-- memory
177+
CREATE TABLE t (val vector(2000));
178+
CREATE INDEX ON t USING ivfflat (val vector_l2_ops) WITH (lists = 4096);
179+
NOTICE: ivfflat index created with little data
180+
DETAIL: This will cause low recall.
181+
HINT: Drop the index until the table has more data.
182+
DROP TABLE t;
183+
SET maintenance_work_mem = '1MB';
184+
CREATE TABLE t (val vector(2000));
185+
CREATE INDEX ON t USING ivfflat (val vector_l2_ops);
186+
NOTICE: ivfflat index created with little data
187+
DETAIL: This will cause low recall.
188+
HINT: Drop the index until the table has more data.
189+
DROP TABLE t;
190+
RESET maintenance_work_mem;
191+
SET maintenance_work_mem = '5MB';
192+
CREATE TABLE t (val vector(2000));
193+
INSERT INTO t (val) VALUES (array_fill(0, ARRAY[2000]));
194+
CREATE INDEX ON t USING ivfflat (val vector_l2_ops);
195+
NOTICE: ivfflat index created with little data
196+
DETAIL: This will cause low recall.
197+
HINT: Drop the index until the table has more data.
198+
DROP TABLE t;
199+
RESET maintenance_work_mem;
Collapse file

‎test/sql/ivfflat_vector.sql‎

Copy file name to clipboardExpand all lines: test/sql/ivfflat_vector.sql
+19Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,3 +97,22 @@ SET ivfflat.max_probes = 0;
9797
SET ivfflat.max_probes = 32769;
9898

9999
DROP TABLE t;
100+
101+
-- memory
102+
103+
CREATE TABLE t (val vector(2000));
104+
CREATE INDEX ON t USING ivfflat (val vector_l2_ops) WITH (lists = 4096);
105+
DROP TABLE t;
106+
107+
SET maintenance_work_mem = '1MB';
108+
CREATE TABLE t (val vector(2000));
109+
CREATE INDEX ON t USING ivfflat (val vector_l2_ops);
110+
DROP TABLE t;
111+
RESET maintenance_work_mem;
112+
113+
SET maintenance_work_mem = '5MB';
114+
CREATE TABLE t (val vector(2000));
115+
INSERT INTO t (val) VALUES (array_fill(0, ARRAY[2000]));
116+
CREATE INDEX ON t USING ivfflat (val vector_l2_ops);
117+
DROP TABLE t;
118+
RESET maintenance_work_mem;

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.