Bug Report
1. Minimal reproduce step (Required)
Run this only on an isolated test cluster. The configuration changes below are cluster-wide.
- Save the current Load Base Split configuration:
SHOW CONFIG
WHERE TYPE = 'tikv'
AND NAME IN (
'split.qps-threshold',
'split.byte-threshold',
'split.region-cpu-overload-threshold-ratio'
);
- Lower only the QPS threshold and disable the other triggering conditions:
SET CONFIG tikv `split.qps-threshold` = 100;
SET CONFIG tikv `split.byte-threshold` = 107374182400;
SET CONFIG tikv `split.region-cpu-overload-threshold-ratio` = 0;
- Create a table whose physical rows occupy only the high end of a sparse clustered-primary-key keyspace:
CREATE DATABASE IF NOT EXISTS lbs_repro;
DROP TABLE IF EXISTS lbs_repro.sparse_pk;
CREATE TABLE lbs_repro.sparse_pk (
id VARBINARY(64) NOT NULL,
payload VARBINARY(128),
PRIMARY KEY (id) CLUSTERED
);
INSERT INTO lbs_repro.sparse_pk
VALUES
('z-real-0001', REPEAT('x', 64)),
('z-real-0002', REPEAT('x', 64));
The following hot keys are intentionally absent and sort before all physical rows.
- Record the initial Region information:
SELECT
REGION_ID,
START_KEY,
END_KEY,
APPROXIMATE_SIZE,
APPROXIMATE_KEYS
FROM INFORMATION_SCHEMA.TIKV_REGION_STATUS
WHERE DB_NAME = 'lbs_repro'
AND TABLE_NAME = 'sparse_pk'
AND IS_INDEX = 0
ORDER BY START_KEY;
- Through persistent MySQL connections, repeatedly execute two missing-key point reads at more than 100 QPS for at least 30 seconds:
SELECT id
FROM lbs_repro.sparse_pk
WHERE id = 'a-hot-missing';
SELECT id
FROM lbs_repro.sparse_pk
WHERE id = 'b-hot-missing';
For example:
mysqlslap \
--defaults-extra-file=/path/to/test-only-client.cnf \
--create-schema=lbs_repro \
--concurrency=16 \
--iterations=100 \
--number-of-queries=20000 \
--query="SELECT id FROM sparse_pk WHERE id='a-hot-missing'; SELECT id FROM sparse_pk WHERE id='b-hot-missing';"
- Check the following signals:
- TiKV logs contain
load base split region for the target Region, with split reason Load.
tikv_load_base_split_event{type="ready_to_split"} increases.
- A new record Region appears.
- One resulting Region has
APPROXIMATE_KEYS=0 or a very small value.
- Confirm that the affected logical range contains no rows:
SELECT COUNT(*)
FROM lbs_repro.sparse_pk
WHERE id < 'z-real-0001';
The result should be 0.
- To reproduce Region accumulation, change the two absent hot keys every 15–30 seconds:
phase 1: a-hot-1, b-hot-1
phase 2: c-hot-2, d-hot-2
phase 3: e-hot-3, f-hot-3
...
Record Region ID sets after each phase. Empty or near-empty Regions continue to accumulate as the hot missing-key range moves.
- Restore the original TiKV configuration recorded in step 1 and drop the test database.
2. What did you expect to see? (Required)
Load Base Split should avoid a split that leaves one child Region with no live physical keys, or it should merge the cold empty child promptly after the access pattern moves.
A candidate split key could be moved to a nearby physical key boundary, rejected when either child has insufficient keys or bytes, or protected by an empty-child cooldown.
The number of empty Regions should not continue growing from moving negative-read hotspots.
3. What did you see instead (Required)
Production impact observed
Our production workload performs high-frequency historical lookups on a sparse archive table created using IMPORT INTO ... SELECT The primary query pattern is:
SELECT ...
FROM archive_tmp
WHERE original_id = ?
AND block_height <= ?
ORDER BY block_height DESC
LIMIT 1;
Under this workload, approximately 851 empty or near-empty Regions were generated within a short period. Additional empty Regions continued to appear as long as the workload remained active.
After manually merging the confirmed empty Regions, we observed a significant and immediate performance improvement:
| Metric |
Before Merging |
After Merging |
| Query/request p99 latency |
~400 ms |
~20 ms |
| TiKV gRPC poll CPU usage |
~300% |
~100% |
The p99 latency improved by roughly 20×, while TiKV gRPC poll CPU usage decreased by about two-thirds.
These results strongly suggest that the accumulation of empty Regions is not merely a metadata concern. Instead, it introduces substantial runtime overhead in the request path. Although the precise mechanism behind the increased CPU usage requires further investigation, the observed before-and-after differences demonstrate a clear correlation between excessive empty Regions, elevated TiKV gRPC polling CPU, and increased query latency.
Because negative-read hotspots can continue to generate new empty Regions, manually merging existing Regions only addresses the symptom. Without safeguards in Load Base Split, the performance degradation is likely to recur.
Load Base Split samples requested key ranges rather than existing physical keys. Hot point reads for keys that do not exist can therefore become candidate split keys.
When the selected request key is inside a sparse physical-data gap:
- TiKV performs a load-based Region split.
- One resulting Region can contain no logical rows and report
APPROXIMATE_KEYS=0.
- Moving the hot missing keys causes additional empty or near-empty Regions to be created.
- These Regions are not merged promptly because hot Regions are skipped or delayed by merge checks.
In a production sparse archive table created using IMPORT INTO ... SELECT, we observed approximately 851 empty or near-empty Regions. TiKV logs showed Load Base Split events for this table with Region QPS around 10,000–12,000, while read throughput remained below the byte threshold.
This appears to be a performance and operability bug rather than a data-correctness bug. Using request load to choose split positions is intentional, but the algorithm lacks safeguards against repeated empty-child creation on sparse keyspaces.
Relevant implementation:
The split validation checks key ordering and Region boundaries, but does not verify that both resulting Regions contain live physical keys.
4. What is your TiDB version? (Required)
Output of SELECT tidb_version():
Release Version: v8.5.6
Edition: Community
Git Commit Hash: ae18096e023780bb56bfce33698abec0d4640d0a
Git Branch: HEAD
UTC Build Time: 2026-06-09 03:39:19
GoVersion: go1.25.8
Race Enabled: false
Check Table Before Drop: false
Store: tikv
Bug Report
1. Minimal reproduce step (Required)
The following hot keys are intentionally absent and sort before all physical rows.
For example:
mysqlslap \ --defaults-extra-file=/path/to/test-only-client.cnf \ --create-schema=lbs_repro \ --concurrency=16 \ --iterations=100 \ --number-of-queries=20000 \ --query="SELECT id FROM sparse_pk WHERE id='a-hot-missing'; SELECT id FROM sparse_pk WHERE id='b-hot-missing';"load base split regionfor the target Region, with split reasonLoad.tikv_load_base_split_event{type="ready_to_split"}increases.APPROXIMATE_KEYS=0or a very small value.The result should be
0.Record Region ID sets after each phase. Empty or near-empty Regions continue to accumulate as the hot missing-key range moves.
2. What did you expect to see? (Required)
Load Base Split should avoid a split that leaves one child Region with no live physical keys, or it should merge the cold empty child promptly after the access pattern moves.
A candidate split key could be moved to a nearby physical key boundary, rejected when either child has insufficient keys or bytes, or protected by an empty-child cooldown.
The number of empty Regions should not continue growing from moving negative-read hotspots.
3. What did you see instead (Required)
Production impact observed
Our production workload performs high-frequency historical lookups on a sparse archive table created using
IMPORT INTO ... SELECTThe primary query pattern is:Under this workload, approximately 851 empty or near-empty Regions were generated within a short period. Additional empty Regions continued to appear as long as the workload remained active.
After manually merging the confirmed empty Regions, we observed a significant and immediate performance improvement:
The p99 latency improved by roughly 20×, while TiKV gRPC poll CPU usage decreased by about two-thirds.
These results strongly suggest that the accumulation of empty Regions is not merely a metadata concern. Instead, it introduces substantial runtime overhead in the request path. Although the precise mechanism behind the increased CPU usage requires further investigation, the observed before-and-after differences demonstrate a clear correlation between excessive empty Regions, elevated TiKV gRPC polling CPU, and increased query latency.
Because negative-read hotspots can continue to generate new empty Regions, manually merging existing Regions only addresses the symptom. Without safeguards in Load Base Split, the performance degradation is likely to recur.
Load Base Split samples requested key ranges rather than existing physical keys. Hot point reads for keys that do not exist can therefore become candidate split keys.
When the selected request key is inside a sparse physical-data gap:
APPROXIMATE_KEYS=0.In a production sparse archive table created using
IMPORT INTO ... SELECT, we observed approximately 851 empty or near-empty Regions. TiKV logs showed Load Base Split events for this table with Region QPS around 10,000–12,000, while read throughput remained below the byte threshold.This appears to be a performance and operability bug rather than a data-correctness bug. Using request load to choose split positions is intentional, but the algorithm lacks safeguards against repeated empty-child creation on sparse keyspaces.
Relevant implementation:
The split validation checks key ordering and Region boundaries, but does not verify that both resulting Regions contain live physical keys.
4. What is your TiDB version? (Required)
Output of
SELECT tidb_version():