Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add memory metrics and refact memory allocation #14710

Merged
merged 220 commits into from
Mar 7, 2025
Merged
Changes from 1 commit
Commits
Show all changes
220 commits
Select commit Hold shift + click to select a range
a4a0e29
fix memory concurrency problem
SpriCoder Jul 22, 2024
8302be7
Merge branch 'master' into feature/optimize_memory
SpriCoder Jul 22, 2024
f4f65e4
merge
SpriCoder Dec 25, 2024
4e3fa96
Merge branch 'master' into feature/optimize_memory
SpriCoder Dec 26, 2024
7307bc5
add memory related metric name and tag
SpriCoder Dec 26, 2024
b5d88a6
add memory threshold code
SpriCoder Dec 26, 2024
05b0757
add total
SpriCoder Dec 27, 2024
4a4092d
Merge branch 'master' into feature/optimize_memory
SpriCoder Dec 28, 2024
01ad415
add part of write metrics
SpriCoder Dec 30, 2024
d019c60
remove hard string code
SpriCoder Dec 30, 2024
b97b406
fix error code(set storage engine size to schema engine
SpriCoder Dec 30, 2024
57d529b
degrade metric level
SpriCoder Dec 30, 2024
1d23592
fix error level
SpriCoder Dec 30, 2024
44adcdb
fix error type and add datanode device path cache
SpriCoder Dec 30, 2024
713e39b
finish write part
SpriCoder Dec 30, 2024
a093a1c
fix format
SpriCoder Dec 30, 2024
79e7a9e
fix name bug
SpriCoder Dec 30, 2024
2a4c275
move buffer pool to memtable
SpriCoder Dec 30, 2024
e06cd79
Merge branch 'master' into feature/optimize_memory
SpriCoder Jan 2, 2025
8fc262d
fix comment
SpriCoder Jan 2, 2025
deadee7
Merge branch 'master' into feature/optimize_memory
SpriCoder Jan 2, 2025
2de1a03
optimize the code
SpriCoder Jan 3, 2025
81665bc
Merge branch 'master' into feature/optimize_memory
SpriCoder Jan 3, 2025
4f1db4e
add Query Memory Threshold(level=2)
SpriCoder Jan 3, 2025
3fa6088
add Schema Memory Threshold(level=2)
SpriCoder Jan 3, 2025
e0ce0af
add Schema Memory Threshold(update comment)
SpriCoder Jan 3, 2025
468e1c9
move Consensus memory threshold
SpriCoder Jan 3, 2025
d26cc05
move Consensus memory threshold
SpriCoder Jan 3, 2025
3273bff
split class to optimize code
SpriCoder Jan 3, 2025
5d17be6
Add Off Heap Memory Total
SpriCoder Jan 3, 2025
6442813
Merge branch 'master' into feature/optimize_memory
SpriCoder Jan 6, 2025
9bcfe05
Merge branch 'master' into feature/optimize_memory
SpriCoder Jan 6, 2025
9bf1bc8
change metric name
SpriCoder Jan 6, 2025
ba88514
add Metric memory_actual_size
SpriCoder Jan 6, 2025
a9a0030
Merge branch 'master' into feature/optimize_memory
SpriCoder Jan 8, 2025
7c4aeb9
init first version of memory manager and memory block
SpriCoder Jan 8, 2025
cc89021
add IIoTDBMemoryBlock
SpriCoder Jan 8, 2025
e33d234
fix template
SpriCoder Jan 8, 2025
bbd4383
Merge branch 'master' into feature/optimize_memory
SpriCoder Jan 9, 2025
eb465bd
rollback
SpriCoder Jan 9, 2025
72bc9c5
rollback 2
SpriCoder Jan 9, 2025
8c6af76
Merge branch 'master' into feature/memory_collect
SpriCoder Jan 13, 2025
03a3daf
Merge branch 'master' into feature/memory_collect
SpriCoder Jan 16, 2025
57f1f7b
Rename IoTDBRuntimeOutOfMemoryException to IoTDBMemoryException
SpriCoder Jan 16, 2025
d4c5d17
Rename IoTDBRuntimeOutOfMemoryException to IoTDBMemoryException
SpriCoder Jan 16, 2025
5b5fe77
add version2 memory
SpriCoder Jan 16, 2025
8611198
add name of MemoryManager and MemoryBlock
SpriCoder Jan 16, 2025
721143b
add global memory manager
SpriCoder Jan 16, 2025
8e988fc
add search memory manager
SpriCoder Jan 16, 2025
0967abd
Merge branch 'master' into feature/memory_collect
SpriCoder Jan 20, 2025
9b36c61
add default method to get memory manager
SpriCoder Jan 20, 2025
3596c09
modify storage engine
SpriCoder Jan 20, 2025
681f6f3
fix
SpriCoder Jan 20, 2025
94d57f9
update
SpriCoder Jan 20, 2025
d184523
update
SpriCoder Jan 20, 2025
fd336a6
fix init problem
SpriCoder Jan 20, 2025
b4d4b2d
fix init problem
SpriCoder Jan 20, 2025
dcfd4b9
TimePartitionInfo
SpriCoder Jan 21, 2025
1ddd0be
Merge branch 'master' into feature/memory_collect
SpriCoder Jan 21, 2025
6356b48
Add TimePartitionInfo
SpriCoder Jan 21, 2025
105ddf1
Add Consensus Memory Manager
SpriCoder Jan 21, 2025
ba536e8
Add Pipe Memory Manager
SpriCoder Jan 21, 2025
d7ce233
Merge branch 'master' into feature/memory_collect
SpriCoder Jan 23, 2025
e63306e
Add Schema Memory Manager
SpriCoder Jan 23, 2025
9864aa7
add query memory manager
SpriCoder Jan 23, 2025
6e97092
fix operators memory
SpriCoder Jan 23, 2025
0429a02
Merge branch 'master' into feature/memory_collect
SpriCoder Jan 24, 2025
d7db51f
move now into memory manager
SpriCoder Jan 24, 2025
552b4b4
add wal buffer queue size metrics
SpriCoder Jan 24, 2025
0eabba8
remove TestOnly Annotation
SpriCoder Jan 24, 2025
002c97d
add comment and optimize code
SpriCoder Jan 25, 2025
ca96958
Merge branch 'master' into feature/memory_collect
SpriCoder Jan 25, 2025
026f257
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 2, 2025
a6c2467
split off heap
SpriCoder Feb 2, 2025
599f04a
bug fix
SpriCoder Feb 2, 2025
743b539
move compaction into block
SpriCoder Feb 4, 2025
fcafc2b
add metric of actual size
SpriCoder Feb 4, 2025
7274b84
fix tag name
SpriCoder Feb 4, 2025
1ba1ad7
fix level
SpriCoder Feb 4, 2025
9f974e9
add actural size of memory of memtable
SpriCoder Feb 4, 2025
f1d3e35
have a try
SpriCoder Feb 4, 2025
1a54125
test
SpriCoder Feb 4, 2025
3642591
test
SpriCoder Feb 4, 2025
16fd24e
return
SpriCoder Feb 4, 2025
fd70a8a
Revert "return"
SpriCoder Feb 4, 2025
099107d
Revert "test"
SpriCoder Feb 4, 2025
6899eb7
Revert "test"
SpriCoder Feb 4, 2025
a10db70
Revert "have a try"
SpriCoder Feb 4, 2025
f94a60d
Revert "add actural size of memory of memtable"
SpriCoder Feb 4, 2025
13f143e
add storage level=4
SpriCoder Feb 5, 2025
60fbc1b
fix wait-notify
SpriCoder Feb 5, 2025
7d2078f
fix wait-notify and statistic
SpriCoder Feb 5, 2025
a7430f0
spotless
SpriCoder Feb 5, 2025
a01495f
init IT test
SpriCoder Feb 5, 2025
3e2efb5
avoid notify
SpriCoder Feb 5, 2025
016b2d6
Fix bug in Memory Manager and Memory Block
SpriCoder Feb 5, 2025
af53ad8
Add License
SpriCoder Feb 5, 2025
4125aa6
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 5, 2025
ad429ff
Empty-Commit
SpriCoder Feb 5, 2025
db47a1a
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 6, 2025
b0b833f
rollback message in PipeMemoryManager
SpriCoder Feb 6, 2025
d2193ce
fix
SpriCoder Feb 6, 2025
a6f86ea
Add some log
SpriCoder Feb 6, 2025
89037f0
fix reclaim bug
SpriCoder Feb 6, 2025
768e606
avoid multi-create
SpriCoder Feb 6, 2025
f79eacc
remove debug log
SpriCoder Feb 6, 2025
eb7e506
fix typo
SpriCoder Feb 6, 2025
f4360c8
Modify TimePartitionMemoryController
SpriCoder Feb 6, 2025
07d4bce
Modify BloomFilterCache Memory Block
SpriCoder Feb 6, 2025
f7592ba
Modify ChunkCache Memory Block
SpriCoder Feb 6, 2025
175e2d5
Modify TimeSeriesMetadataCache Memory Block
SpriCoder Feb 6, 2025
d205162
Modify Operators Memory Block
SpriCoder Feb 6, 2025
52ba772
Modify DataExchange Memory Block
SpriCoder Feb 6, 2025
173a6d5
spotless
SpriCoder Feb 6, 2025
09d0a2e
use allocate method
SpriCoder Feb 7, 2025
50e620c
Modify TimeIndex Memory Block
SpriCoder Feb 7, 2025
3d7f891
Update StorageEngineMemoryMetrics
SpriCoder Feb 7, 2025
1acea93
Modify Gauge to AutoGauge
SpriCoder Feb 7, 2025
b2aaecf
Fix error metric name
SpriCoder Feb 7, 2025
4735711
Update StreamEngineMetrics
SpriCoder Feb 7, 2025
0b9f26c
Add Consensus Memory Block
SpriCoder Feb 7, 2025
8355289
Add Schema Memory Block
SpriCoder Feb 7, 2025
33dc17b
Fix NPE in consensus test
SpriCoder Feb 7, 2025
4661a3c
avoid same name memory block
SpriCoder Feb 8, 2025
99ff70b
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 8, 2025
7a415b3
Fix return bug in clearAll
SpriCoder Feb 9, 2025
1fcd72b
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 10, 2025
ac47dba
Upgrade Metric Level
SpriCoder Feb 10, 2025
a4db9c4
Fix info log
SpriCoder Feb 10, 2025
ae79167
Fix info log
SpriCoder Feb 10, 2025
15c216c
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 11, 2025
a3617f4
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 13, 2025
e38b155
Fix concurrent problem
SpriCoder Feb 13, 2025
41f734d
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 17, 2025
1ff79a7
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 18, 2025
963cb11
Rename IoTDBRuntimeOutOfMemoryException to IoTDBMemoryException
SpriCoder Feb 19, 2025
7b94bd5
Fix conflict
SpriCoder Feb 19, 2025
69774c9
Fix conflict
SpriCoder Feb 20, 2025
4ddd968
Spotless
SpriCoder Feb 20, 2025
80899c9
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 20, 2025
fea16a8
Fix method name
SpriCoder Feb 20, 2025
ae5b6f3
Fix bug around remove of memory block
SpriCoder Feb 20, 2025
69d7840
solve conflict
SpriCoder Feb 21, 2025
7ec745d
solve all conflict
SpriCoder Feb 21, 2025
ca8687b
add simple auto
SpriCoder Feb 21, 2025
db3aa3c
fix format
SpriCoder Feb 21, 2025
2ba78d5
Fix start
SpriCoder Feb 21, 2025
04124a9
Add Log
SpriCoder Feb 21, 2025
576b3a1
Add Memory Service
SpriCoder Feb 21, 2025
aa6ed4e
Change higher and lower
SpriCoder Feb 21, 2025
b2d0567
Add Config
SpriCoder Feb 22, 2025
839f9db
Add test log
SpriCoder Feb 22, 2025
35a0fa3
Add test log
SpriCoder Feb 22, 2025
bccae9d
Add test log
SpriCoder Feb 22, 2025
7d07a86
Fix auto
SpriCoder Feb 22, 2025
2981b62
Change higher and lower
SpriCoder Feb 22, 2025
001a1dc
Change higher and lower
SpriCoder Feb 22, 2025
cc5d542
Change higher and lower and add check
SpriCoder Feb 22, 2025
9cad84d
Optimize judge
SpriCoder Feb 22, 2025
54208b0
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 22, 2025
017a48a
Merge branch 'feature/memory_collect' into feature/memory_auto
SpriCoder Feb 22, 2025
b253e99
Fix pipe bug when merge
SpriCoder Feb 22, 2025
2902cf1
Merge branch 'feature/memory_collect' into feature/memory_auto
SpriCoder Feb 22, 2025
14f172c
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 25, 2025
dfb9846
Merge branch 'master' into feature/memory_auto
SpriCoder Feb 25, 2025
ecaac56
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 26, 2025
6f916c1
move some manager to memory config
SpriCoder Feb 26, 2025
90e0078
move some manager to memory config(finish)
SpriCoder Feb 27, 2025
1a4ac0e
Merge branch 'master' into feature/memory_collect
SpriCoder Feb 27, 2025
1e6e126
merge feature/memory_collect
SpriCoder Feb 27, 2025
26319c7
update format
SpriCoder Feb 27, 2025
79f602a
Fix Agent
SpriCoder Feb 27, 2025
ff8b096
Fix IT
SpriCoder Feb 27, 2025
d7dd29f
Merge branch 'feature/memory_collect' into feature/memory_auto
SpriCoder Feb 27, 2025
24770b9
Fix Symbol
SpriCoder Feb 27, 2025
b7c0339
Fix Format
SpriCoder Feb 27, 2025
063dc86
Merge branch 'feature/memory_collect' into feature/memory_auto
SpriCoder Feb 27, 2025
5205e3c
Merge branch 'master' into feature/memory_collect
SpriCoder Mar 3, 2025
bcbab69
Merge branch 'feature/memory_collect' into feature/memory_auto
SpriCoder Mar 3, 2025
42474ce
Add MemoryBlock UT Test
SpriCoder Mar 3, 2025
48b89da
Merge branch 'feature/memory_collect' into feature/memory_auto
SpriCoder Mar 3, 2025
024ec34
Add MemoryBlock UT Test
SpriCoder Mar 3, 2025
00677f1
Update Auto Logic
SpriCoder Mar 3, 2025
8e74cd6
Add Score Method
SpriCoder Mar 3, 2025
b7d4485
Move Logic
SpriCoder Mar 3, 2025
c981138
Remove to DataNodeMemoryConfig
SpriCoder Mar 3, 2025
4f61ff4
Add Config
SpriCoder Mar 3, 2025
685a03c
Move into IoTDBDescriptor
SpriCoder Mar 3, 2025
728af7d
Merge
SpriCoder Mar 3, 2025
82373cb
Add RAT
SpriCoder Mar 3, 2025
3c2b26d
Change Name
SpriCoder Mar 3, 2025
db8e504
Merge branch 'feature/memory_collect' into feature/memory_auto
SpriCoder Mar 3, 2025
4d371f5
Add Import
SpriCoder Mar 3, 2025
2b64073
Add Import
SpriCoder Mar 3, 2025
ca75879
Merge branch 'feature/memory_collect' into feature/memory_auto
SpriCoder Mar 3, 2025
f37f2cc
Add Import
SpriCoder Mar 3, 2025
f08b37b
Merge branch 'master' into feature/memory_collect
SpriCoder Mar 3, 2025
4714945
Merge Master
SpriCoder Mar 3, 2025
b1f047a
Fix Format
SpriCoder Mar 3, 2025
27f87b6
Merge branch 'feature/memory_collect' into feature/memory_auto
SpriCoder Mar 3, 2025
b00d1b1
Fix Format
SpriCoder Mar 3, 2025
7095c32
Fix Format
SpriCoder Mar 3, 2025
fa37a34
Merge branch 'feature/memory_auto' into feature/memory_collect
SpriCoder Mar 3, 2025
558d696
Fix IT/UT
SpriCoder Mar 3, 2025
7089406
Fix Some
SpriCoder Mar 4, 2025
4dc0e69
Fix Before
SpriCoder Mar 4, 2025
c4a2176
Merge branch 'master' into feature/memory_collect
SpriCoder Mar 5, 2025
f0e8b03
Add ratio check
SpriCoder Mar 5, 2025
8eb0603
Reset
SpriCoder Mar 5, 2025
146635b
Reset and remove
SpriCoder Mar 5, 2025
7b10993
Reset and remove
SpriCoder Mar 5, 2025
1cda223
Reset and remove
SpriCoder Mar 5, 2025
3403861
Remove
SpriCoder Mar 5, 2025
8a77f36
Merge branch 'master' into feature/memory_collect
SpriCoder Mar 6, 2025
2a370df
Rename forceAllocate to exactAllocate
SpriCoder Mar 6, 2025
ebc84e6
Rename registerMemoryBlock to getOrCreate
SpriCoder Mar 6, 2025
a2d3dca
Update getOrCreateMemoryManager
SpriCoder Mar 6, 2025
24d7f9a
Update getOrCreateMemoryManager
SpriCoder Mar 6, 2025
d9436cb
Update Memory Manager
SpriCoder Mar 6, 2025
92479a6
Fix Log
SpriCoder Mar 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix format
SpriCoder committed Dec 30, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit a093a1c9bcd2761cd04554a56903f27e7032dce9
Original file line number Diff line number Diff line change
@@ -58,7 +58,8 @@ public class ThresholdMemoryMetrics implements IMetricSet {
private static final String STORAGE_ENGINE = "StorageEngine";
private static final String STORAGE_ENGINE_WRITE = "StorageEngine-Write";
private static final String STORAGE_ENGINE_WRITE_MEMTABLE = "StorageEngine-Write-Memtable";
private static final String STORAGE_ENGINE_WRITE_MEMTABLE_CACHE = "StorageEngine-Write-Memtable-DevicePathCache";
private static final String STORAGE_ENGINE_WRITE_MEMTABLE_CACHE =
"StorageEngine-Write-Memtable-DevicePathCache";
private static final String STORAGE_ENGINE_WRITE_TIME_PARTITION_INFO =
"StorageEngine-Write-TimePartitionInfo";
private static final String STORAGE_ENGINE_WRITE_BUFFERED_ARRAYS =
@@ -162,8 +163,9 @@ private void bindStorageEngineRelatedMemoryMetrics(AbstractMetricService metricS
(config.getAllocateMemoryForStorageEngine() * (1 - config.getCompactionProportion()));
long compactionSize = storageEngineSize - writeSize;
long bufferedArraySize =
(long) (config.getAllocateMemoryForStorageEngine()
* config.getBufferedArraysMemoryProportion());
(long)
(config.getAllocateMemoryForStorageEngine()
* config.getBufferedArraysMemoryProportion());
writeMemorySize =
metricService.getOrCreateGauge(
Metric.THRESHOLD_MEMORY_SIZE.toString(),
@@ -202,7 +204,8 @@ private void bindStorageEngineRelatedMemoryMetrics(AbstractMetricService metricS
Tag.LEVEL.toString(),
LEVELS[3]);
memtableMemorySize.set(memtableSize);
// The memtable memory of storage engine contain DataNodeDevicePathCache (NOTICE: This part of memory is not divided)
// The memtable memory of storage engine contain DataNodeDevicePathCache (NOTICE: This part of
// memory is not divided)
long dataNodeDevicePathCacheSize =
(long)
(config.getAllocateMemoryForStorageEngine()
@@ -302,7 +305,10 @@ public void unbindFrom(AbstractMetricService metricService) {
memtableMemorySize = DoNothingMetricManager.DO_NOTHING_GAUGE;
timePartitionInfoMemorySize = DoNothingMetricManager.DO_NOTHING_GAUGE;
bufferedArraysMemorySize = DoNothingMetricManager.DO_NOTHING_GAUGE;
Arrays.asList(STORAGE_ENGINE_WRITE_MEMTABLE, STORAGE_ENGINE_WRITE_TIME_PARTITION_INFO, STORAGE_ENGINE_WRITE_BUFFERED_ARRAYS)
Arrays.asList(
STORAGE_ENGINE_WRITE_MEMTABLE,
STORAGE_ENGINE_WRITE_TIME_PARTITION_INFO,
STORAGE_ENGINE_WRITE_BUFFERED_ARRAYS)
.forEach(
name ->
metricService.remove(