Skip to content

Improve pipe runtime memory allocation#18051

Open
Caideyipi wants to merge 1 commit into
masterfrom
pipe-dynamic-memory-allocation
Open

Improve pipe runtime memory allocation#18051
Caideyipi wants to merge 1 commit into
masterfrom
pipe-dynamic-memory-allocation

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Summary

  • allocate pipe TsFile parser, sink batch, and TsFile read buffers from PipeMemoryManager only while active
  • resize tablet and IoTConsensusV2 batch memory by actual buffered data, and release it on success, discard, or close
  • bound TsFile read buffer allocation by the current file size across thrift, airgap, legacy, and IoTConsensusV2 sinks

Tests

  • mvn spotless:apply -pl iotdb-core/datanode
  • git diff --check
  • mvn -pl iotdb-core/datanode -DskipTests compile (fails on existing non-pipe compile issues: missing commons-collections4 ComparatorChain, missing SystemConstant authentication keys, missing ColumnHeaderConstant showCreate headers, and related generated aggregation symbols)

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
11.7% Duplication on New Code (required ≤ 5%)

See analysis details on SonarQube Cloud

@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 17.99163% with 196 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.40%. Comparing base (fcd52d7) to head (b8a28f5).

Files with missing lines Patch % Lines
.../protocol/thrift/sync/IoTDBDataRegionSyncSink.java 0.00% 55 Missing ⚠️
...builder/IoTConsensusV2TransferBatchReqBuilder.java 0.00% 50 Missing ⚠️
...oad/evolvable/batch/PipeTabletEventPlainBatch.java 14.28% 18 Missing ⚠️
...ler/IoTConsensusV2TsFileInsertionEventHandler.java 0.00% 18 Missing ⚠️
...hrift/async/handler/PipeTransferTsFileHandler.java 0.00% 18 Missing ⚠️
.../payload/evolvable/batch/PipeTabletEventBatch.java 61.76% 13 Missing ⚠️
...rotocol/iotconsensusv2/IoTConsensusV2SyncSink.java 0.00% 10 Missing ⚠️
...pipe/sink/protocol/legacy/IoTDBLegacyPipeSink.java 0.00% 9 Missing ⚠️
...ad/evolvable/batch/PipeTabletEventTsFileBatch.java 44.44% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             master   #18051    +/-   ##
==========================================
  Coverage     41.40%   41.40%            
  Complexity      318      318            
==========================================
  Files          5286     5286            
  Lines        369547   369685   +138     
  Branches      47815    47828    +13     
==========================================
+ Hits         153008   153086    +78     
- Misses       216539   216599    +60     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant