Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor iterate git tree to reduce memory usage #31076

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

lunny
Copy link
Member

@lunny lunny commented May 25, 2024

Make git tree recursive iteration stoppable and reduce the memory usage when calculating language stats.

@lunny lunny added type/refactoring Existing code has been cleaned up. There should be no new functionality. performance/memory Performance issues affecting memory use labels May 25, 2024
@lunny lunny added this to the 1.23.0 milestone May 25, 2024
@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label May 25, 2024
@pull-request-size pull-request-size bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 25, 2024
@github-actions github-actions bot added the modifies/go Pull requests that update Go code label May 25, 2024
@lunny lunny modified the milestones: 1.23.0, 1.24.0 Sep 7, 2024
@lunny lunny force-pushed the lunny/performance_big_reference_repo branch from eaae69e to 7bdf686 Compare December 30, 2024 01:50
@lunny lunny marked this pull request as ready for review December 31, 2024 07:02
@wxiaoguang
Copy link
Contributor

It doesn't seem right, especially the different "ctx" are mixed together.

And in the end, you still read everything into memory entries = append(entries, entry)


To make it overall right, it needs to consume the outputs on the fly, and make the git command has ability to limit the read data size.

@wxiaoguang wxiaoguang removed their request for review December 31, 2024 17:34
@lunny
Copy link
Member Author

lunny commented Dec 31, 2024

It doesn't seem right, especially the different "ctx" are mixed together.

And in the end, you still read everything into memory entries = append(entries, entry)

To make it overall right, it needs to consume the outputs on the fly, and make the git command has ability to limit the read data size.

This PR introduced an iteration method and only reduces memory when calculating language stats. For tree list, they are kept as before and we can improve it in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. modifies/go Pull requests that update Go code performance/memory Performance issues affecting memory use size/L Denotes a PR that changes 100-499 lines, ignoring generated files. type/refactoring Existing code has been cleaned up. There should be no new functionality.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants