Skip to content

Fix unnecesary data copy during the curl calls. #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 7, 2025

Conversation

Dwordcito
Copy link
Member

Description

This PR aims to remove unnecesary copies during the some methods calls(when we have a data as parameter)

Copy link
Contributor

@pereyra-m pereyra-m left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Dwordcito

I've made a local test and I'm not sure about the result I have.

I've run the Post100Mbs test with both the dev branch and 1.0.0, adding an extra message:

    std::cout << "RSS usage: " << postPost - prePost << std::endl;

These are the results

dev

root@042fbd6253cf:/workspaces/wazuh-http-request/build/test/component# ./urlrequest_component_test --gtest_filter="*Post100Mbs*"
Note: Google Test filter = *Post100Mbs*
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ComponentTestInterface
[ RUN      ] ComponentTestInterface.Post100Mbs
RSS usage: 237320
[       OK ] ComponentTestInterface.Post100Mbs (514 ms)
[----------] 1 test from ComponentTestInterface (514 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (521 ms total)
[  PASSED  ] 1 test.

1.0.0


root@042fbd6253cf:/workspaces/wazuh-http-request/build/test/component# ./urlrequest_component_test --gtest_filter="*Post100Mbs*"
Note: Google Test filter = *Post100Mbs*
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ComponentTestInterface
[ RUN      ] ComponentTestInterface.Post100Mbs
RSS usage: 233072
[       OK ] ComponentTestInterface.Post100Mbs (694 ms)
[----------] 1 test from ComponentTestInterface (694 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (700 ms total)
[  PASSED  ] 1 test.

This comparison doesn't reflect the improvement in RSS (although it seems faster), what could be the problem?

@Dwordcito
Copy link
Member Author

add the log/call(to get the rss) inner the post method in httpRequest.cpp because the peak is removed from this section to check the enhancement.

Copy link
Member

@Damian-Mangold Damian-Mangold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VDP was updated to make use of these changes and no errors were detected in its execution.

(1/16) ALAS1 Migration: Completed
(2/16) ALAS2 Migration: Completed
(3/16) ALAS2022 Migration: Completed
(4/16) ALAS2023 Migration: Completed
(5/16) AlmaLinux Migration: Completed
(6/16) ArchLinux Migration: Completed
(7/16) NVD Migration: Completed
(8/16) Rocky Migration: Completed
(9/16) Suse Migration: Completed
(10/16) Npm Migration: Completed
(11/16) PyPI Migration: Completed
(12/16) Data Sanitization: Completed
(13/16) Debian Migration: Completed
(14/16) OracleLinux Migration: Completed
(15/16) RedHat Migration: Completed
(16/16) Ubuntu Migration: Completed

(1/1) Content merge: Completed

Copy link
Member

@Damian-Mangold Damian-Mangold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

2025-03-07 09:55:36,215:INFO:root:     Stage 'Content migration (1/3) and sanitization': finished processes 0/9
2025-03-07 09:55:36,216:INFO:root:      (1/9) ALAS1 Migration: Running
2025-03-07 09:55:36,216:INFO:root:      (2/9) ALAS2 Migration: Running
2025-03-07 09:55:36,216:INFO:root:      (3/9) ALAS2022 Migration: Running
2025-03-07 09:55:36,216:INFO:root:      (4/9) ALAS2023 Migration: Running
2025-03-07 09:55:36,216:INFO:root:      (5/9) AlmaLinux Migration: Running
2025-03-07 09:55:36,216:INFO:root:      (6/9) ArchLinux Migration: Running
2025-03-07 09:55:36,216:INFO:root:      (7/9) Npm Migration: Running
2025-03-07 09:55:36,216:INFO:root:      (8/9) PyPI Migration: Running
2025-03-07 09:55:36,216:INFO:root:      (9/9) Data Sanitization: Running
2025-03-07 09:56:08,249:INFO:root:     Stage 'Content migration (1/3) and sanitization': finished processes 9/9
2025-03-07 09:56:08,249:INFO:root:      (1/9) ALAS1 Migration: Completed
2025-03-07 09:56:08,249:INFO:root:      (2/9) ALAS2 Migration: Completed
2025-03-07 09:56:08,249:INFO:root:      (3/9) ALAS2022 Migration: Completed
2025-03-07 09:56:08,249:INFO:root:      (4/9) ALAS2023 Migration: Completed
2025-03-07 09:56:08,249:INFO:root:      (5/9) AlmaLinux Migration: Completed
2025-03-07 09:56:08,249:INFO:root:      (6/9) ArchLinux Migration: Completed
2025-03-07 09:56:08,249:INFO:root:      (7/9) Npm Migration: Completed
2025-03-07 09:56:08,250:INFO:root:      (8/9) PyPI Migration: Completed
2025-03-07 09:56:08,250:INFO:root:      (9/9) Data Sanitization: Completed
2025-03-07 09:56:08,250:INFO:root:   Stage 'Content migration (1/3) and sanitization' finished. Processes completed successfully.
2025-03-07 09:56:08,250:INFO:root:   Starting stage 'Content merge'
2025-03-07 09:56:08,250:INFO:root:     Starting process 'Content merge'
2025-03-07 09:56:08,250:INFO:root:     Waiting for all processes to finish...
2025-03-07 09:56:08,250:INFO:root:     Stage 'Content merge': finished processes 0/1
2025-03-07 09:56:08,250:INFO:root:      (1/1) Content merge: Running
2025-03-07 09:56:31,272:INFO:root:     Stage 'Content merge': finished processes 1/1
2025-03-07 09:56:31,272:INFO:root:      (1/1) Content merge: Completed
2025-03-07 09:56:31,272:INFO:root:   Stage 'Content merge' finished. Processes completed successfully.
2025-03-07 09:56:31,272:INFO:root: Full Pipeline finished.

@Dwordcito Dwordcito linked an issue Mar 7, 2025 that may be closed by this pull request
2 tasks
Copy link
Member

@GabrielEValenzuela GabrielEValenzuela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Indexer connector memory peak during bulk queries
4 participants