Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for agent self-restarting on Linux #386

Merged
merged 1 commit into from
Jan 28, 2025

Conversation

lchico
Copy link
Member

@lchico lchico commented Dec 4, 2024

Related issue
#77

Description

Base on the design phase, this PR implement how to handles the Wazuh agent restart process. Upon receiving a restart command, the program identifies the restart method (systemd or manual). It ensures the agent is stopped with a 30-second timeout, then restarts it. If the agent doesn’t stop gracefully, it is forcefully terminated. The process includes confirmation of the agent's successful restart.

Implementation Details:

  • Once the command is received, the system will notify that the restart process is initializing before proceeding.
  • Depending on how the agent was initially started, this module will use the same method for the restart, which could either be systemd or a manual method.
  • If systemd was used, it will be responsible for stopping and restarting the agent, ensuring the process is killed if it hasn't restarted within 30 seconds. Any unexpected errors will be reported.
  • For manual restarts, a fork is used to create a child process, which is responsible for stopping the current agent process, checking if it was closed before the timeout (also 30 seconds). If not, the agent will be killed and a new process will be started. Any unexpected errors will be reported.
  • Once the agent has restarted, a success notification will be sent.

Extra changes:

  • The SIGTERM signal name was not recognized, so I replaced it with the corresponding value to avoid the following error:
# apt remove wazuh-agent
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  lsb-release
Use 'apt autoremove' to remove it.
The following packages will be REMOVED:
  wazuh-agent
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 15.5 MB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 9917 files and directories currently installed.)
Removing wazuh-agent (5.0.0-0) ...
Call pid 4119 with sigterm to stop the service
/var/lib/dpkg/info/wazuh-agent.prerm: 43: kill: Illegal option -S
dpkg: error processing package wazuh-agent (--remove):
 installed wazuh-agent package pre-removal script subprocess returned error exit status 2
dpkg: too many errors, stopping
Errors were encountered while processing:
 wazuh-agent
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)	
  • Fixed the following warning:
root@52347fca4dff:~/output# apt purge wazuh-agent
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  lsb-release
Use 'apt autoremove' to remove it.
The following packages will be REMOVED:
  wazuh-agent*
0 upgraded, 0 newly installed, 1 to remove and 11 not upgraded.
After this operation, 16.5 MB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 7766 files and directories currently installed.)
Removing wazuh-agent (5.0.0-0) ...
(Reading database ... 7759 files and directories currently installed.)
Purging configuration files for wazuh-agent (5.0.0-0) ...
dpkg: warning: while removing wazuh-agent, directory '/etc/wazuh-agent' not empty so not removed
root@52347fca4dff:~/output# ls /etc/wazuh-agent
shared
root@52347fca4dff:~/output#  ls /etc/wazuh-agent/shared/
validYaml.yml

Test

Server

wazuh-agent/src/tests/mock-server#   ./mock-server --http

Add the restart command action to src/tests/mock-server/config/commands.groovy

def actions = [
    ...
    ["name": "restart", "version": "v5.0.0"],
    ...
]

Agent

Configure the agent to use the server address:

sed -i 's|server_url:.*|server_url: http://192.168.100.250:27000 |' /etc/wazuh-agent/wazuh-agent.yml

Register the agent

# sudo SPDLOG_LEVEL=TRACE /usr/share/wazuh-agent/bin/wazuh-agent --user wazuh --password topsecret --url http://192.168.100.250:27000 --name dummy --register

vagrant@ubuntu2204:~$ sudo SPDLOG_LEVEL=TRACE /usr/share/wazuh-agent/bin/wazuh-agent --user wazuh --password topsecret --url http://192.168.100.250:27000 --name dummy --register
[2025-01-22 00:20:25.075] [wazuh-agent] [debug] [DEBUG] [configuration_parser.cpp:54] [LoadLocalConfig] Loading local config file: /etc/wazuh-agent/wazuh-agent.yml.
[2025-01-22 00:20:25.076] [wazuh-agent] [debug] [DEBUG] [configuration_parser.hpp:147] [GetConfig] Requested setting not found, default value used. Key not found: path.data
Starting wazuh-agent registration
[2025-01-22 00:20:25.217] [wazuh-agent] [debug] [DEBUG] [http_client.cpp:296] [PerformHttpRequest] Request /security/user/authenticate: Status 200
[2025-01-22 00:20:25.217] [wazuh-agent] [trace] [TRACE] [http_client.cpp:297] [PerformHttpRequest] Request endpoint: /security/user/authenticate
Response: HTTP/1.1 200 OK
X-Imposter-Request: 6a273dcf-c7d0-4a47-86b6-0f049cb614d7
Server: imposter
content-length: 47

{ "data": { "token": "a1b2c3d4e5f6g7h8i9j0" } }
[2025-01-22 00:20:25.320] [wazuh-agent] [debug] [DEBUG] [http_client.cpp:296] [PerformHttpRequest] Request /agents: Status 201
[2025-01-22 00:20:25.320] [wazuh-agent] [trace] [TRACE] [http_client.cpp:297] [PerformHttpRequest] Request endpoint: /agents
Response: HTTP/1.1 201 Created
X-Imposter-Request: 9cdc2d69-3018-438b-b05a-bc89b36db9ff
Server: imposter
content-length: 0


wazuh-agent registered

Manually run the agent

root@ubuntu2204:~$ sudo /usr/share/wazuh-agent/bin/wazuh-agent
...
[2025-01-22 03:27:33.168] [wazuh-agent] [info] [INFO] [agent.cpp:170] [operator()] Restart: Initiating self-restart
[2025-01-22 03:27:33.174] [wazuh-agent] [info] [INFO] [inventory.cpp:62] [Stop] Inventory module stopping...
[2025-01-22 03:27:33.182] [wazuh-agent] [info] [INFO] [command_handler.hpp:105] [CommandsProcessingTask] Done processing command: restart(restart)
[2025-01-22 03:27:33.519] [wazuh-agent] [info] [INFO] [logcollector.cpp:96] [Stop] Logcollector module stopped.
[2025-01-22 03:27:33.540] [wazuh-agent] [info] [INFO] [inventoryImp.cpp:1165] [Scan] Evaluation finished.
[2025-01-22 03:27:33.587] [wazuh-agent] [info] [INFO] [inventory.cpp:37] [Start] Inventory module stopped.
root@ubuntu2204:/home/vagrant# [2025-01-22 03:27:34.175] [wazuh-agent] [info] [INFO] [restart_unix.cpp:66] [StopAgent] Agent gracefully stopped.
[2025-01-22 03:27:34.175] [wazuh-agent] [info] [INFO] [restart_unix.cpp:94] [RestartWithFork] Starting wazuh agent in a new process.
[2025-01-22 03:27:34.180] [wazuh-agent] [info] [INFO] [process_options_unix.cpp:24] [StartAgent] Starting wazuh-agent
[2025-01-22 03:27:34.181] [wazuh-agent] [info] [INFO] [communicator.hpp:55] [Communicator] Using insecure connection.
[2025-01-22 03:27:34.188] [wazuh-agent] [info] [INFO] [communicator.cpp:28] [SendAuthenticationRequest] Successfully authenticated with the manager.
[2025-01-22 03:27:34.189] [wazuh-agent] [info] [INFO] [inventory.cpp:18] [Start] Inventory module started.
[2025-01-22 03:27:34.189] [wazuh-agent] [info] [INFO] [inventoryImp.cpp:1170] [SyncLoop] Module started.
[2025-01-22 03:27:34.189] [wazuh-agent] [info] [INFO] [inventoryImp.cpp:1153] [Scan] Starting evaluation.
[2025-01-22 03:27:34.190] [wazuh-agent] [info] [INFO] [logcollector.cpp:32] [Start] Logcollector module started.
[2025-01-22 03:27:34.191] [wazuh-agent] [info] [INFO] [file_reader.cpp:61] [AddLocalfiles] Reading log file: /var/log/auth.log
[2025-01-22 03:27:34.199] [wazuh-agent] [info] [INFO] [command_handler.hpp:146] [CleanUpInProgressCommands] Agent restarted successfully

Using Systemd to run the agent

root@ubuntu2204:/home/vagrant# journalctl -xu wazuh-agent -f
...
Jan 22 16:31:18 ubuntu2204.localdomain env[10942]: [2025-01-22 16:31:18.237] [wazuh-agent] [info] [INFO] [agent.cpp:170] [operator()] Restart: Initiating self-restart
Jan 22 16:31:18 ubuntu2204.localdomain env[10942]: [2025-01-22 16:31:18.239] [wazuh-agent] [info] [INFO] [restart_unix.cpp:36] [RestartWithSystemd] Systemctl restarting wazuh agent service.
Jan 22 16:31:18 ubuntu2204.localdomain env[10942]: [2025-01-22 16:31:18.243] [wazuh-agent] [info] [INFO] [inventory.cpp:62] [Stop] Inventory module stopping...
Jan 22 16:31:18 ubuntu2204.localdomain systemd[1]: Stopping Wazuh agent...
░░ Subject: A stop job for unit wazuh-agent.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A stop job for unit wazuh-agent.service has begun execution.
░░ 
░░ The job identifier is 32220.
Jan 22 16:31:18 ubuntu2204.localdomain env[10942]: [2025-01-22 16:31:18.247] [wazuh-agent] [info] [INFO] [command_handler.hpp:105] [CommandsProcessingTask] Done processing command: restart(restart)
Jan 22 16:31:18 ubuntu2204.localdomain env[10942]: [2025-01-22 16:31:18.352] [wazuh-agent] [info] [INFO] [logcollector.cpp:96] [Stop] Logcollector module stopped.
Jan 22 16:31:18 ubuntu2204.localdomain env[10942]: [2025-01-22 16:31:18.353] [wazuh-agent] [info] [INFO] [inventoryImp.cpp:1165] [Scan] Evaluation finished.
Jan 22 16:31:18 ubuntu2204.localdomain env[10942]: [2025-01-22 16:31:18.361] [wazuh-agent] [info] [INFO] [inventory.cpp:37] [Start] Inventory module stopped.
Jan 22 16:31:18 ubuntu2204.localdomain systemd[1]: wazuh-agent.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit wazuh-agent.service has successfully entered the 'dead' state.
Jan 22 16:31:18 ubuntu2204.localdomain systemd[1]: Stopped Wazuh agent.
░░ Subject: A stop job for unit wazuh-agent.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A stop job for unit wazuh-agent.service has finished.
░░ 
░░ The job identifier is 32220 and the job result is done.
Jan 22 16:31:18 ubuntu2204.localdomain systemd[1]: Started Wazuh agent.
░░ Subject: A start job for unit wazuh-agent.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit wazuh-agent.service has finished successfully.
░░ 
░░ The job identifier is 32220.
Jan 22 16:31:18 ubuntu2204.localdomain env[10964]: [2025-01-22 16:31:18.373] [wazuh-agent] [info] [INFO] [process_options_unix.cpp:24] [StartAgent] Starting wazuh-agent
Jan 22 16:31:18 ubuntu2204.localdomain env[10964]: [2025-01-22 16:31:18.374] [wazuh-agent] [info] [INFO] [communicator.hpp:55] [Communicator] Using insecure connection.
Jan 22 16:31:18 ubuntu2204.localdomain env[10964]: [2025-01-22 16:31:18.378] [wazuh-agent] [info] [INFO] [communicator.cpp:28] [SendAuthenticationRequest] Successfully authenticated with the manager.
Jan 22 16:31:18 ubuntu2204.localdomain env[10964]: [2025-01-22 16:31:18.379] [wazuh-agent] [info] [INFO] [inventory.cpp:18] [Start] Inventory module started.
Jan 22 16:31:18 ubuntu2204.localdomain env[10964]: [2025-01-22 16:31:18.379] [wazuh-agent] [info] [INFO] [logcollector.cpp:32] [Start] Logcollector module started.
Jan 22 16:31:18 ubuntu2204.localdomain env[10964]: [2025-01-22 16:31:18.379] [wazuh-agent] [info] [INFO] [file_reader.cpp:61] [AddLocalfiles] Reading log file: /var/log/auth.log
Jan 22 16:31:18 ubuntu2204.localdomain env[10964]: [2025-01-22 16:31:18.380] [wazuh-agent] [info] [INFO] [inventoryImp.cpp:1170] [SyncLoop] Module started.
Jan 22 16:31:18 ubuntu2204.localdomain env[10964]: [2025-01-22 16:31:18.380] [wazuh-agent] [info] [INFO] [inventoryImp.cpp:1153] [Scan] Starting evaluation.
Jan 22 16:31:18 ubuntu2204.localdomain env[10964]: [2025-01-22 16:31:18.390] [wazuh-agent] [info] [INFO] [command_handler.hpp:146] [CleanUpInProgressCommands] Agent restarted successfully
...

@lchico lchico linked an issue Dec 4, 2024 that may be closed by this pull request
5 tasks
@lchico lchico force-pushed the enhancement/77-add-support-agent-self-restarting branch from e65a3ff to d0089ef Compare December 4, 2024 14:20
@lchico lchico linked an issue Dec 4, 2024 that may be closed by this pull request
@lchico lchico force-pushed the enhancement/77-add-support-agent-self-restarting branch 4 times, most recently from fdda710 to c009b4a Compare December 10, 2024 01:33
@lchico lchico force-pushed the enhancement/77-add-support-agent-self-restarting branch 9 times, most recently from 6124ac7 to 3a71fae Compare December 20, 2024 00:09
src/agent/CMakeLists.txt Outdated Show resolved Hide resolved
@lchico lchico force-pushed the enhancement/77-add-support-agent-self-restarting branch 9 times, most recently from 6a2deb8 to 3ca559f Compare December 24, 2024 03:09
@lchico lchico force-pushed the enhancement/77-add-support-agent-self-restarting branch 4 times, most recently from ab9cf22 to f40fc98 Compare January 14, 2025 02:53
@lchico lchico force-pushed the enhancement/77-add-support-agent-self-restarting branch 11 times, most recently from 74ecea0 to e3a107d Compare January 27, 2025 13:36
@lchico lchico force-pushed the enhancement/77-add-support-agent-self-restarting branch from e3a107d to f15e38e Compare January 27, 2025 16:09
Copy link
Member

@jr0me jr0me left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a few comments, mostly on formatting and style, but already LGTM

packages/debs/SPECS/wazuh-agent/debian/postrm Outdated Show resolved Hide resolved
docs/ref/commands.md Show resolved Hide resolved
src/agent/restart_handler/CMakeLists.txt Show resolved Hide resolved
src/agent/restart_handler/CMakeLists.txt Outdated Show resolved Hide resolved
src/agent/restart_handler/include/restart_handler.hpp Outdated Show resolved Hide resolved
src/agent/restart_handler/include/restart_handler.hpp Outdated Show resolved Hide resolved
src/agent/restart_handler/src/restart_handler_unix.cpp Outdated Show resolved Hide resolved
feat: Address comments and improve the code
@lchico lchico force-pushed the enhancement/77-add-support-agent-self-restarting branch from f15e38e to 762a256 Compare January 28, 2025 02:52
@vikman90 vikman90 changed the title Add support for agent self-restarting: Development Phase Add support for agent self-restarting on Linux Jan 28, 2025
Copy link
Member

@vikman90 vikman90 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now, we've got just one process:

SessionLeader───Relay(130993)───bash───wazuh-agent───6*[{wazuh-agent}]

Thanks @lchico!

@MarcelKemp MarcelKemp merged commit b8c1d36 into master Jan 28, 2025
5 checks passed
@MarcelKemp MarcelKemp deleted the enhancement/77-add-support-agent-self-restarting branch January 28, 2025 17:30
@vikman90 vikman90 linked an issue Jan 29, 2025 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for agent self-restarting for Linux
6 participants