Linux (DEB) Package Creation

[vikman90]

Parent Issue:

• Create Installation Packages for Wazuh Agent #128

Description

This issue focuses on creating a DEB package for Debian-based distributions such as Debian and Ubuntu. The package should manage all necessary dependencies and include pre/post-installation scripts to ensure that the Wazuh agent service is automatically enabled and running after installation.

Functional Requirements

• Create a DEB package compatible with Debian-based distributions like Ubuntu and Debian.
• Ensure all dependencies are correctly handled during the installation process.
• • The Wazuh agent service must be disabled by default.
• Test the installation and uninstallation workflows to ensure compatibility with Debian-based systems.

Acceptance Criteria

• The DEB package is built successfully and integrates with native package managers.
• Installation, uninstallation, and upgrade processes are tested and verified on Debian-based distributions.

[lchico]

Update

I've begun researching this issue and have started working on a solution.

2024-09-13

I encountered some issues when generating packages using the SPECS files from the repository. After some investigation, I discovered the cause: the files were uploaded with Windows-style line endings (CRLF) instead of Unix-style (LF).
This caused the following error during the build process:

dpkg-buildpackage -rsudo -D -us -uc -b -nc
dpkg-buildpackage: warning: using a gain-root-command while being root
dpkg-buildpackage: source package wazuh-agent
dpkg-buildpackage: source version 4.9.0-0
dpkg-buildpackage: source changed by Wazuh, Inc <[email protected]>
 dpkg-source --before-build wazuh-agent-5.0.0
dpkg-buildpackage: host architecture amd64
'pkg-source: error: invalid Format field `3.0 (quilt)
dpkg-buildpackage: error: dpkg-source --before-build wazuh-agent-5.0.0 gave error exit status 25
debuild: fatal error at line 1357:
dpkg-buildpackage -rsudo -D -us -uc -b -nc failed
root@4be8bb734266:/# 

2024-09-16

To avoid potential issues with older OS versions (meaning that when we generate Debian packages using a newer version of Debian, the packages might not work on older Debian versions), I am trying to use the same OS we currently use for generating Debian packages:

lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 7.11 (wheezy)
Release:	7.11
Codename:	wheezy

• During the build process, I had to update:
  • CMake to version 3.30.3, as it's required by the code (the code requires 3.22, but I’m trying with this version). I downloaded and installed it locally as a test, but we need to push it to packages.wazuh.com, just like the previous version, 3.18.3.
  • While building the package, I encountered an issue with vcpkg and its dependencies:
    • The curl version in the current image is curl 7.26.0, and with this version, I encountered the following error:

/bootstrap-vcpkg.sh
Downloading vcpkg-glibc...
curl: option --tlsv1.2: is unknown
curl: try 'curl --help' or 'curl --manual' for more information

• I attempted to update curl using the apt-get package manager, but since this OS version is no longer supported, it wasn’t possible. So, I installed curl from source with at least the required version 7.35, which supports --tlsv1.2. However, I ran into the following error:

 ./bootstrap-vcpkg.sh 
Downloading vcpkg-glibc...
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

• I tried using --insecure, but it didn’t resolve the issue.
• Then, I attempted to use a newer version of curl (4.8.10), but I couldn’t build it from the source due to unresolved dependencies:

Build curl 8-10.0

curl-8.10.0# ./configure --with-gnutls --with-secure-transport --with-openssl 
checking whether to enable maintainer-specific portions of Makefiles... no
checking whether make supports nested variables... yes
checking whether to enable debug build options... no
checking whether to enable compiler optimizer... (assumed) yes
checking whether to enable strict compiler warnings... no
checking whether to enable compiler warnings as errors... no
checking whether to enable curl debug memory tracking... no
checking whether to enable hiding of library internal symbols... yes
checking whether to enable c-ares for DNS lookups... no
checking whether to disable dependency on -lrt... (assumed no)
checking whether to enable HTTPSRR support... no
checking whether to enable ECH support... no
checking for path separator... :
...
checking default CA cert bundle/path... configure: want unset ca /etc/ssl/certs/ca-certificates.crt
/etc/ssl/certs/ca-certificates.crt
/etc/ssl/certs (capath)
checking whether to use built-in CA store of SSL library... no
checking CA cert bundle path to embed... no
checking for pkg-config... (cached) /usr/bin/pkg-config
checking for libpsl options with pkg-config... no
checking for psl_builtin in -lpsl... no
configure: error: libpsl libs and/or directories were not found where specified!

2024-09-17

Based on tier-1, I am trying to generate Debian packages using Debian 10. With this change, I got vcpkg to download the dependencies. However, when the build process started, I encountered problems with certain dependencies and g++ versions. After trying g++-10, it didn't work, so I had to build g++-14 from the source. It looks like I am close to completing the build process. I just have this pending issue to fix:

[ 96%] Building CXX object agent/command_store/CMakeFiles/CommandStore.dir/src/command_store.cpp.o
In file included from /build_wazuh/agent/wazuh-agent-5.0.0/src/agent/command_store/include/command_store.hpp:4,
                 from /build_wazuh/agent/wazuh-agent-5.0.0/src/agent/command_store/src/command_store.cpp:1:
In copy constructor 'constexpr sqlite_manager::Column::Column(const sqlite_manager::Column&)',
    inlined from 'bool command_store::CommandStore::UpdateCommand(const command_store::Command&)' at /build_wazuh/agent/wazuh-agent-5.0.0/src/agent/command_store/src/command_store.cpp:203:5:
/build_wazuh/agent/wazuh-agent-5.0.0/src/agent/sqlite_manager/include/sqlite_manager.hpp:25:11: error: '*(__vector(2) unsigned char*)(&condition + offsetof(sqlite_manager::Column, sqlite_manager::Column::m_notNull))' may be used uninitialized [-Werror=maybe-uninitialized]
   25 |     class Column
      |           ^~~~~~
/build_wazuh/agent/wazuh-agent-5.0.0/src/agent/command_store/src/command_store.cpp: In member function 'bool command_store::CommandStore::UpdateCommand(const command_store::Command&)':
/build_wazuh/agent/wazuh-agent-5.0.0/src/agent/command_store/src/command_store.cpp:192:32: note: 'condition' declared here
  192 |         sqlite_manager::Column condition("id", sqlite_manager::ColumnType::INTEGER, std::to_string(cmd.m_id));
      |                                ^~~~~~~~~
In copy constructor 'constexpr sqlite_manager::Column::Column(const sqlite_manager::Column&)',
    inlined from 'bool command_store::CommandStore::UpdateCommand(const command_store::Command&)' at /build_wazuh/agent/wazuh-agent-5.0.0/src/agent/command_store/src/command_store.cpp:203:5:
/build_wazuh/agent/wazuh-agent-5.0.0/src/agent/sqlite_manager/include/sqlite_manager.hpp:25:11: error: 'condition.sqlite_manager::Column::m_primaryKey' may be used uninitialized [-Werror=maybe-uninitialized]
   25 |     class Column
      |           ^~~~~~
/build_wazuh/agent/wazuh-agent-5.0.0/src/agent/command_store/src/command_store.cpp: In member function 'bool command_store::CommandStore::UpdateCommand(const command_store::Command&)':
/build_wazuh/agent/wazuh-agent-5.0.0/src/agent/command_store/src/command_store.cpp:192:32: note: 'condition' declared here
  192 |         sqlite_manager::Column condition("id", sqlite_manager::ColumnType::INTEGER, std::to_string(cmd.m_id));
      |                                ^~~~~~~~~
cc1plus: all warnings being treated as errors
make[2]: *** [agent/command_store/CMakeFiles/CommandStore.dir/build.make:76: agent/command_store/CMakeFiles/CommandStore.dir/src/command_store.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:396: agent/command_store/CMakeFiles/CommandStore.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

I haven't had time to investigate this further.

2024-09-18

• Finally, I created a Docker image with all the dependencies required for generating the deb packages.
• I managed to generate the first deb package, but there’s still a lot of work to be done in defining the new structure.
  • Installation path: /opt/wazuh-agent — based on this documentation and considering that Google also installs their packages there, I believe this is the best option.
  • Configuration files, like wazuh.conf, should be placed in /etc/opt/wazuh-agent/, according to this document (page 15, chapter 3.7.4).
  • I have started working on the rules and Debian scripts.

[lchico]

Update

• Researched how to generate packages on Debian and identified best practices.
  I have detected some inconsistencies in my Dockerfile that are preventing the successful build of GCC 14. I am comparing it with a working container to find the differences.
• Encountered a dependency issue where the first round of package generation works on Ubuntu but fails on Debian:

/opt/wazuh-agent/bin/wazuh-agent: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /opt/wazuh-agent/bin/wazuh-agent)
/opt/wazuh-agent/bin/wazuh-agent: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.31' not found (required by /opt/wazuh-agent/bin/wazuh-agent)
/opt/wazuh-agent/bin/wazuh-agent: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.15' not found (required by /opt/wazuh-agent/bin/wazuh-agent)

I am not prioritizing this issue for now as there are other things that need to be defined first.

[vikman90]

Dear @lchico, let me write here a brief of our discussion:

1. Service Definition:
   The man 7 file-hierarchy document clearly indicates that the /usr directory is reserved for vendor-provided packages. Therefore, it's more appropriate for us to maintain the service file under /etc/systemd/system/.

2. User and Group:
   We haven't defined a change in user/group yet, but it's likely we will do so via Systemd in the future. For now, we should continue using wazuh for both the user and group. This may change later, but we can address it at that point.

3. Configuration Files:
   Following the vendor recommendations is the best approach here. I propose we use /etc/wazuh-agent/wazuh-agent.yml for the main configuration file. This aligns with the structure of other files (like those for OpenSearch), and allows us to manage multiple files if needed. Additionally, this would allow for both a manager and an agent to coexist on the same system.

4. Generated Files Location:
   For generated files, it's more appropriate to place them in directories like /var/local/wazuh-agent or /var/opt/wazuh-agent, rather than /opt, which is reserved for static files. We'll need to follow system recommendations here as well.

5. Logs:
   Currently, agent logs will be sent to stdout and captured by Systemd via Journald. In the future, we may define an option to write logs to files, in which case using a directory like /var/log/wazuh-agent/*.log might be more suitable. This isn't yet fully defined, though.

6. Dependencies:
   This requires further discussion. Since we're moving to a single binary, it would be best to statically link as many libraries as possible. If certain libraries are absolutely necessary, we may need to ship them with the agent (similar to how we handle /var/ossec/lib). The most problematic libraries here may be libstdc++ and libgcc_s.

7. Removing Package:
   I agree with your approach: remove /etc/opt/wazuh-agent but keep /etc/wazuh.conf. For backups, there's nothing crucial that we need to preserve for now. However, it may make sense to remove the wazuh user as well, since it's only required at runtime.

8. Purge:
   The purge process looks good. Removing /etc/opt/wazuh-agent, /etc/wazuh.conf, and the wazuh user/group should suffice.

---

In conclusion, for all of these items, we need to ensure we follow the system standards for each platform (Debian, RedHat, Windows, macOS), reference the vendor's documentation, and review our requirements. We should adhere to the default standards unless we have a clear reason to deviate, which should be documented thoroughly.

References

• Filesystem Hierarchy Standard
• systemd.service — Service unit configuration

[lchico]

First, thank you @vikman90. That sounds very clear to me.

Update

• Fixed the Docker issue, and I can now build a deb package.
• Updated Debian scripts and successfully generated the first Debian package with the binary.

[mjcr99]

Update

(24/09/2024) Updating Dockerfile to enable compiling with the new repository, covering different dependencies. The GCC version has been downgraded to 13.2.0 since using 14.2 was causing procps port-overlay not to compile.
(25/09/2024) Added 33047de to make the procps port-overlay finally compilable. And added more dependencies to the Dockerfile.
(26/09/2024) Finally committed c1ff35c to upload Dockerfile final version which builds the agent with no problem.

[jotacarma90]

Update 26/09/2024

• Reading and adaptation to the whole new project.
• I have been preparing environment to have the new docker image.
• Changes in the dockerfile and in the generation script so that the necessary files are used in a shared volume, so they don't need to be inside the image.
• Package created correctly.
• Testing and checking that everything is correct.

Update 27/09/2024

• The package still has many things to be developed.
• Analyzing all the pre and post scripts of the package.
• Created a separate PR for the image volume issue: Remove entrypoints from Dockerfiles and create volumes for scripts #182
• Modifying the default installation path /var/ossec of all the sites where it appears.
• Trying to analyze what is needed to create an install script.

Update 30/09/2024

• I have created a basic DEB package to do the necessary tests.
• Researching all the necessary standards with paths and complete package structure.
• Meeting with the team discussing the next steps to take regarding the generation.

[jotacarma90]

Current DEB package

• wazuh-agent_5.0.0-3_amd64_c1ff35cdb.zip

• This package contains the current minimum files. It is not functional with respect to the Wazuh agent, it is only a sample that makes it possible to install and uninstall the wazuh-agent package.

• Install output:

root@ubuntu24:/home/vagrant/wazuh-agent# dpkg -i wazuh-agent_5.0.0-3_amd64_c1ff35cdb.deb
Selecting previously unselected package wazuh-agent.
(Reading database ... 101846 files and directories currently installed.)
Preparing to unpack wazuh-agent_5.0.0-3_amd64_c1ff35cdb.deb ...
+ export INSTALLATION_WAZUH_DIR=/opt/wazuh-agent
+ WAZUH_TMP_DIR=/opt/wazuh-agent/tmp
+ [ ! -d /opt/wazuh-agent/tmp ]
+ mkdir -p /opt/wazuh-agent/tmp
+ [ install = upgrade ]
+ exit 0
Unpacking wazuh-agent (5.0.0-3) ...
Setting up wazuh-agent (5.0.0-3) ...
+ export INSTALLATION_WAZUH_DIR=/opt/wazuh-agent
+ lsb_release -si
+ OS=Ubuntu
+ lsb_release -sr
+ VER=24.04
+ USER=wazuh
+ GROUP=wazuh
+ OSMYSHELL=/sbin/nologin
+ [ -d /run/systemd/system ]
+ rm -f /etc/init.d/wazuh-agent
+ [ ! -f /sbin/nologin ]
+ getent group wazuh
+ addgroup --system wazuh
+ getent passwd wazuh
+ adduser --system --home /opt/wazuh-agent --shell /sbin/nologin --ingroup wazuh wazuh
+ [ -f /etc/systemd/system/wazuh-agent.service ]
+ rm -f /etc/systemd/system/wazuh-agent.service
+ command -v systemctl
+ systemctl
+ systemctl daemon-reload
+ exit 0

• Remove output:

root@ubuntu24:/home/vagrant/wazuh-agent# apt remove wazuh-agent
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  wazuh-agent
0 upgraded, 0 newly installed, 1 to remove and 4 not upgraded.
After this operation, 2,929 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 101856 files and directories currently installed.)
Removing wazuh-agent (5.0.0-3) ...
+ export INSTALLATION_WAZUH_DIR=/opt/wazuh-agent
+ command -v systemctl
+ systemctl
+ systemctl is-active --quiet wazuh-agent
+ command -v service
+ service wazuh-agent status
+ grep running
+ exit 0
+ export INSTALLATION_WAZUH_DIR=/opt/wazuh-agent
+ WAZUH_TMP_DIR=/opt/wazuh-agent/tmp
+ [ -d /opt/wazuh-agent/tmp ]
+ rm -rf /opt/wazuh-agent/tmp
+ [ remove = remove ]
+ rm -rf /opt/wazuh-agent
+ getent passwd wazuh
+ deluser wazuh
+ exit 0
Scanning processes...
Scanning linux images...

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

[MarcelKemp]

Update

Test branch: test/deb-package-generation

In the branch, I have rebased the rest of the branches and added some steps to the Dockerfile to try to generate a 5.0 package.
It still needs modifications to add the service functionality.

(10/21) - Compilation problems due to some vcpkg dependencies, which are not needed for Debian (librpm), but cannot be filtered out in vcpkg in a simple way.
Also adapting the code to install, get and run services, and uninstall correctly.

(10/22) - Fixed vcpkg dependencies issue.
Found new problem with wazuh-agent compilation, related to flags and logcollector development.
Trying to generate a complete package.

(10/23) - Package generated successfully.

• Update packages to be able to create Linux (DEB) packages. #159