Implement commands' expansion logic #88

AlexRuiz7 · 2024-10-02T15:30:53Z

Description

The Command Manager plugin processes the commands received from its Rest API before persisting the information into the .commands index. The processing of these commands consists of:

Expanding the commands whose type is groups.
To expand this type of commands which are targeted to a group of agents, the Command Manager plugin needs to generate a processed command for each of the agents that belong to that group. For example, if the group has N agents, N commands need to be generated, inheriting the command details from the raw command.
Transform the received data into the commands data model.

Pre-requisites

That .agents index must exist, so the Command Manager can query their info (groups).
The commands index template exists.*

Functional requirements

All the commands are transformed from the API Model to the Index model.
The commands whose type is groups are expanded to as many agents as there are in the target group (target field in the API model).
The processed commands are indexed into the .commands index.

---
title: Command Manager - Expansion of commands targeted to a group of agents
---
stateDiagram-v2

    [*] --> RestAPI: A list of commands
    RestAPI --> Command: Validation to model
    Command --> Expansion: if target.type == group
    Expansion --> Order: From Command to Order
    Order --> Index: N orders
    Index --> [*]

---
title: Command Manager - Expansion of commands targeted to a group of agents
---

flowchart TD

subgraph Indexer["Indexer cluster"]

    subgraph Data_streams["Data stream"]
        commands_stream["Commands stream"]
        agents_list["Agents"]
    end

    subgraph Plugins["Modules"]
        content_manager["Content manager"]
        command_manager["Command manager"]
    end

end

server["Management API"] -- 1- /commands --> command_manager
content_manager -- 1- /commands --> command_manager


command_manager <-. 2- reads agents (n) in group .-> agents_list
command_manager -- 3- index orders (n) --> commands_stream


style Data_streams fill:#abc2eb
style Plugins fill:#abc2eb

Suitable to change. The command's data model must fit the data after the expansion, not before, as it's made currently.

References: https://github.com/wazuh/wazuh-indexer/blob/master/ecs/docs/commands.md

The text was updated successfully, but these errors were encountered:

QU3B1M · 2025-01-20T13:22:48Z

Implemented a search term query to the .agent index using the groups field to filter the results, this solution does not require to be run inside of a thread, as difference from jobscheduler/SearchThread, but it is a similar approach.

For now as it is a work in progress, the code is stored on the handlePost of the class RestPostCommandAction, will improve its structure and move it to its own class once its functioning correctly.

  for (Command command : commands) {
      log.info("Command {}", command);
      if (Objects.equals(command.getTarget().getType(), "group")){
          log.info("[GROUP] Target id {}", command.getTarget().getId());

          // Build the search query
          SearchRequest searchRequest = new SearchRequest(".agents");
          SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
          BoolQueryBuilder boolQuery = QueryBuilders.boolQuery()
                  .must(QueryBuilders.termQuery("agent.groups", command.getTarget().getId()));
          searchSourceBuilder.query(boolQuery);
          searchRequest.source(searchSourceBuilder);

          // Create the listener for the async search request
          client.search(searchRequest,  new ActionListener<SearchResponse>() {
              @Override
              public void onResponse(SearchResponse searchResponse) {
                  // Process the search response
                  SearchHits hits = searchResponse.getHits();
                  for (SearchHit hit : hits) {
                      log.info("Agent found: {}", hit.getSourceAsString());
                  }
              }

              @Override
              public void onFailure(Exception e) {
                  log.error("Search failed", e);
              }
          });
      }

Posted a group command through the API and the search is performed correctly

POST request

request

{
  "commands": [
    {
      "source": "Engine",
      "user": "user53",
      "target": {
        "id": "group000",
        "type": "group"
      },
      "action": {
        "name": "restart",
        "args": {
          "estd": true,
          "in9": 95468839,
          "arg1": "/path/to/executable/arg6"
        },
        "version": "v4"
      },
      "timeout": 30
    }
  ]
}

response

{
  "_index": ".commands",
  "_documents": [
    {
      "_id": "uqTVg5QB6vYXRzTdotSC"
    }
  ],
  "result": "OK"
}

 Agent found: {
  "agent": {
    "id": "agent27",
    "name": "Agent46",
    "type": "linux",
    "version": "v5-stable",
    "status": "inactive",
    "last_login": "2025-01-20T09:02:11.3NZ",
    "groups": ["group000","group000"],
    "key": "key635",
    "host": {
    "architecture": "arm64",
    "boot": {"id": "boot597"},
    "cpu": {"usage": 0},
    "disk": {"read": {"bytes": 23643}, "write": {"bytes": 30843}},
    "domain": "domain786",
    "geo": {
      "city_name": "Berlin",
      "continent_code": "NA",
      "continent_name": "North America",
      "country_iso_code": "DE",
      "country_name": "United States",
      "location": {"lat": -89.999860, "lon": -179.999840},
      "name": "geo28",
      "postal_code": "25046",
      "region_iso_code": "region398",
      "region_name": "Region 246",
      "timezone": "PST"
    },
    "hostname": "host4529",
    "id": "hostid6587",
    "ip": "27.192.173.128",
    "mac": "fa:7f:75:e7:bb:0c",
    "name": "hostname8095",
    "network": {"egress": {"bytes": 27795, "packets": 6429}, "ingress": {"bytes": 26312, "packets": 27189}},
    "os": {"family": "macos", "full": "macos 5.40", "kernel": "kernel799", "name": "macos", "platform": "windows", "type": "macos", "version": "5.40"},
    "pid_ns_ino": "1014157",
    "risk": {"calculated_level": "high", "calculated_score": 0, "calculated_score_norm": 0, "static_level": "high", "static_score": 0, "static_score_norm": 0},
    "uptime": 15332
  }
  }
}

QU3B1M · 2025-01-22T14:25:30Z

Implemented command expansion logic, all the search-related code was moved to the Search util class (utils/Search.java)

Check command expansion using a group with two different agents assigned

 % curl http://127.0.0.1:9200/.agents/_search | grep "group000"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15337  100 15337    0     0  1799k      0 --:--:-- --:--:-- --:--:-- 1872k
    "groups": ["group000","group001","group003"],
    "groups": ["group000","group002","group002"],

POST request

request

{
  "commands": [
    {
      "source": "Engine",
      "user": "user53",
      "target": {
        "id": "group000",
        "type": "group"
      },
      "action": {
        "name": "restart",
        "args": {
          "arg1": "/path/to/executable/arg6"
        },
        "version": "v4"
      },
      "timeout": 30
    }
  ]
}

The request has generated two commands and the response shows the corresponding IDs

{
  "_index": ".commands",
  "_orders": [
    {
      "_id": "WS5fjpQBDZQg8-hA8AYd"
    },
    {
      "_id": "Wi5fjpQBDZQg8-hA8AYe"
    }
  ],
  "result": "OK"
}

[2025-01-22T11:17:19,646][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [WS5fjpQBDZQg8-hA8AYd] to the bulk request
[2025-01-22T11:17:19,646][INFO ][c.w.c.i.CommandIndex     ] [integTest-0] Adding command with id [Wi5fjpQBDZQg8-hA8AYe] to the bulk request

And the commands are correctly indexed

% curl http://127.0.0.1:9200/.commands/_search                
{"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":2,"relation":"eq"},"max_score":1.0,"hits":[{"_index":".commands","_id":"WS5fjpQBDZQg8-hA8AYd","_score":1.0,"_source":{"agent":{"groups":["group000","group001","group003"]},"@timestamp":"2025-01-22T14:17:19Z","delivery_timestamp":"2025-01-22T14:17:49Z","command":{"action":{"args":{"arg1":"/path/to/executable/arg6"},"name":"restart","version":"v4"},"source":"Engine","user":"user53","order_id":"WC5fjpQBDZQg8-hA8AYP","request_id":"Vy5fjpQBDZQg8-hA8AYP","timeout":30,"target":{"id":"group000","type":"group"},"status":"failure"}}},{"_index":".commands","_id":"Wi5fjpQBDZQg8-hA8AYe","_score":1.0,"_source":{"agent":{"groups":["group000","group002","group002"]},"@timestamp":"2025-01-22T14:17:19Z","delivery_timestamp":"2025-01-22T14:17:49Z","command":{"action":{"args":{"arg1":"/path/to/executable/arg6"},"name":"restart","version":"v4"},"source":"Engine","user":"user53","order_id":"WC5fjpQBDZQg8-hA8AYP","request_id":"Vy5fjpQBDZQg8-hA8AYP","timeout":30,"target":{"id":"group000","type":"group"},"status":"failure"}}}]}}

Test with a group assigned to the same agent more than once

Using the group 001

% curl http://127.0.0.1:9200/.agents/_search | grep "group001"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15451  100 15451    0     0  2741k      0 --:--:-- --:--:-- --:--:-- 3017k
    "groups": ["group001"],
    "groups": ["group004","group004","group001","group004"],
    "groups": ["group001","group005","group000","group001"],
    "groups": ["group004","group000","group003","group001","group002"],
    "groups": ["group004","group000","group001"],

Request

{
  "commands": [
    {
      "source": "Engine",
      "user": "user53",
      "target": {
        "id": "group001",
        "type": "group"
      },
      "action": {
        "name": "restart",
        "args": {
          "arg1": "/path/to/executable/arg6"
        },
        "version": "v4"
      },
      "timeout": 30
    }
  ]
}

Response

{
  "_index": ".commands",
  "_orders": [
    {
      "_id": "wP9ojpQBpOpp5jjTDc0s"
    },
    {
      "_id": "wf9ojpQBpOpp5jjTDc0t"
    },
    {
      "_id": "wv9ojpQBpOpp5jjTDc0t"
    },
    {
      "_id": "w_9ojpQBpOpp5jjTDc0t"
    },
    {
      "_id": "xP9ojpQBpOpp5jjTDc0t"
    }
  ],
  "result": "OK"
}

Search on the index

% curl http://127.0.0.1:9200/.commands/_search       
{
    "took": 2,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 5,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": ".commands",
                "_id": "wP9ojpQBpOpp5jjTDc0s",
                "_score": 1.0,
                "_source": {
                    "agent": {
                        "groups": [
                            "group001"
                        ]
                    },
                    "@timestamp": "2025-01-22T14:26:11Z",
                    "delivery_timestamp": "2025-01-22T14:26:41Z",
                    "command": {
                        "action": {
                            "args": {
                                "arg1": "/path/to/executable/arg6"
                            },
                            "name": "restart",
                            "version": "v4"
                        },
                        "source": "Engine",
                        "user": "user53",
                        "order_id": "v_9ojpQBpOpp5jjTDc0h",
                        "request_id": "vv9ojpQBpOpp5jjTDc0h",
                        "timeout": 30,
                        "target": {
                            "id": "group001",
                            "type": "group"
                        },
                        "status": "failure"
                    }
                }
            },
            {
                "_index": ".commands",
                "_id": "wf9ojpQBpOpp5jjTDc0t",
                "_score": 1.0,
                "_source": {
                    "agent": {
                        "groups": [
                            "group001",
                            "group005",
                            "group000",
                            "group001"
                        ]
                    },
                    "@timestamp": "2025-01-22T14:26:11Z",
                    "delivery_timestamp": "2025-01-22T14:26:41Z",
                    "command": {
                        "action": {
                            "args": {
                                "arg1": "/path/to/executable/arg6"
                            },
                            "name": "restart",
                            "version": "v4"
                        },
                        "source": "Engine",
                        "user": "user53",
                        "order_id": "v_9ojpQBpOpp5jjTDc0h",
                        "request_id": "vv9ojpQBpOpp5jjTDc0h",
                        "timeout": 30,
                        "target": {
                            "id": "group001",
                            "type": "group"
                        },
                        "status": "failure"
                    }
                }
            },
            {
                "_index": ".commands",
                "_id": "wv9ojpQBpOpp5jjTDc0t",
                "_score": 1.0,
                "_source": {
                    "agent": {
                        "groups": [
                            "group004",
                            "group000",
                            "group001"
                        ]
                    },
                    "@timestamp": "2025-01-22T14:26:11Z",
                    "delivery_timestamp": "2025-01-22T14:26:41Z",
                    "command": {
                        "action": {
                            "args": {
                                "arg1": "/path/to/executable/arg6"
                            },
                            "name": "restart",
                            "version": "v4"
                        },
                        "source": "Engine",
                        "user": "user53",
                        "order_id": "v_9ojpQBpOpp5jjTDc0h",
                        "request_id": "vv9ojpQBpOpp5jjTDc0h",
                        "timeout": 30,
                        "target": {
                            "id": "group001",
                            "type": "group"
                        },
                        "status": "failure"
                    }
                }
            },
            {
                "_index": ".commands",
                "_id": "w_9ojpQBpOpp5jjTDc0t",
                "_score": 1.0,
                "_source": {
                    "agent": {
                        "groups": [
                            "group004",
                            "group004",
                            "group001",
                            "group004"
                        ]
                    },
                    "@timestamp": "2025-01-22T14:26:11Z",
                    "delivery_timestamp": "2025-01-22T14:26:41Z",
                    "command": {
                        "action": {
                            "args": {
                                "arg1": "/path/to/executable/arg6"
                            },
                            "name": "restart",
                            "version": "v4"
                        },
                        "source": "Engine",
                        "user": "user53",
                        "order_id": "v_9ojpQBpOpp5jjTDc0h",
                        "request_id": "vv9ojpQBpOpp5jjTDc0h",
                        "timeout": 30,
                        "target": {
                            "id": "group001",
                            "type": "group"
                        },
                        "status": "failure"
                    }
                }
            },
            {
                "_index": ".commands",
                "_id": "xP9ojpQBpOpp5jjTDc0t",
                "_score": 1.0,
                "_source": {
                    "agent": {
                        "groups": [
                            "group004",
                            "group000",
                            "group003",
                            "group001",
                            "group002"
                        ]
                    },
                    "@timestamp": "2025-01-22T14:26:11Z",
                    "delivery_timestamp": "2025-01-22T14:26:41Z",
                    "command": {
                        "action": {
                            "args": {
                                "arg1": "/path/to/executable/arg6"
                            },
                            "name": "restart",
                            "version": "v4"
                        },
                        "source": "Engine",
                        "user": "user53",
                        "order_id": "v_9ojpQBpOpp5jjTDc0h",
                        "request_id": "vv9ojpQBpOpp5jjTDc0h",
                        "timeout": 30,
                        "target": {
                            "id": "group001",
                            "type": "group"
                        },
                        "status": "failure"
                    }
                }
            }
        ]
    }
}

QU3B1M · 2025-01-23T10:26:57Z

Found a Bug where in a list of commands, once a command has a valid agent or group as target, the rest of the commands, even if the target is invalid, receives the same search result than the previous command, indexing them repeatedly

[2025-01-23T07:22:01,303][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Received POST /_plugins/_command_manager/commands request id [16] from host [127.0.0.1:9200]
[2025-01-23T07:22:01,307][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Searching for agents using field agent.groups with value group001
[2025-01-23T07:22:01,319][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Search retrieved 6 agents.
[2025-01-23T07:22:01,322][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Searching for agents using field agent.id with value target4
[2025-01-23T07:22:01,324][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Search retrieved 6 agents.
[2025-01-23T07:22:01,324][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Searching for agents using field agent.id with value target4
[2025-01-23T07:22:01,326][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Search retrieved 6 agents.
[2025-01-23T07:22:01,326][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Searching for agents using field agent.id with value target4
[2025-01-23T07:22:01,328][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Search retrieved 6 agents.
[2025-01-23T07:22:01,328][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Searching for agents using field agent.id with value target4
[2025-01-23T07:22:01,329][INFO ][c.w.c.r.RestPostCommandAction] [integTest-0] Search retrieved 6 agents.

Only the first command has a valid value, there is no agent with ID target4 in my system, but the search returns the values anyway

Post response

{
  "_index": ".commands",
  "_orders": [
    {
      "_id": "7EyukpQB8BzCwOA63mto"
    },
    {
      "_id": "7UyukpQB8BzCwOA63mtq"
    },
    {
      "_id": "7kyukpQB8BzCwOA63mtq"
    },
    {
      "_id": "70yukpQB8BzCwOA63mtq"
    },
    {
      "_id": "8EyukpQB8BzCwOA63mtq"
    },
    {
      "_id": "8UyukpQB8BzCwOA63mtq"
    },
    {
      "_id": "8kyukpQB8BzCwOA63mts"
    },
    {
      "_id": "80yukpQB8BzCwOA63mts"
    },
    {
      "_id": "9EyukpQB8BzCwOA63mts"
    },
    {
      "_id": "9UyukpQB8BzCwOA63mts"
    },
    {
      "_id": "9kyukpQB8BzCwOA63mts"
    },
    {
      "_id": "90yukpQB8BzCwOA63mts"
    },
    {
      "_id": "-EyukpQB8BzCwOA63mtu"
    },
    {
      "_id": "-UyukpQB8BzCwOA63mtu"
    },
    {
      "_id": "-kyukpQB8BzCwOA63mtu"
    },
    {
      "_id": "-0yukpQB8BzCwOA63mtu"
    },
    {
      "_id": "_EyukpQB8BzCwOA63mtu"
    },
    {
      "_id": "_UyukpQB8BzCwOA63mtu"
    },
    {
      "_id": "_kyukpQB8BzCwOA63mtw"
    },
    {
      "_id": "_0yukpQB8BzCwOA63mtw"
    },
    {
      "_id": "AEyukpQB8BzCwOA63mxw"
    },
    {
      "_id": "AUyukpQB8BzCwOA63mxw"
    },
    {
      "_id": "AkyukpQB8BzCwOA63mxw"
    },
    {
      "_id": "A0yukpQB8BzCwOA63mxw"
    },
    {
      "_id": "BEyukpQB8BzCwOA63mxx"
    },
    {
      "_id": "BUyukpQB8BzCwOA63mxx"
    },
    {
      "_id": "BkyukpQB8BzCwOA63mxx"
    },
    {
      "_id": "B0yukpQB8BzCwOA63mxx"
    },
    {
      "_id": "CEyukpQB8BzCwOA63mxx"
    },
    {
      "_id": "CUyukpQB8BzCwOA63mxx"
    }
  ],
  "result": "OK"
}

Solved The cause of the error was the agentList being initialized before looping through the commands and not being cleaned after each iteration.

     private static Orders commandsToOrders(NodeClient client, List<Command> commands) {
         Orders orders = new Orders();
-       List<Agent> agentList = new ArrayList<>();
         for (Command command : commands) {
+            List<Agent> agentList = new ArrayList<>();

Fix applied in commit: b5afbc9

AlexRuiz7 added level/task Task issue type/enhancement Enhancement issue labels Oct 2, 2024

AlexRuiz7 mentioned this issue Oct 2, 2024

Command manager development wazuh/wazuh-indexer#349

Open

22 tasks

wazuhci added this to XDR+SIEM/Release 5.0.0 Oct 3, 2024

wazuhci moved this to Backlog in XDR+SIEM/Release 5.0.0 Oct 3, 2024

AlexRuiz7 added mvp Minimum Viable Product and removed mvp Minimum Viable Product labels Dec 5, 2024

AlexRuiz7 mentioned this issue Dec 12, 2024

Data persistence model redesign wazuh/wazuh#22887

Open

25 tasks

AlexRuiz7 assigned QU3B1M Jan 14, 2025

wazuhci moved this from Backlog to In progress in XDR+SIEM/Release 5.0.0 Jan 14, 2025

wazuhci moved this from In progress to Backlog in XDR+SIEM/Release 5.0.0 Jan 15, 2025

wazuhci moved this from Backlog to In progress in XDR+SIEM/Release 5.0.0 Jan 16, 2025

QU3B1M linked a pull request Jan 17, 2025 that will close this issue

Implement commands expansion logic #229

Open

wazuhci moved this from In progress to Pending review in XDR+SIEM/Release 5.0.0 Jan 23, 2025

wazuhci moved this from Pending review to On hold in XDR+SIEM/Release 5.0.0 Jan 24, 2025

wazuhci moved this from On hold to In progress in XDR+SIEM/Release 5.0.0 Jan 27, 2025

wazuhci moved this from In progress to Pending review in XDR+SIEM/Release 5.0.0 Jan 27, 2025

wazuhci moved this from Pending review to In review in XDR+SIEM/Release 5.0.0 Jan 27, 2025

wazuhci moved this from In review to Pending final review in XDR+SIEM/Release 5.0.0 Jan 27, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement commands' expansion logic #88

Implement commands' expansion logic #88

AlexRuiz7 commented Oct 2, 2024 •

edited

Loading

QU3B1M commented Jan 20, 2025

QU3B1M commented Jan 22, 2025 •

edited

Loading

QU3B1M commented Jan 23, 2025 •

edited

Loading

Implement commands' expansion logic #88

Implement commands' expansion logic #88

Comments

AlexRuiz7 commented Oct 2, 2024 • edited Loading

Description

Pre-requisites

Functional requirements

QU3B1M commented Jan 20, 2025

QU3B1M commented Jan 22, 2025 • edited Loading

QU3B1M commented Jan 23, 2025 • edited Loading

AlexRuiz7 commented Oct 2, 2024 •

edited

Loading

QU3B1M commented Jan 22, 2025 •

edited

Loading

QU3B1M commented Jan 23, 2025 •

edited

Loading