Tracee, OPA, and WebAssembly (WASM) - as of May 2022

[danielpacak]

Summary

Based on my experiments with OPA WASM and wazero¹ runtime implementation I'd not recommend using it for high-throughput Tracee-Rules application. Mainly, because initial benchmark results² do not show significant improvements in comparison to OPA Go SDK. What's more, the implementation gets complex very quickly.

OPA is a powerful and generic tool for evaluating policies. However, there is a price to pay for such generic implementation in performance. Most of the Tracee-Rules signatures evaluate to simple boolean values. Using OPA, however, causes lots of objects on the heap to be created and decreases the overall processing speed. Beyond that, there's data serialization overhead whenever we cross WebAssembly runtime boundaries.

What we probably need is a less generic but more performant alternative. Notice also that some Tracee-Rules signatures do (document) transformation of the input event, but this functionality, for performance reasons, can be implemented in an imperative programming language after the policy is evaluated.

Recommended Actions

1. Consider alternatives to OPA Rego as Domain Specific Language (DSL) to author signatures. For example, common expression language³ can be used to write the Anti-Debugging signature as follows:

    input.EventName == 'ptrace' && arg(input, 'request') == 'PTRACE_TRACEME'
   

   The same or similar expression language can be used for filtering events in user-space and (maybe one day) in kernel.
2. (Optional) See how "All-in-one" OPA Rego policy evaluation performs with wazero in comparison to evaluating policies one by one.
3. (Optional) Follow up with OPA maintainers for more efficient ast.ParseTerm implementation that’s optimized for boolean Rego rules.

Remarks

Even though we proved it’s possible to compile OPA Rego policy to WebAssembly module (WASM) and evaluate rules with a specified input event using wazero runtime⁴, this solution has a few drawbacks:

1. We are not solving the main performance problem in Tracee-Rules. We are moving it from OPA Go SDK somewhere else, i.e. wazero implementation.
2. The wazero runtime implementation is not feature complete. See, for example Is it possible to grow api.Memory programmatically? tetratelabs/wazero#483
3. It's non trivial task to use WebAssembly / wazero API to write efficient signatures evaluation engine and success mainly depends on the upstream dependency implementation.

[danielpacak]

We further discussed alternatives to OPA Rego with @AsafEitani and here's a snippet of YAML that may be used to add expression-based signatures to Tracee-Rules. It's a rough idea, not a design:

# tracee-config.yml

signatures:
  - metadata:
      id: "TRC -2"
      varsion: "0.1.0"
      name: "Anti-Debugging"
      description: "Process uses anti-debugging technique to block debugger"
      tags: ["linux", "container"]
      properties:
        "Severity": 3,
        "MITRE ATT&CK": "Defense Evasion: Execution Guardrails"
    eventSelector:
      - source: "tracee"
        name: "ptrace"
    expression: |
      input.EventName == 'ptrace' &&
        argument(input, 'request') == 'PTRACE_TRACEME'"
    outputData:
      foo: input.EventName
      bar: argument(input, "flags")
    outputTemplate: |
        // go template
    outputCommand: |
        // add example

[AsafEitani]

It's important that we try to maintain the current capabilities as we discussed, like preserving the ability to easily extract information on the alert (outputData) in the example.
We also will always have the Go solution for more complex signatures.
We can choose either YAML or JSON as a template and seperate the signatures so we can easily copy-paste them if needed.

I agree the rego is not the best choice and that we need to find a better solution.

The only concern I have regarding the DSL is a vulnerability in the engine that will lead to code execution in tracee-rules.
We should implement security measures for the solution.

[danielpacak]

We've been working on a POC to validate an idea of using Common Expression Language (CEL) signatures: #1766

[itaysk]

seems to be some upstream attempts to simplify object serialization entering while entering opa, which was previously identified as a bottleneck. just bookmarking this here for future reference:
open-policy-agent/opa#4708
open-policy-agent/opa#4709
open-policy-agent/opa#4714

[TristonianJones]

Hi All, I happened to notice the latest CEL proof of concept in Tracee and was curious what your experience has been so far. Any feedback or rough edges you'd like to see improved going forward?

-Tristan (CEL maintainer)

[rafaeldtinoco]

Hello @TristonianJones! Thanks for checking by!

We have not committed (or gave up) go-cel still (yet ?). It actually outperformed other attempts of expression evaluation we tried and/or had.

The benchmarks done in a recent past were very promising. It outperformed REGO by 3 or 4x in high frequency events for the types evaluations we tested (our "go-cel" signatures), and outperformed WASM signature as well (Our regular benchmarks for REGO, REGO+Wasm, Golang: https://github.com/aquasecurity/tracee/tree/main/pkg/signatures/benchmark and the benchmarks we used witH Go Cel: https://github.com/aquasecurity/tracee/blob/main/pkg/signatures/celsig/signature_test.go#L217).

Our main problem back then was:

1. We did not (and still don't) use protobuf for our main event type, so we needed to create a wrapper to do a fake protobuf (and, even with this wrapper, it was still much better in performance than other expression evaluation approaches) for our internal types so we could use go-cel macros feature (https://github.com/aquasecurity/tracee/blob/main/pkg/signatures/celsig/wrapper/wrapper.go).

2. We saw we would need to create macros for the "special types" coming from the kernel for the evaluation (

   tracee/pkg/signatures/celsig/library.go

   Line 66 in 0213c08

   func (c *customLib) sockaddrArg(lhs ref.Val, rhs ref.Val) ref.Val {

   ), to avoid often type conversion (because of performance reasons), but I guess this is super ok.

3. We have a legacy of signatures (some not open sourced) that migrated from REGO to Go (go plugin) and we would have to migrate again all of them to Go Cel.

Recently @NDStrahilevitz and I discussed that we could use it also for our cmdline parsing. We have multiple type of filters and each type requires a "filtering mechanism". For our complex types (structs coming from kernel) we're thinking in using go-cel (but it will also require us using protobuf I guess) for a "generic type" arguments evaluation. Lets see...

Anyway, TL;DR story is: we like it, but we're dealing with "bigger" architectural changes right now to give it a higher focus (unfortunately actually).