Merge pull request #16 from doganulus/onebase

Onebase

Author: doganulus

docs/performance.md

@@ -1,32 +1,83 @@
-# Benchmark Results
 
-We use the benchmark generator [timescales](https://github.com/doganulus/timescales) to benchmark the performance of reelay MTL monitoring app.  For measurements, we use [multitime](https://github.com/ltratt/multitime) utility. Tests are performed using a laptop computer (Intel Core i5-7200U CPU @2.50 Ghz, 8GB RAM) on Ubuntu 18.04 on WSL (Build 18362.449).
+# Performance
 
-The full benchmark suite of `timescales` can be built using the command:
+This document present the performance of `ryjson1` application,which checks Rye specifications over multiline JSON logfiles and provide a command line interface to Reelay monitors.
+
+We use `timescales` benchmark generator to measure the performance of `ryjson1`. The benchmark covers a number of common classes of specifications with timing constraints. More information regarding the benchmark generator and motivations can be found in the paper of [Timescales: A Benchmark Generator for MTL Monitoring Tools](https://link.springer.com/chapter/10.1007/978-3-030-32079-9_25). 
+
+
+## Benchmark Reelay
+
+The benchmark can be replicated by typing commands at the Reelay project directory. First generate some data using `timescales` by typing
 
     make timescales
 
-This command will clone `timescales` repository and generate the full suite.
+This command will clone `timescales` generator and generates discrete and dense time behaviors for benchmarking. The total size of generated benchmarks is about 2.2GB. There are two variant for dense time behaviors with two different degrees of [condensation]().
+
+    make test_performance_discrete
+    make test_performance_dense10
+    make test_performance_dense100
+
+To execute these commands, you need `multitime` utility for benchmarking with these commands, which can be installed via `apt` or [from source](https://github.com/ltratt/multitime).
+
+## Results
+
+Below we present performance of `ryjson1` aapplication on a laptop computer (Intel Core i5-7200U CPU @2.50 Ghz, 8GB RAM) on Ubuntu 20.04 and using the version 2008 of Reelay. And we report minimum user time of 10 executions. 
+
+
+### 1) Discrete results
+
+Results over processing discrete time behaviors with a length of 1M.
+
+    | Property   | 10x   | 100x  | 1000x |
+    | ---------- | ----- | ----- | ----- |
+    | AbsentAQ   | 0.246 | 0.224 | 0.234 |
+    | AbsentBR   | 0.265 | 0.268 | 0.277 |
+    | AbsentBQR  | 0.376 | 0.347 | 0.339 |
+    | AlwaysAQ   | 0.242 | 0.223 | 0.219 |
+    | AlwaysBR   | 0.265 | 0.264 | 0.261 |
+    | AlwaysBQR  | 0.388 | 0.342 | 0.342 |
+    | RecurGLB   | 0.211 | 0.164 | 0.155 |
+    | RecurBQR   | 0.473 | 0.412 | 0.411 |
+    | RespondGLB | 0.395 | 0.321 | 0.310 |
+    | RespondBQR | 0.597 | 0.544 | 0.529 |
+    | ---------- | ----- | ----- | ----- |
+
+
+
+### 2) Dense10 results
 
-## Discrete Time MTL Application 
+Condensation allows to represent periods of the same JSON object with a single timestamped object. Dense10 behaviors limits this feature maximum 10 successive time points. Consequently the same temporal behavior is expressed a shorter logfile (up to 10x). Dense time monitors may work over such condensed (dense) logfiles.
 
-Test command:
+    | Property   | 10x   | 100x  | 1000x |
+    | ---------- | ----- | ----- | ----- |
+    | AbsentAQ   | 0.446 | 0.356 | 0.334 |
+    | AbsentBR   | 0.398 | 0.344 | 0.335 |
+    | AbsentBQR  | 0.678 | 0.189 | 0.124 |
+    | AlwaysAQ   | 0.453 | 0.358 | 0.342 |
+    | AlwaysBR   | 0.392 | 0.346 | 0.329 |
+    | AlwaysBQR  | 0.896 | 0.243 | 0.169 |
+    | RecurGLB   | 0.302 | 0.103 | 0.083 |
+    | RecurBQR   | 0.676 | 0.222 | 0.169 |
+    | RespondGLB | 0.751 | 0.222 | 0.155 |
+    | RespondBQR | 1.452 | 0.402 | 0.253 |
+    | ---------- | ----- | ----- | ----- |
 
-    make test_mtl_performance_discrete
+### Dense100 results
 
-Performance results (real, min) of for monitoring common temporal properties over traces with a length of 1 million rows:
+Condensation allows to represent periods of the same JSON object with a single timestamped object. Dense100 behaviors limits this feature maximum 100 successive time points. Consequently the same temporal behavior is expressed a shorter logfile (up to 100x). Dense time monitors may work over such condensed (dense) logfiles.
 
-| Property/Scale | 10x | 100x | 1000x |
-| ---------- | ----- | ----- | ----- |
-| AbsentAQ   | 0.834 | 0.828 | 0.830 |
-| AbsentBR   | 0.809 | 0.833 | 0.843 |
-| AbsentBQR  | 0.985 | 0.952 | 0.957 |
-| AlwaysAQ   | 0.827 | 0.815 | 0.814 |
-| AlwaysBR   | 0.785 | 0.790 | 0.793 |
-| AlwaysBQR  | 0.973 | 0.951 | 0.955 |
-| RecurGLB   | 0.754 | 0.698 | 0.702 |
-| RecurBQR   | 1.102 | 1.068 | 1.068 |
-| RespondGLB | 1.008 | 0.927 | 0.943 |
-| RespondBQR | 1.285 | 1.237 | 1.215 |
+    | Property   | 10x   | 100x  | 1000x |
+    | ---------- | ----- | ----- | ----- |
+    | AbsentAQ   | 0.430 | 0.290 | 0.276 |
+    | AbsentBR   | 0.387 | 0.301 | 0.294 |
+    | AbsentBQR  | 0.627 | 0.103 | 0.033 |
+    | AlwaysAQ   | 0.432 | 0.271 | 0.266 |
+    | AlwaysBR   | 0.376 | 0.273 | 0.280 |
+    | AlwaysBQR  | 0.828 | 0.126 | 0.044 |
+    | RecurGLB   | 0.272 | 0.046 | 0.027 |
+    | RecurBQR   | 0.671 | 0.097 | 0.037 |
+    | RespondGLB | 0.729 | 0.108 | 0.041 |
+    | RespondBQR | 1.378 | 0.217 | 0.060 |
+    | ---------- | ----- | ----- | ----- |
 
-Multitime batch files for these tests can be found [here](https://github.com/doganulus/reelay/tree/master/test/timescales/discrete/multitime).

mkdocs.yml

@@ -32,8 +32,8 @@ nav:
   - User Manual: 
     - 'Temporal Behaviors' : 'behaviors.md'
     - 'Rye Format' : 'rye.md'
-    - 'C++ API' : 'cpp_api.md'
-    - 'Python API' : 'python_api.md'
+    # - 'C++ API' : 'cpp_api.md'
+    # - 'Python API' : 'python_api.md'
     - 'Benchmarking' : 'performance.md'
 
   - Advanced Topics: