Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add boundary testing #17

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add boundary testing #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
350bfce
test: add benchmark
giraud10 Jan 26, 2022
cad5154
test: add boundary testing
giraud10 Jan 27, 2022
b0ac135
test: generate data with pimo
giraud10 Jan 28, 2022
8b4a153
test: generate data with pimo
giraud10 Jan 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
218 changes: 218 additions & 0 deletions examples/boundary tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
# Boundary Tests on SIGO

This document presents the limitations of sigo regarding the size of the datasets
We will generate different datasets by varying the number of rows and the number of columns.
We will test these parameters on the different anonymization methods.

## Number of rows

We generate datasets of different sizes using `pimo`.
Below the `masking.yml` file allowing to generate a flow of jsonline with random float.

```yaml
version: "1"
seed: 42
masking:
- selector:
jsonpath: "A"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
- selector:
jsonpath: "B"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
```

We change the size of the datasets by using the `--repeat,-r` flag of `pimo` (***N = [100, 1000, 10000, 100000, 1000000]***).
And we anonymize the data with `sigo` using the anonymization method of our choice with the `--anonymizer,-a` flag.

```console
pimo < test.json -c masking.yml -r 100 > test1_1.json
sigo -q A,B -a general < test1_1.json > output.json
```

A bash script is written to automate the tests in the `rows.sh` file.

```console
cd rows
sudo chmod u+x rows.sh
. ./rows.sh
```

The results are listed in the `log.txt` file.

| NoAnonymizer | Size | Execution time (sec) | Results |
|-------|:----------:|:----------------------:|:-------:|
| Test1 | 100 | 0.00 | SUCCESS |
| Test2 | 1 000 | 0.00 | SUCCESS |
| Test3 | 10 000 | 2.00 | SUCCESS |
| Test4 | 100 000 | 27.00 | SUCCESS |
| Test5 | 1 000 000 | 418.00 | SUCCESS |
| Test6 | 10 000 000 | | FAILED |

<table>
<tr><th> Generalization </th><th> Aggregation </th><th> Top Bottom Coding </th><th> Random Noise </th></tr>
<tr><td>

| | Size | Execution time (sec) | Results |
|-------|:----------:|:----------------------:|:-------:|
| Test1 | 100 | 1.00 | SUCCESS |
| Test2 | 1 000 | 0.00 | SUCCESS |
| Test3 | 10 000 | 3.00 | SUCCESS |
| Test4 | 100 000 | 30.00 | SUCCESS |
| Test5 | 1 000 000 | 395.00 | SUCCESS |

</td><td>

| | Size | Execution time (sec) | Results |
|-------|:----------:|:----------------------:|:-------:|
| Test1 | 100 | 0.00 | SUCCESS |
| Test2 | 1 000 | 0.00 | SUCCESS |
| Test3 | 10 000 | 3.00 | SUCCESS |
| Test4 | 100 000 | 29.00 | SUCCESS |
| Test5 | 1 000 000 | 386.00 | SUCCESS |

</td><td>

| | Size | Execution time (sec) | Results |
|-------|:----------:|:----------------------:|:-------:|
| Test1 | 100 | 0.00 | SUCCESS |
| Test2 | 1 000 | 0.00 | SUCCESS |
| Test3 | 10 000 | 3.00 | SUCCESS |
| Test4 | 100 000 | 28.00 | SUCCESS |
| Test5 | 1 000 000 | 398.00 | SUCCESS |

</td><td>

| | Size | Execution time (sec) | Results |
|-------|:----------:|:----------------------:|:-------:|
| Test1 | 100 | 0.00 | SUCCESS |
| Test2 | 1 000 | 1.00 | SUCCESS |
| Test3 | 10 000 | 3.00 | SUCCESS |
| Test4 | 100 000 | 37.00 | SUCCESS |
| Test5 | 1 000 000 | 420.00 | SUCCESS |

</td></tr> </table>

![rows](rows/rows.png)

## Number of columns

Now we generate a dataset of 1000 rows using `pimo` and we change the number of attributes.
To do this we take the `masking.yml` file from the **rows folder** and add additional masks for each new attribute.
Here an example for the test with 4 attributes,

```yaml
version: "1"
seed: 42
masking:
- selector:
jsonpath: "A"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
- selector:
jsonpath: "B"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
- selector:
jsonpath: "C"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
- selector:
jsonpath: "D"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
```

So we get 5 masking.yml files :

- `masking1.yml` for 2 attributes.
- `masking2.yml` for 4 attributes.
- `masking3.yml` for 8 attributes.
- `masking4.yml` for 16 attributes.
- `masking5.yml` for 32 attributes.

```console
pimo < test2.json -c masking2.yml -r 1000 > test2_2.json
sigo -q A,B,C,D -a general < test2_2.json > output.json
```

The bash script for test automation is in the `columns.sh` file and the results are in the `log.txt` file.

```console
cd columns
sudo chmod u+x columns.sh
. ./columns.sh
```

| NoAnonymizer | Attributes | Execution time (sec) | Results |
|-------|:----------:|:----------------------:|:-------:|
| Test1 | 2 | 0.00 | SUCCESS |
| Test2 | 4 | 1.00 | SUCCESS |
| Test3 | 8 | 0.00 | SUCCESS |
| Test4 | 16 | 1.00 | SUCCESS |
| Test5 | 32 | 3.00 | SUCCESS |

<table>
<tr><th> Generalization </th><th> Aggregation </th><th> Top Bottom Coding </th><th> Random Noise </th></tr>
<tr><td>

| | Attributes | Execution time (sec) | Results |
|-------|:----------:|:----------------------:|:-------:|
| Test1 | 2 | 0.00 | SUCCESS |
| Test2 | 4 | 1.00 | SUCCESS |
| Test3 | 8 | 1.00 | SUCCESS |
| Test4 | 16 | 2.00 | SUCCESS |
| Test5 | 32 | 4.00 | SUCCESS |

</td><td>

| | Attributes | Execution time (sec) | Results |
|-------|:----------:|:----------------------:|:-------:|
| Test1 | 2 | 0.00 | SUCCESS |
| Test2 | 4 | 1.00 | SUCCESS |
| Test3 | 8 | 1.00 | SUCCESS |
| Test4 | 16 | 2.00 | SUCCESS |
| Test5 | 32 | 4.00 | SUCCESS |

</td><td>

| Top Bottom Coding | Attributes | Execution time (sec) | Results |
|-------|:----------:|:----------------------:|:-------:|
| Test1 | 2 | 0.00 | SUCCESS |
| Test2 | 4 | 0.00 | SUCCESS |
| Test3 | 8 | 0.00 | SUCCESS |
| Test4 | 16 | 1.00 | SUCCESS |
| Test5 | 32 | 4.00 | SUCCESS |

</td><td>

| | Attributes | Execution time (sec) | Results |
|-------|:----------:|:----------------------:|:-------:|
| Test1 | 2 | 0.00 | SUCCESS |
| Test2 | 4 | 0.00 | SUCCESS |
| Test3 | 8 | 0.00 | SUCCESS |
| Test4 | 16 | 2.00 | SUCCESS |
| Test5 | 32 | 4.00 | SUCCESS |

</td></tr> </table>

![columns](columns/columns.png)
Binary file added examples/boundary tests/columns/columns.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
86 changes: 86 additions & 0 deletions examples/boundary tests/columns/columns.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
#!/bin/bash

rm -f log.txt

size_array=("2" "4" "8" "16" "32")
qi_array=("A,B" "A,B,C,D" "A,B,C,D,E,F,G,H" "A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P" "A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,AA,BB,CC,DD,EE,FF")

echo "NOANONYMIZER" >> log.txt
echo "--------------------------------------------------------------------------------------------------------------" >> log.txt

for i in 1 2 3 4 5
do
pimo < test${i}.json -c masking${i}.yml -r 1000 > test${i}_2.json
START=$(date +%s)
sigo -q ${qi_array[$i]} < test${i}_2.json > output.json
END=$(date +%s)
DIFF=$(( $END - $START ))
echo "Test2_$i: (n=1000, x=${size_array[$i]}, method=NoAnomymizer) Execution time was $DIFF seconds." >> log.txt
rm -f test${i}_2.json
rm -f output.json
done

echo "" >> log.txt
echo "GENERALIZATION" >> log.txt
echo "--------------------------------------------------------------------------------------------------------------" >> log.txt

for i in 1 2 3 4 5
do
pimo < test${i}.json -c masking${i}.yml -r 1000 > test${i}_2.json
START=$(date +%s)
sigo -q ${qi_array[$i]} -a general < test${i}_2.json > output.json
END=$(date +%s)
DIFF=$(( $END - $START ))
echo "Test2_$i: (n=1000, x=${size_array[$i]}, method=Generalization) Execution time was $DIFF seconds." >> log.txt
rm -f test${i}_2.json
rm -f output.json
done

echo "" >> log.txt
echo "AGGREGATION" >> log.txt
echo "--------------------------------------------------------------------------------------------------------------" >> log.txt

for i in 1 2 3 4 5
do
pimo < test${i}.json -c masking${i}.yml -r 1000 > test${i}_2.json
START=$(date +%s)
sigo -q ${qi_array[$i]} -a meanAggregation < test${i}_2.json > output.json
END=$(date +%s)
DIFF=$(( $END - $START ))
echo "Test2_$i: (n=1000, x=${size_array[$i]}, method=Aggregation) Execution time was $DIFF seconds." >> log.txt
rm -f test${i}_2.json
rm -f output.json
done

echo "" >> log.txt
echo "TOPBOTTOMCODING" >> log.txt
echo "--------------------------------------------------------------------------------------------------------------" >> log.txt

for i in 1 2 3 4 5
do
pimo < test${i}.json -c masking${i}.yml -r 1000 > test${i}_2.json
START=$(date +%s)
sigo -q ${qi_array[$i]} -a outlier < test${i}_2.json > output.json
END=$(date +%s)
DIFF=$(( $END - $START ))
echo "Test2_$i: (n=1000, x=${size_array[$i]}, method=TopBottomCoding) Execution time was $DIFF seconds." >> log.txt
rm -f test${i}_2.json
rm -f output.json
done

echo "" >> log.txt
echo "RANDOMNOISE" >> log.txt
echo "--------------------------------------------------------------------------------------------------------------" >> log.txt

for i in 1 2 3 4 5
do
pimo < test${i}.json -c masking${i}.yml -r 1000 > test${i}_2.json
START=$(date +%s)
sigo -q ${qi_array[$i]} -a laplaceNoise < test${i}_2.json > output.json
END=$(date +%s)
DIFF=$(( $END - $START ))
echo "Test2_$i: (n=1000, x=${size_array[$i]}, method=RandomNoise) Execution time was $DIFF seconds." >> log.txt
rm -f test${i}_2.json
rm -f output.json
done

39 changes: 39 additions & 0 deletions examples/boundary tests/columns/log.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
NOANONYMIZER
--------------------------------------------------------------------------------------------------------------
Test2_1: (n=1000, x=2, method=NoAnomymizer) Execution time was 0 seconds.
Test2_2: (n=1000, x=4, method=NoAnomymizer) Execution time was 1 seconds.
Test2_3: (n=1000, x=8, method=NoAnomymizer) Execution time was 0 seconds.
Test2_4: (n=1000, x=16, method=NoAnomymizer) Execution time was 1 seconds.
Test2_5: (n=1000, x=32, method=NoAnomymizer) Execution time was 3 seconds.

GENERALIZATION
--------------------------------------------------------------------------------------------------------------
Test2_1: (n=1000, x=2, method=Generalization) Execution time was 0 seconds.
Test2_2: (n=1000, x=4, method=Generalization) Execution time was 1 seconds.
Test2_3: (n=1000, x=8, method=Generalization) Execution time was 1 seconds.
Test2_4: (n=1000, x=16, method=Generalization) Execution time was 2 seconds.
Test2_5: (n=1000, x=32, method=Generalization) Execution time was 4 seconds.

AGGREGATION
--------------------------------------------------------------------------------------------------------------
Test2_1: (n=1000, x=2, method=Aggregation) Execution time was 0 seconds.
Test2_2: (n=1000, x=4, method=Aggregation) Execution time was 1 seconds.
Test2_3: (n=1000, x=8, method=Aggregation) Execution time was 1 seconds.
Test2_4: (n=1000, x=16, method=Aggregation) Execution time was 2 seconds.
Test2_5: (n=1000, x=32, method=Aggregation) Execution time was 4 seconds.

TOPBOTTOMCODING
--------------------------------------------------------------------------------------------------------------
Test2_1: (n=1000, x=2, method=TopBottomCoding) Execution time was 0 seconds.
Test2_2: (n=1000, x=4, method=TopBottomCoding) Execution time was 0 seconds.
Test2_3: (n=1000, x=8, method=TopBottomCoding) Execution time was 0 seconds.
Test2_4: (n=1000, x=16, method=TopBottomCoding) Execution time was 1 seconds.
Test2_5: (n=1000, x=32, method=TopBottomCoding) Execution time was 4 seconds.

RANDOMNOISE
--------------------------------------------------------------------------------------------------------------
Test2_1: (n=1000, x=2, method=RandomNoise) Execution time was 0 seconds.
Test2_2: (n=1000, x=4, method=RandomNoise) Execution time was 0 seconds.
Test2_3: (n=1000, x=8, method=RandomNoise) Execution time was 0 seconds.
Test2_4: (n=1000, x=16, method=RandomNoise) Execution time was 2 seconds.
Test2_5: (n=1000, x=32, method=RandomNoise) Execution time was 4 seconds.
17 changes: 17 additions & 0 deletions examples/boundary tests/columns/masking1.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
version: "1"
seed: 42
masking:
- selector:
jsonpath: "A"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
- selector:
jsonpath: "B"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
31 changes: 31 additions & 0 deletions examples/boundary tests/columns/masking2.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
version: "1"
seed: 42
masking:
- selector:
jsonpath: "A"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
- selector:
jsonpath: "B"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
- selector:
jsonpath: "C"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
- selector:
jsonpath: "D"
mask:
randomDecimal:
min: 0
max: 100.00
precision: 2
Loading