Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluation on docker stats #5

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Evaluation on docker stats #5

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
11c32ec
1. When compiling Model.sol, files that needs to be removed will be a…
majalinderholm Nov 25, 2020
38c2dd5
Now we can collect stats and plot them for one implementation. Will c…
majalinderholm Dec 1, 2020
7a28d49
Merge remote-tracking branch 'origin/master' into evaluation/docker-s…
majalinderholm Dec 1, 2020
545198b
Now possible to plot stats from the original implementation and enhan…
majalinderholm Dec 1, 2020
015c98c
Made scatter plots of CPU% and memory usage for all containers. Will …
majalinderholm Dec 2, 2020
5b491be
Now we are able to plot both the original and enhanced implementation…
majalinderholm Dec 2, 2020
7bf3bd2
Just added some units to the axis
majalinderholm Dec 8, 2020
d4f7518
Docker stats in runs about every 2s and therefore we want to track th…
majalinderholm Dec 8, 2020
a2cb4aa
Collect stats now keeps track of time stamps and the python script ev…
majalinderholm Dec 9, 2020
2980ae5
Just added a time column to the worker dataframes after the split of …
majalinderholm Dec 9, 2020
2ba7c17
Now we have five workers. Edited eval_stats.py so that it takes five …
majalinderholm Dec 10, 2020
789ada2
Rearranged the script into functions :)
majalinderholm Dec 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions collect_stats.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
while true;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this file still needed?

do
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}" --no-stream >> stats.csv
sleep 1;
done
3 changes: 3 additions & 0 deletions model/chain/ethereum/contract/compile.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
#!/bin/bash
rm -rf /tmp/ethbuild
rm -rf ./-p/

mkdir /tmp/ethbuild -p && solc --bin --abi Model.sol -o /tmp/ethbuild/ --overwrite && abigen --bin=/tmp/ethbuild/Model.bin --abi=/tmp/ethbuild/Model.abi --pkg=contract --out=Model.go
5 changes: 5 additions & 0 deletions performance/collect_stats.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
while true;
do
docker stats --format "table {{.Name}},{{.CPUPerc}},{{.MemUsage}},{{.MemPerc}}" --no-stream >> stats.csv
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(2/2) Follow comment (1/2), change this line to **** >> performance/stats.csv

sleep 0.5;
done
160 changes: 160 additions & 0 deletions performance/eval_stats.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

###READ CSV AND PREPROCESS DATA###
df_enh = pd.read_csv("stats_project1.csv", sep=',')
df_enh = df_enh.loc[df_enh['NAME']!='NAME']
df_enh = df_enh.replace({'%':''}, regex=True)
df_enh['MEM USAGE / LIMIT'] = df_enh['MEM USAGE / LIMIT'].str.split('M').str[0]
df_enh['MEM USAGE / LIMIT'] = df_enh['MEM USAGE / LIMIT'].str.split('B').str[0]
df_enh[['CPU %', 'MEM USAGE / LIMIT', 'MEM %']] = df_enh[['CPU %', 'MEM USAGE / LIMIT', 'MEM %']].apply(pd.to_numeric)

df_ori = pd.read_csv("stats_felix1.csv", sep=',')
df_ori = df_ori.loc[df_ori['NAME']!='NAME']
df_ori = df_ori.replace({'%':''}, regex=True)
df_ori['MEM USAGE / LIMIT'] = df_ori['MEM USAGE / LIMIT'].str.split('M').str[0]
df_ori['MEM USAGE / LIMIT'] = df_ori['MEM USAGE / LIMIT'].str.split('B').str[0]
df_ori[['CPU %', 'MEM USAGE / LIMIT', 'MEM %']] = df_ori[['CPU %', 'MEM USAGE / LIMIT', 'MEM %']].apply(pd.to_numeric)

###SPLIT DATA AND ADD STATS FOR WORKERS###
group = df_enh.groupby('NAME')
redis_enh = group.get_group('redis')
chain_enh = group.get_group('chain')
worker1_enh = group.get_group('local_worker1_1')
worker2_enh = group.get_group('local_worker2_1')
worker3_enh = group.get_group('local_worker3_1')

worker1_enh.index = np.arange(1, len(worker1_enh)+1)
worker2_enh.index = np.arange(1, len(worker2_enh)+1)
worker3_enh.index = np.arange(1, len(worker3_enh)+1)

group = df_ori.groupby('NAME')
redis_ori = group.get_group('redis')
chain_ori = group.get_group('chain')
worker1_ori = group.get_group('local_worker1_1')
worker2_ori = group.get_group('local_worker2_1')
worker3_ori = group.get_group('local_worker3_1')

worker1_ori.index = np.arange(1, len(worker1_ori)+1)
worker2_ori.index = np.arange(1, len(worker2_ori)+1)
worker3_ori.index = np.arange(1, len(worker3_ori)+1)

###MAKE SOME CALCULATIONS###
cpu_enh = worker1_enh['CPU %'] + worker2_enh['CPU %'] + worker3_enh['CPU %']
mem_enh = worker1_enh['MEM USAGE / LIMIT'] + worker2_enh['MEM USAGE / LIMIT'] + worker3_enh['MEM USAGE / LIMIT']
workers_enh = pd.DataFrame(data={'CPU %': cpu_enh, 'MEM USAGE / LIMIT': mem_enh})
workers_enh['ACCUMULATED CPU %'] = workers_enh['CPU %'].cumsum()
workers_enh['ACCUMULATED MEM USAGE'] = workers_enh['MEM USAGE / LIMIT'].cumsum()

cpu_ori = worker1_ori['CPU %'] + worker2_ori['CPU %'] + worker3_ori['CPU %']
mem_ori = worker1_ori['MEM USAGE / LIMIT'] + worker2_ori['MEM USAGE / LIMIT'] + worker3_ori['MEM USAGE / LIMIT']
workers_ori = pd.DataFrame(data={'CPU %': cpu_ori, 'MEM USAGE / LIMIT': mem_ori})
workers_ori['ACCUMULATED CPU %'] = workers_ori['CPU %'].cumsum()
workers_ori['ACCUMULATED MEM USAGE'] = workers_ori['MEM USAGE / LIMIT'].cumsum()

redis_enh['ACCUMULATED CPU %'] = redis_enh['CPU %'].cumsum()
redis_ori['ACCUMULATED CPU %'] = redis_ori['CPU %'].cumsum()

chain_enh['ACCUMULATED CPU %'] = chain_enh['CPU %'].cumsum()
chain_ori['ACCUMULATED CPU %'] = chain_ori['CPU %'].cumsum()

###ADD TIME STAMPS TO THE DATAFRAME###
redis_enh['TIME'] = np.arange(0,len(redis_enh)/2,0.5)
chain_enh['TIME'] = np.arange(0,len(chain_enh)/2,0.5)
workers_enh['TIME'] = np.arange(0,len(workers_enh)/2,0.5)

redis_ori['TIME'] = np.arange(0,len(redis_ori)/2,0.5)
chain_ori['TIME'] = np.arange(0,len(chain_ori)/2,0.5)
workers_ori['TIME'] = np.arange(0,len(workers_ori)/2,0.5)

###PLOT CPU% FOR REDIS###
ax1 = redis_enh.plot(kind='scatter',x='TIME',y='CPU %',color='red', label='Enhanced implementation')
ax2 = redis_ori.plot(kind='scatter',x='TIME',y='CPU %',color='blue', label='Original implementation', ax=ax1)
red = mpatches.Patch(color='red', label='Enhanced implementation')
blue = mpatches.Patch(color='blue', label='Original implementation')
plt.legend(handles=[red, blue])
plt.title('Redis CPU%')
plt.show()

###PLOT ACCUMULATED CPU% FOR REDIS###
ax1 = redis_enh.plot(kind='line',x='TIME',y='ACCUMULATED CPU %',color='red', label='Enhanced implementation')
ax2 = redis_ori.plot(kind='line',x='TIME',y='ACCUMULATED CPU %',color='blue', label='Original implementation', ax=ax1)
red = mpatches.Patch(color='red', label='Enhanced implementation')
blue = mpatches.Patch(color='blue', label='Original implementation')
plt.legend(handles=[red, blue])
plt.title('Accumulated Redis CPU%')
plt.show()

###PLOT CPU% FOR CHAIN###
ax1 = chain_enh.plot(kind='scatter',x='TIME',y='CPU %',color='red', label='Enhanced implementation')
ax2 = chain_ori.plot(kind='scatter',x='TIME',y='CPU %',color='blue', label='Original implementation', ax=ax1)
red = mpatches.Patch(color='red', label='Enhanced implementation')
blue = mpatches.Patch(color='blue', label='Original implementation')
plt.legend(handles=[red, blue])
plt.title('Chain CPU%')
plt.show()

###PLOT ACCUMULATED CPU% FOR CHAIN###
ax1 = chain_enh.plot(kind='line',x='TIME',y='ACCUMULATED CPU %',color='red', label='Enhanced implementation')
ax2 = chain_ori.plot(kind='line',x='TIME',y='ACCUMULATED CPU %',color='blue', label='Original implementation', ax=ax1)
red = mpatches.Patch(color='red', label='Enhanced implementation')
blue = mpatches.Patch(color='blue', label='Original implementation')
plt.legend(handles=[red, blue])
plt.title('Accumulated Chain CPU%')
plt.show()

###PLOT SUM OF CPU% FOR WORKERS###
ax1 = workers_enh.plot(kind='scatter',x='TIME',y='CPU %',color='red', label='Enhanced implementation')
ax2 = workers_ori.plot(kind='scatter',x='TIME',y='CPU %',color='blue', label='Original implementation', ax=ax1)
red = mpatches.Patch(color='red', label='Enhanced implementation')
blue = mpatches.Patch(color='blue', label='Original implementation')
plt.legend(handles=[red, blue])
plt.title('Sum of Workers CPU%')
plt.show()

###PLOT ACCUMULATED SUM OF CPU% FOR WORKERS###
ax1 = workers_enh.plot(kind='line',x='TIME',y='ACCUMULATED CPU %',color='red', label='Enhanced implementation')
ax2 = workers_ori.plot(kind='line',x='TIME',y='ACCUMULATED CPU %',color='blue', label='Original implementation', ax=ax1)
red = mpatches.Patch(color='red', label='Enhanced implementation')
blue = mpatches.Patch(color='blue', label='Original implementation')
plt.legend(handles=[red, blue])
plt.title('Accumulated Sum of Workers CPU%')
plt.show()

###PLOT MEM FOR REDIS###
ax1 = redis_enh.plot(kind='line',x='TIME',y='MEM USAGE / LIMIT',color='red', label='Enhanced implementation')
ax2 = redis_ori.plot(kind='line',x='TIME',y='MEM USAGE / LIMIT',color='blue', label='Original implementation', ax=ax1)
red = mpatches.Patch(color='red', label='Enhanced implementation')
blue = mpatches.Patch(color='blue', label='Original implementation')
plt.legend(handles=[red, blue])
plt.title('Redis Memory Usage')
plt.show()

###PLOT MEM FOR CHAIN###
ax1 = chain_enh.plot(kind='line',x='TIME',y='MEM USAGE / LIMIT',color='red', label='Enhanced implementation')
ax2 = chain_ori.plot(kind='line',x='TIME',y='MEM USAGE / LIMIT',color='blue', label='Original implementation', ax=ax1)
red = mpatches.Patch(color='red', label='Enhanced implementation')
blue = mpatches.Patch(color='blue', label='Original implementation')
plt.legend(handles=[red, blue])
plt.title('Chain Memory Usage')
plt.show()

###PLOT SUM OF MEM FOR WORKERS###
ax1 = workers_enh.plot(kind='scatter',x='TIME',y='MEM USAGE / LIMIT',color='red', label='Enhanced implementation')
ax2 = workers_ori.plot(kind='scatter',x='TIME',y='MEM USAGE / LIMIT',color='blue', label='Original implementation', ax=ax1)
red = mpatches.Patch(color='red', label='Enhanced implementation')
blue = mpatches.Patch(color='blue', label='Original implementation')
plt.legend(handles=[red, blue])
plt.title('Sum of Workers Memory Usage')
plt.show()

###PLOT ACCUMULATED SUM OF MEM FOR WORKERS
ax1 = workers_enh.plot(kind='line',x='TIME',y='ACCUMULATED MEM USAGE',color='red', label='Enhanced implementation')
ax2 = workers_ori.plot(kind='line',x='TIME',y='ACCUMULATED MEM USAGE',color='blue', label='Original implementation', ax=ax1)
red = mpatches.Patch(color='red', label='Enhanced implementation')
blue = mpatches.Patch(color='blue', label='Original implementation')
plt.legend(handles=[red, blue])
plt.title('Accumulated Sum of Workers Memory Usage')
plt.show()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice plots and handling the data with Pandas!
A small suggestion: wrapping repeated code blocks into reusable functions, time-saving and easier for future maintenance.

3 changes: 3 additions & 0 deletions start_training.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ fi

if [ "$SCENARIO" == "local" ];
then
cd performance
./collect_stats.sh &
cd -
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(1/2) How about replacing these 3 lines with 1 line sh ./performance/collect_stats.sh &, and also make a small change in collect_stats.sh, see (2/2)

cd scenarios/local
sudo MODEL=$CONTRACT docker-compose up
else
Expand Down