diff --git a/experiment.tex b/experiment.tex index 77467a8..c964cfd 100644 --- a/experiment.tex +++ b/experiment.tex @@ -1,34 +1,27 @@ \section{Experiments}\label{sec:experiment} -We experimentally evaluated the performance and quality of our methodology, and corresponding heuristic implementation in \cref{subsec:heuristics}, -and compare them against the exhaustive approach in Section~\cref{sec:heuristics}. In the following, -\cref{subsec:experiments_infrastructure} presents the simulator and testing infrastructure adopted in our experiments, as well as the complete experimental settings; \cref{subsec:experiments_performance} analyses the performance of our solution in terms of execution time; \cref{subsec:experiments_quality} presents the quality of our heuristic algorithm in terms of the metrics in \cref{subsec:metrics}. +We experimentally evaluated the performance and quality of our methodology (heuristic algorithm in \cref{subsec:heuristics}), and compared it against the exhaustive approach in~\cref{sec:nphard}. In the following, +\cref{subsec:experiments_infrastructure} presents the simulator and testing infrastructure adopted in our experiments; +%, as well as the complete experimental settings; +\cref{subsec:experiments_performance} analyses the performance of our solution in terms of execution time; \cref{subsec:experiments_quality} presents the quality of our heuristic algorithm in generating of the best pipeline instance according to the metrics $M_J$ and $M_{JSD}$ in \cref{subsec:metrics}. \subsection{Testing Infrastructure and Experimental Settings}\label{subsec:experiments_infrastructure} -Our testing infrastructure is a Swift-based simulator of a service-based ecosystem, including service execution, comparison, and composition. -The simulator first defines the pipeline template as a sequence of vertexes in the range $3-7$. -We recall that alternative vertexes are modeled in different pipeline templates, -while parallel vertexes only add a fixed execution time that is negligible and do not affect the quality of our approach. -Each vertex is associated with a (set of) policy with transformations varying in two classes: -\begin{enumerate*}[label=\textit{\roman*})] - \item \average: data removal percentage within $[0.5,0.8]$. - \item \wide: data removal percentage within $[0.20,1]$. -\end{enumerate*} - -Upon setting the sliding window size, the simulator selects a subset of vertexes along with their corresponding candidate services. -It then generates all possible service combinations for the chosen vertexes. -For each combination, the simulator calculates a metric and selects the first service from the optimal combination before shifting the sliding window. -When the end of the vertex list is reached, or when the window size equals the vertex count, the simulator computes the optimal service combination for the remaining vertexes. - -An hash function is to simulate the natural interdependence between services. -This is particularly important when the removal of data by one service may impact another. -By assigning weights to the services using this function, the system aims to reflect the interconnected dynamics among the services. - -The simulator is used to assess the performance and quality of our sliding window heuristic in Section \ref{sec:heuristics} for the generation of the best pipeline instance (Section \ref{sec:instance}). +Our testing infrastructure is a Swift-based simulator of a service-based ecosystem, including service execution, comparison, and composition. The simulator first defines the pipeline template as a sequence of vertices, with $l$ the length of the pipeline template) and defines the size \windowsize\ of the sliding window, such that \windowsize$\leq$$l$. We recall that alternative vertices are modeled in different pipeline templates, while parallel vertices are not considered since they only add a fixed execution time that is negligible and do not affect the performance and quality of our approach. Each vertex is associated with a (set of) policy that applies a filtering transformation that remove a given percentage of data. +% Our testing infrastructure is a Swift-based simulator of a service-based ecosystem, including service execution, comparison, and composition. The simulator first defines the pipeline template as a sequence of vertices in the range 3$-$7 (the length $l$ of the pipeline template) and defines the size \windowsize\ of the sliding window, such that \windowsize$<$$l$. We recall that alternative vertices are modeled in different pipeline templates, while parallel vertices are not considered since they only add a fixed execution time that is negligible and do not affect the performance and quality of our approach. Each vertex is associated with a (set of) policy that applies a filtering transformation that either remove a percentage of data in $[0.5,0.8]$ (\average) or in $[0.20,1]$ (\wide). +% % \begin{enumerate*}[label=\textit{\roman*})] +% % \item \average: data removal percentage within $[0.5,0.8]$. +% % \item \wide: data removal percentage within +% % \end{enumerate*} + +The simulator then starts the instantiation process as shown in Figure~\ref{fig:execution_example}\hl{DA MODIFICARE COME SEQUENZA E NON ALTERNATIVA/PARALLELO}. At each step $i$, it selects the subset \{\vi{i},$\ldots$,$v_{\windowsize+i-1}$\} of vertices with their corresponding candidate services, and generates all possible service combinations. For each combination, the simulator calculates a given metric $M$ and selects the service that instantiates \vi{i} from the optimal combination according to $M$. The window is shifted step 1 (i.e., $i$=$i$+1) and the instantiation process restart. When the sliding window reach the end of the pipeline template, that is, $v_{\windowsize+i-1}$$=$$\vi{l}$, the simulator computes the optimal service combination and instantiates the remaining vertices with the corresponding services. + +We note that a hash function randomly simulates the natural interdependence between services, modeling a data removal on one service that might impact another one. \hl{LA SPECIFICHIAMO UN PO' MEGLIO?} %By assigning weights to the services using this function, the system aims to reflect the interconnected dynamics among the services. + +%The simulator is used to assess the performance and quality of our sliding window heuristic in Section \ref{sec:heuristics} for the generation of the best pipeline instance (Section \ref{sec:instance}). % Performance measures the heuristics execution time in different settings, while quality compares the results provided by our heuristics in terms of selected services with the optimal solution retrieved using the exhaustive approach. %We note that the exhaustive approach generates the best pipeline instance by executing all possible combinations of candidate services. %The emulator simplifies the execution of the service composition by removing the service selection phase, which is not relevant for the purpose of the experiment. Our experiments have been run on a virtual machine equipped with a Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz CPU and 32GB RAM. -Each experiment was repeated ten times and the results averaged to improve the reliability of the data. +Each experiment was repeated 10 times and the results averaged to improve the reliability of findings. \begin{figure}[!t] \centering @@ -105,24 +98,10 @@ \subsection{Perfomance}\label{subsec:experiments_performance} % \item % \end{itemize} % \subsection{Metriche/Euristiche} -We first calculated the execution time required by our exhaustive solution. -We incrementally varied the number of vertexes and the number of services per vertex. -The results of these evaluations are presented in \cref{fig:perf_exhaustive}. -As anticipated, the trend in execution times is exponential. \cref{fig:perf_exhaustive} displays the execution time plots, -clearly showing that as the number of vertexes increases, the execution time grows exponentially. -Execution times for up to 5 vertexes and 6 services were computed directly, -while the remaining data points were obtained through interpolation. -Subsequently, the logical extension of this empirical inquiry involves evaluating the execution time efficiency attributable to the implementation of the sliding window heuristic. - -We then evaluated our heuristics to quantify the execution time reduction achieved through the application of heuristics. -In this context, the number of vertexes and services per vertex was incrementally increased, -with the addition of a sliding window whose size was progressively enlarged in each experiment. -The outcomes are depicted in \cref{fig:perf_window}, and as expected, -we observed a marked reduction in execution times with the implementation of the sliding window heuristic. -This empirical evidence highlights the heuristic's ability to reduce computational demands, -an aspect that becomes increasingly pivotal as the problem's complexity grows. -The use of a logarithmic scale to illustrate the results linearizes the exponential growth associated with the exhaustive method, -offering a clear visual confirmation of the heuristic's efficiency in decreasing computational time. +We first measured the performance (execution time) of our exhaustive and heuristic solutions. To this aim, we varied the number of vertices from 2 to 5 (step 1) and the number of services per vertex from 2 to 6 (step 1). \cref{fig:perf_exhaustive} presents our results for the exhaustive solution, showing an exponential trend in the execution time. We can observe that, as the number of vertices increases, the execution time grows exponentially. Execution times for up to 5 vertices and 6 services were computed, while the remaining data points were obtained through interpolation.\hl{QUALI SONO INTERPOLATI? 5 E 6 SONO TUTTI I CASI.} +%Subsequently, the logical extension of this empirical inquiry involves evaluating the execution time efficiency attributable to the implementation of the sliding window heuristic. + +We then evaluated our heuristic and measured its performance (execution time) varying the window size \windowsize\ from X to Y (step 1). \cref{fig:perf_window} (log scale) presents our results for the heuristic solution. We can observe a substantial reduction in execution time, with the heuristic always able to produce an instance in less than \hl{TO ADD}.\hl{NON VEDO LA FIGURA.} \begin{figure}[htb!] \includegraphics[width=0.95\columnwidth]{graphs/exhaustive_performance.eps} \caption{Exhaustive execution time evaluation. The x-axis represents the number of services, while the y-axis represents the execution time in seconds. The execution time is expressed both in linear and logarithmic scales.} @@ -131,14 +110,21 @@ \subsection{Perfomance}\label{subsec:experiments_performance} \subsection{Quality}\label{subsec:experiments_quality} -We finally evaluated the quality of our heuristic comparing, where possible, its results with the optimal solution retrieved by executing the exhaustive approach. The latter is executed with window size equals to the number of vertexes in the pipeline template, and provides the best, among all possible, solutions. +We finally evaluated the quality of our heuristic with different \windowsize\ comparing, where possible, its results with the optimal solution retrieved by executing the exhaustive approach. %The latter is executed with window size equals to the number of vertices in the pipeline template, and provides the best, among all possible, solutions. +The quality of the heuristic was then calculated as the ratio between the value of metric $M$ retrieved by the heuristic and the one obtained by the exhaustive approach. -We run our experiments using the two settings in Section \cref{subsec:experiments_infrastructure}, namely, \average and \wide, and varied: \emph{i)} the number of vertexes in the pipeline template in [3,7], \emph{ii)} the window size in [1,$|$max$_v$$|$], where max$_v$ is the number of vertexes in the pipeline template, and \emph{iii)} the number of candidate services for each vertex in the pipeline template in [2, 7]. Values are calculated as the ratio of a given metric to the best metric, where the best metric is obtained through an exhaustive approach. It should be noted that, consequently, there is a constant upper boundary of 1 in the charts. +We run our experiments varying: \emph{i)} the length $l$ of the pipeline template in [3,7], that is, the number of vertices composed in a sequence, \emph{ii)} the window size \windowsize\ in [1,$l$], and \emph{iii)} the number of candidate services for each vertex in the pipeline template in [2, 7]. Each vertex is associated with a (set of) policy that applies a filtering transformation that either remove a percentage of data in $[0.5,0.8]$ (\average) or in $[0.20,1]$ (\wide). -\cref{fig:quality_window_average_perce,fig:quality_window_perce_wide} present our results the quantitive metrics in \cref{subsec:metrics} for the \wide and \average settings, respectively. -% + +\cref{fig:quality_window_average_perce,fig:quality_window_perce_wide} present our results with metric $M_J$ in \cref{subsec:metrics} for settings \wide and \average, respectively. +In general, we observe that the quality of our heuristic approach increases as the window size increases, providing a quality comparable to the exhaustive approach when the window size \windowsize\ approaches the length $l$ of the pipeline template. +When considering setting \wide, the greedy approach (\windowsize=1) provides good results on average (from X to Y), it shows substantial oscillations, for instance, varying the quality between X and Y for 3 vertices, X and Y for 4 vertices, X and Y for 5 vertices, X and Y for 6 vertices, X and Y for 7 vertices. This same trends emerge when the window size is less than $l$/2, while it starts approaching the optimum when the window size is higher than $l$/2. For instance, when \windowsize=$l$-1, the quality varis between X and Y for 3 vertices, X and Y for 4 vertices, X and Y for 5 vertices, X and Y for 6 vertices, X and Y for 7 vertices. +When considering setting \average, ... + +\hl{RIVEDI DA QUA COME SOPRA FINO A...} + In a configuration employing a \wide setting with three nodes, the average quality ratio for a window size of one is 0.93, with a standard deviation of 0.02; for a window size of two, it is 0.98 with a standard deviation of 0.01. When the number of nodes is increased to four, the average quality ratios observed are 0.88 for a window size of one (standard deviation = 0.04), 0.96 for a window size of two (standard deviation = 0.02), and 0.99 for a window size of three. Further increasing the node count to five yields average quality ratios of 0.76 for a window size of one (standard deviation = 0.09), 0.90 for a window size of two (standard deviation = 0.04), with further increments in window sizes of three and four maintaining standard deviations of 0.01. For six nodes, the average quality ratios are as follows: 0.78 for a window size of one (standard deviation = 0.09); 0.87 for a window size of two (standard deviation = 0.04); 0.95 for a window size of three (standard deviation = 0.02); 0.97 for a window size of four (standard deviation = 0.02); and 0.99 for a window size of five. Notably, the range of quality ratios (maximum-minimum) narrows as the window size increases, from 0.261 for a window size of one to 0.14 for a window size of two, and further to 0.02 for a window size of five. In a seven-node configuration, the average quality ratios are as follows: 0.73 for a window size of one (standard deviation = 0.08); 0.87 for a window size of two (standard deviation = 0.08); 0.95 for a window size of three (standard deviation = 0.02); 0.96 for a window size of four (standard deviation = 0.02); and 0.99 for both window sizes of five and six (standard deviation = 0.02 each). @@ -160,6 +146,8 @@ \subsection{Quality}\label{subsec:experiments_quality} For seven nodes, with window sizes extending from one to six, the quality ratios are 0.96 (standard deviation = 0.019), 0.98 (standard deviation = 0.009), 0.99 for both window sizes three and four (standard deviation = 0.005), and 1.00 for window sizes five and six (standard deviations = 0.003 and 0.000). +\hl{...FINO A QUI} + % We note that the benefit of an increasing window size can be appreciated with lower numbers, reaching a sort of saturation around the average length (e.g., window of length 6 with a 7-vertex pipeline) where the quality ratio overlaps. The only exception is for 6-vertex pipeline where the overapping starts with window size 2. However, this might be due to the specific setting and therefore does not generalize. % %Thus because the heuristic has more services to choose from and can find a better combination. % We also observe that, as the window size increase, the quality increase as well. This suggests that the heuristic performs better when it has a broader perspective of the data it is governing. diff --git a/macro.tex b/macro.tex index 26618ac..c56a635 100644 --- a/macro.tex +++ b/macro.tex @@ -57,7 +57,7 @@ \newcommand{\tChartFunction}{\ensuremath{G^{\myLambda,\myGamma}(\V,\E,\myLambda,\myGamma)}\xspace} % \newcommand{\iChartFunction}{\ensuremath{\fChartFunction,\myGamma{},\myLambda{}}} \newcommand{\iChartFunction}{\ensuremath{G'(\V',\E,\myLambda)}\xspace} - +\newcommand{\windowsize}{\ensuremath{|w|}} \newcommand{\pipeline}{Pipeline\xspace}