From 38acd662c5d5fa20f50c1308386e31e86f503b45 Mon Sep 17 00:00:00 2001 From: Claudio Ardagna Date: Tue, 7 May 2024 09:41:33 +0200 Subject: [PATCH] Experiment - Claudio --- experiment.tex | 36 +++++++++++++++++++----------------- main.tex | 3 ++- 2 files changed, 21 insertions(+), 18 deletions(-) diff --git a/experiment.tex b/experiment.tex index 0d39de6..541b71c 100644 --- a/experiment.tex +++ b/experiment.tex @@ -1,20 +1,23 @@ \section{Experiments}\label{sec:experiment} We experimentally evaluated the performance and quality of our methodology (heuristic algorithm in \cref{subsec:heuristics}), and compared it against the exhaustive approach in~\cref{sec:nphard}. In the following, -\cref{subsec:experiments_infrastructure} presents the simulator and testing infrastructure adopted in our experiments; +\cref{subsec:experiments_infrastructure} presents the simulator and experimental settings used in our experiments; %, as well as the complete experimental settings; -\cref{subsec:experiments_performance} analyses the performance of our solution in terms of execution time; \cref{subsec:experiments_quality} presents the quality of our heuristic algorithm in generating of the best pipeline instance according to the metrics $M_J$ and $M_{JSD}$ in \cref{subsec:metrics}. +\cref{subsec:experiments_performance} analyses the performance of our solution in terms of execution time; \cref{subsec:experiments_quality} discusses the quality of the best pipeline instance generated by our solution according to the metrics $M_J$ and $M_{JSD}$ in \cref{subsec:metrics}. \subsection{Testing Infrastructure and Experimental Settings}\label{subsec:experiments_infrastructure} -Our testing infrastructure is a Swift-based simulator of a service-based ecosystem, including service execution, comparison, and composition. The simulator first defines the pipeline template as a sequence of vertices, with $l$ the length of the pipeline template) and defines the size \windowsize\ of the sliding window, such that \windowsize$\leq$$l$. We recall that alternative vertices are modeled in different pipeline templates, while parallel vertices are not considered since they only add a fixed execution time that is negligible and do not affect the performance and quality of our approach. Each vertex is associated with a (set of) policy that applies a filtering transformation that remove a given percentage of data. +Our testing infrastructure is a Swift-based simulator of a service-based ecosystem, including service execution, comparison, and composition. The simulator first defines the pipeline template as a sequence of vertices, with $l$ the length of the pipeline template, and defines the size \windowsize\ of the sliding window, such that \windowsize$\leq$$l$. We recall that alternative vertices are modeled in different pipeline templates, while parallel vertices are not considered in our experiments since they only add a fixed execution time that is negligible and do not affect the performance and quality of our solution. Each vertex is associated with a (set of) policy that applies a filtering transformation that remove a given percentage of data. % Our testing infrastructure is a Swift-based simulator of a service-based ecosystem, including service execution, comparison, and composition. The simulator first defines the pipeline template as a sequence of vertices in the range 3$-$7 (the length $l$ of the pipeline template) and defines the size \windowsize\ of the sliding window, such that \windowsize$<$$l$. We recall that alternative vertices are modeled in different pipeline templates, while parallel vertices are not considered since they only add a fixed execution time that is negligible and do not affect the performance and quality of our approach. Each vertex is associated with a (set of) policy that applies a filtering transformation that either remove a percentage of data in $[0.5,0.8]$ (\average) or in $[0.20,1]$ (\wide). % % \begin{enumerate*}[label=\textit{\roman*})] % % \item \average: data removal percentage within $[0.5,0.8]$. % % \item \wide: data removal percentage within % % \end{enumerate*} -The simulator then starts the instantiation process as shown in Figure~\ref{fig:execution_example}. At each step $i$, it selects the subset \{\vi{i},$\ldots$,$v_{\windowsize+i-1}$\} of vertices with their corresponding candidate services, and generates all possible service combinations. For each combination, the simulator calculates a given metric $M$ and selects the service that instantiates \vi{i} from the optimal combination according to $M$. The window is shifted step 1 (i.e., $i$=$i$+1) and the instantiation process restart. When the sliding window reach the end of the pipeline template, that is, $v_{\windowsize+i-1}$$=$$\vi{l}$, the simulator computes the optimal service combination and instantiates the remaining vertices with the corresponding services. +The simulator then starts the instantiation process. At each step $i$, it selects the subset \{\vi{i},$\ldots$,$v_{\windowsize+i-1}$\} of vertices with their corresponding candidate services, and generates all possible service combinations. The simulator calculates quality $Q$ for all combinations and instantiates \vi{i} with service \sii{i} from the optimal combination with maximum $Q$. The window is shifted by 1 (i.e., $i$=$i$+1) and the instantiation process restarts. When the sliding window reaches the end of the pipeline template, that is, $v_{\windowsize+i-1}$$=$$\vi{l}$, the simulator computes the optimal service combination and instantiates the remaining vertices with the corresponding services. +Figure~\ref{fig:execution_example} shows an example of a simulator execution with $i$$=$2 and \windowsize$=$3. Subset \{\vi{2},\vi{3},\vi{4}\} is selected, all combinations generated, and corresponding quality $Q$ calculated. Optimal service combination \{\sii{11},\sii{22},\sii{31}\} is retrieved and \vii{2} in the pipeline instance instantiated with \sii{11}. + +The simulator defines dependencies between filtering transformations made by candidate services at consecutive vertices of the pipeline template. To this aim, it assigns a dependency rate to each service \si{i} modeling the amount of the filtering transformation done at \si{i} that overlaps the one at \si{i-1}. +For example, let us consider the pairs of services (\si{11},\si{21}) and (\si{11},\si{22}) with the following configurations: \emph{i)} service \si{11} introduces a filtering transformation that removes the 20\% of the dataset, \emph{ii)} service \si{21} introduces a filtering transformation that removes 10\% of the dataset and has dependency rate equal to 1, meaning that the filtering transformation made by \si{21} completely overlaps the one made by \si{11}, \emph{iii)} service \si{22} introduces a filtering transformation that removes 5\% of the dataset and has dependency rate equal to 0.5, meaning that the filtering transformation made by \si{22} overlaps half of the one made by \si{11}. Jaccard Metric $M_{J_{21}}$$=$0.8 at service \si{21}; $M_{J_{22}}$$=$0.75 at \si{22}, showing how dependencies can the pipeline quality and, in turn, the instantiation process. -It is reasonable to assume that within a service pipeline, any data modification made at an earlier stage could affect the performance of the service at the subsequent steps, making the services inderdependent. Consider, for example, the removal of data from the ''name`` feature; a service that relies on that column will be more significantly affected by its removal compared to a service that does not use it. During its execution, the simulator employs a specific combination of services as the seed to assign weights to the services. This reflects how changes in one service might influence others, as previously described. By assigning weights to the services using this approach, the system aims to reflect the interconnected dynamics among the services. %The simulator is used to assess the performance and quality of our sliding window heuristic in Section \ref{sec:heuristics} for the generation of the best pipeline instance (Section \ref{sec:instance}). % Performance measures the heuristics execution time in different settings, while quality compares the results provided by our heuristics in terms of selected services with the optimal solution retrieved using the exhaustive approach. @@ -120,7 +123,7 @@ \subsection{Perfomance}\label{subsec:experiments_performance} % \item % \end{itemize} % \subsection{Metriche/Euristiche} -We first measured the performance (execution time) of our exhaustive and heuristic solutions. To this aim, we varied the number of vertices from 2 to 5 (step 1) and the number of services per vertex from 2 to 6 (step 1). \cref{fig:perf_exhaustive} presents our results for the exhaustive solution, showing an exponential trend in the execution time. We can observe that, as the number of vertices increases, the execution time grows exponentially. Execution times for up to 5 vertices and 6 services were computed, while the remaining data points were obtained through interpolation.\hl{QUALI SONO INTERPOLATI? 5 E 6 SONO TUTTI I CASI.} +We first measured the performance (execution time) of our exhaustive and heuristic solutions. To this aim, we varied the number of vertices in the pipeline template from 2 to 5 (step 1) and the number of services per vertex from 2 to 6 (step 1). \cref{fig:perf_exhaustive} presents our results for the exhaustive solution, showing an exponential trend in the execution time. We can observe that, as the number of vertices increases, the execution time grows exponentially. Execution times for up to 5 vertices and 6 services were computed, while the remaining data points were obtained through interpolation.\hl{QUALI SONO INTERPOLATI? 5 E 6 SONO TUTTI I CASI.} %Subsequently, the logical extension of this empirical inquiry involves evaluating the execution time efficiency attributable to the implementation of the sliding window heuristic. We then evaluated our heuristic and measured its performance (execution time) varying the window size \windowsize\ from X to Y (step 1). \cref{fig:perf_window} (log scale) presents our results for the heuristic solution. We can observe a substantial reduction in execution time, with the heuristic always able to produce an instance in less than \hl{TO ADD}.\hl{NON VEDO LA FIGURA.} @@ -132,20 +135,20 @@ \subsection{Perfomance}\label{subsec:experiments_performance} \subsection{Quality}\label{subsec:experiments_quality} -We finally evaluated the quality of our heuristic with different \windowsize\ comparing, where possible, its results with the optimal solution retrieved by executing the exhaustive approach. %The latter is executed with window size equals to the number of vertices in the pipeline template, and provides the best, among all possible, solutions. -The quality of the heuristic was then calculated as the ratio between the value of metric $M$ retrieved by the heuristic and the one obtained by the exhaustive approach. +We finally evaluated the quality of our heuristic algorithm with different \windowsize\ comparing, where possible, its results with the optimal solution retrieved by executing the exhaustive approach. %The latter is executed with window size equals to the number of vertices in the pipeline template, and provides the best, among all possible, solutions. +The quality $Q$ of the heuristic has been normalized between 0 and 1 by dividing it by the quality $Q*$ retrieved by the exhaustive approach. We run our experiments varying: \emph{i)} the length $l$ of the pipeline template in [3,7], that is, the number of vertices composed in a sequence, \emph{ii)} the window size \windowsize\ in [1,$l$], and \emph{iii)} the number of candidate services for each vertex in the pipeline template in [2, 7]. Each vertex is associated with a (set of) policy that applies a filtering transformation that either remove a percentage of data in $[0.5,0.8]$ (\average) or in $[0.20,1]$ (\wide). - - -\cref{fig:quality_window_average_perce,fig:quality_window_perce_wide} present our results with metric $M_J$ in \cref{subsec:metrics} for settings \wide and \average, respectively. +\cref{fig:quality_window_average_perce,fig:quality_window_perce_wide} present our quality results using metric $M_J$ in \cref{subsec:metrics} for settings \wide and \average, respectively. In general, we observe that the quality of our heuristic approach increases as the window size increases, providing a quality comparable to the exhaustive approach when the window size \windowsize\ approaches the length $l$ of the pipeline template. -When considering setting \wide, the greedy approach (\windowsize=1) provides good results on average (from X to Y), it shows substantial oscillations, for instance, varying the quality between 0.882 and 0.970 for 3 vertices, 0.810 and 0.942 for 4 vertices, 0.580 and 0.853 for 5 vertices, 0.682 and 0.943 for 6 vertices, 0.596 and 0.821 for 7 vertices. This same trends emerge when the window size is less than $l$/2, while it starts approaching the optimum when the window size is higher than $l$/2. For instance, when \windowsize=$l$-1, the quality varis between 0.957 and 1.0 for 3 vertices, 0.982 and 1.0 for 4 vertices, 0.986 and 0.998 for 5 vertices, 0.977 and 1.0 for 6 vertices, 0.996 and 1.0 for 7 vertices. +When considering setting \wide, the greedy approach (\windowsize=1) provides good results on average (from X to Y), while showing substantial quality oscillations in specific runs: between 0.882 and 0.970 for 3 vertices, 0.810 and 0.942 for 4 vertices, 0.580 and 0.853 for 5 vertices, 0.682 and 0.943 for 6 vertices, 0.596 and 0.821 for 7 vertices. This same trend emerges when the window size is $<$$l$/2, while it starts approaching the optimum when the window size is $\geq$$l$/2. For instance, when \windowsize=$l$-1, the quality varies between 0.957 and 1.0 for 3 vertices, 0.982 and 1.0 for 4 vertices, 0.986 and 0.998 for 5 vertices, 0.977 and 1.0 for 6 vertices, 0.996 and 1.0 for 7 vertices. -When considering setting \average, the greedy approach (\windowsize=1) provides good results on average (from X to Y), it shows substantial oscillations, for instance, varying the quality between 0.927 and 0.978 for 3 vertices, 0.903 and 0.962 for 4 vertices, 0.840 and 0.915 for 5 vertices, 0.815 and 0.934 for 6 vertices, 0.721 and 0.935 for 7 vertices. This same trends emerge when the window size is less than $l$/2, while it starts approaching the optimum when the window size is higher than $l$/2. For instance, when \windowsize=$l$-1, the quality varis between 0.980 and 1.0 for 3 vertices, 0.978 and 1.0 for 4 vertices, 0.954 and 1 for 5 vertices, 0.987 and 1.0 for 6 vertices, 0.990 and 1.0 for 7 vertices. +When considering setting \average, the heuristic algorithm still provides good results, limiting the quality oscillations observed for setting \wide\ and approaching the quality of the exhaustive also for lower window sizes. The greedy approach (\windowsize=1) provides good results on average (from X to Y), as well as in specific runs: between 0.927 and 0.978 for 3 vertices, 0.903 and 0.962 for 4 vertices, 0.840 and 0.915 for 5 vertices, 0.815 and 0.934 for 6 vertices, 0.721 and 0.935 for 7 vertices. +%This same trend emerges when the window size is less than $l$/2, while it starts approaching the optimum when the window size is higher than $l$/2. For instance, +When \windowsize=$l$-1, the quality varies between 0.980 and 1.0 for 3 vertices, 0.978 and 1.0 for 4 vertices, 0.954 and 1 for 5 vertices, 0.987 and 1.0 for 6 vertices, 0.990 and 1.0 for 7 vertices. % \hl{RIVEDI DA QUA COME SOPRA FINO A...} % In a configuration employing a \wide setting with three nodes, the average quality ratio for a window size of one is 0.93, with a standard deviation of 0.02; for a window size of two, it is 0.98 with a standard deviation of 0.01. When the number of nodes is increased to four, the average quality ratios observed are 0.88 for a window size of one (standard deviation = 0.04), 0.96 for a window size of two (standard deviation = 0.02), and 0.99 for a window size of three. Further increasing the node count to five yields average quality ratios of 0.76 for a window size of one (standard deviation = 0.09), 0.90 for a window size of two (standard deviation = 0.04), with further increments in window sizes of three and four maintaining standard deviations of 0.01. @@ -155,7 +158,8 @@ \subsection{Quality}\label{subsec:experiments_quality} % In an \average setting with three nodes, the average quality ratios are 0.95 for a window size of one, with a standard deviation of 0.017, and 0.99 for a window size of two, with a standard deviation of 0.006. Expanding to four nodes, the quality ratios observed are 0.93 for a window size of one (standard deviation = 0.019), 0.97 for a window size of two (standard deviation = 0.006), and 0.99 for a window size of three (standard deviation = 0.007). Increasing the node count to five yields average quality ratios of 0.88 for a window size of one (standard deviation = 0.02), 0.93 for a window size of two (standard deviation = 0.03), 0.97 for a window size of three (standard deviation = 0.015), and 0.98 for a window size of four (standard deviation = 0.016). % For six nodes, the quality ratios are as follows: 0.87 for a window size of one (standard deviation = 0.047), 0.92 for a window size of two (standard deviation = 0.058), 0.96 for a window size of three (standard deviation = 0.021), 0.97 for a window size of four (standard deviation = 0.014), and 0.99 for a window size of five (standard deviation = 0.004). For seven nodes, the respective quality ratios are 0.83 for a window size of one (standard deviation = 0.068), 0.92 for a window size of two (standard deviation = 0.030), 0.98 for both window sizes of three and four (standard deviation = 0.007 and 0.017), and 0.99 for window sizes of five and six (standard deviation = 0.006 and 0.004). -\cref{fig:quality_window_wide_qualitative,fig:quality_window_average_qualitative} display the outcomes assessed metric $M_{JSD}$ in \cref{subsec:metrics} for settings \wide and \average, respectively. +\cref{fig:quality_window_wide_qualitative,fig:quality_window_average_qualitative} present our quality results using metric $M_{JSD}$ in \cref{subsec:metrics} for settings \wide and \average, respectively. +In general, \hl{ANTONGIACOMO} When considering setting \wide, the greedy approach (\windowsize=1) provides good results on average (from X to Y), it shows substantial oscillations, for instance, varying the quality between 0.954 and 0.993 for 3 vertices, 0.939 and 0.993 for 4 vertices, 0.899 and 0.960 for 5 vertices, 0.878 and 0.949 for 6 vertices, 0.870 and 0.914 for 7 vertices. This same trends emerge when the window size is less than $l$/2, while it starts approaching the optimum when the window size is higher than $l$/2. For instance, when \windowsize=$l$-1, the quality varis between 0.991 and 1,0 for 3 vertices, 0.989 and 1.0 for 4 vertices, 0.994 and 1.0 for 5 vertices, 0.993 and 1.0 for 6 vertices, 0.997 and 1.0 for 7 vertices. @@ -181,10 +185,8 @@ \subsection{Quality}\label{subsec:experiments_quality} % It's worth noting that lower window sizes are more unstable, with the quality ratio varying significantly between different configuration while higher window sizes tend to stabilize the quality ratio across different configuration. -Finally, the data suggest that while larger window sizes generally lead to better performance, there might exist a point where the balance between window size and performance is optimized. Beyond this point, the incremental gains in metric values may not justify the additional computational resources or the complexity introduced by larger windows. -It's worth noting that lower window sizes are more unstable, with the quality ratio varying significantly between different configuration while higher window sizes tend to stabilize the quality ratio across different configuration. +Our results suggest that the proposed heuristic well approximates the results obtained via an exhaustive approach. While larger window sizes generally lead to better performance, there exists a point where the balance between window size and performance is optimized. Beyond this point, the incremental gains in metric values may not justify the additional computational burden or complexity introduced by larger windows. It is worth noting that lower window sizes are more unstable, especially with setting \wide, meaning that the quality varies significantly among different configurations. This effect stabilizes with higher window sizes (e.g., \windowsize$\geq$$l$/2). -The proposed heuristics show ability to approximate the results obtained via an exhaustive approach. However, to comprehensively understand the influence of dataset selection on the metrics' performance and ensure their robustness across diverse scenarios, a dedicated investigation is essential. This further research will validate the preliminary findings and provide a deeper insight into the applicability of the heuristics in various contexts. \begin{figure*}[!htb] \centering \begin{subfigure}{0.33\textwidth} diff --git a/main.tex b/main.tex index 59056c2..7907c76 100644 --- a/main.tex +++ b/main.tex @@ -51,7 +51,7 @@ \begin{document} -\title{TBA} +\title{Balancing Data Quality and Protection in Distributed Big Data Pipelines} \author{Marco Anisetti~\IEEEmembership{Senior Member,~IEEE,}, Claudio A. Ardagna~\IEEEmembership{Senior Member,~IEEE,} Chiara Braghin, Ernesto Damiani~\IEEEmembership{Senior Member,~IEEE,}, Antongiacomo Polimeno % <-this % stops a space \IEEEcompsocitemizethanks{\IEEEcompsocthanksitem M. Anisetti, C.A. Ardagna, E. Damiani, are with the Dipartimento di Informatica, Universit\`a degli Studi di Milano, Milano, @@ -110,6 +110,7 @@ \input{related} \section{Conclusions}\label{sec:conclusions} +To comprehensively understand the influence of dataset selection on the retrieved quality and ensure heuristic robustness across diverse scenarios, a dedicated investigation is left for our future work. This further research will validate the findings in this paper and provide a deeper insight into the applicability of the heuristics in various contexts. \clearpage %\bibliographystyle{spbasic} % basic style, author-year citations