diff --git a/experiment.tex b/experiment.tex index bd9d216..b0c43a9 100644 --- a/experiment.tex +++ b/experiment.tex @@ -135,38 +135,32 @@ \subsection{Perfomance}\label{subsec:experiments_performance} \subsection{Quality}\label{subsec:experiments_quality} We finally evaluated the quality of our heuristic comparing, where possible, its results with the optimal solution retrieved by executing the exhaustive approach. The latter is executed with window size equals to the number of vertexes in the pipeline template, and provides the best, among all possible, solutions. -We run our experiments using the two settings in Section \cref{subsec:experiments_infrastructure}, namely, \average and \wide, and varied: \emph{i)} the number of vertexes in the pipeline template in [3,7], \emph{ii)} the window size in [1,$|$max$_v$$|$], where max$_v$ is the number of vertexes in the pipeline template, and \emph{iii)} the number of candidate services for each vertex in the pipeline template in [2, 7]. +We run our experiments using the two settings in Section \cref{subsec:experiments_infrastructure}, namely, \average and \wide, and varied: \emph{i)} the number of vertexes in the pipeline template in [3,7], \emph{ii)} the window size in [1,$|$max$_v$$|$], where max$_v$ is the number of vertexes in the pipeline template, and \emph{iii)} the number of candidate services for each vertex in the pipeline template in [2, 7]. Values are calculated as the ratio of a given metric to the best metric, where the best metric is obtained through an exhaustive approach. It should be noted that, consequently, there is a constant upper boundary of 1 in the charts. \cref{fig:quality_window_average_perce,fig:quality_window_perce_wide} presents our results the quantitive metrics in \cref{subsec:metrics} for the \wide and \average settings, respectively. -Value are normalized to the optimal solution retrieved by the exhaustive approach. + % -When a \wide setting is employed, the average quality ratio in a configuration with three nodes for window of size one is 0.93, with a standard deviation of 0.02; it is 0.98 with a standard deviation of 0.01, when a window of size two is used. Increasing the number of nodes to four results in an average quality ratio of 0.88 for a window size of one with a standard deviation of 0.04, 0.96 with a standard deviation of 0.02,when a window of size two is used, 0.99 for a window size of three. Increasing to five nodes, the average quality ratio is 0.76 for a window size of one with a standard deviation of 0.09, 0.90 for a window of size two. Size two exhibited a standard deviation of 0.04, size three a standard deviation of 0.01, size 4 a standard deviation of 0.01. -For 6 nodes, the average quality ratio was 0.78 for a window of size one with a standard deviation of 0.09; 0.87 for a window of size two with a standard deviation of 0.04; 0.95 for a window of size three with a standard deviation of 0.02; 0.97 for a window of size four with a standard deviation of 0.02; and 0.99 for a window of size five.In a configuration of 7 nodes, the average quality ratio was 0.73 for a window sizeof one with a standard deviation of 0.08; it is 0.87 for a window size of size two, with a standard deviation of 0.08. window size of 3 nodes exhibited a quality ratio of 0.95 with a standard deviation of 0.02, it is 0.96 for a window size of 4 nodes, with a standard deviation of 0.02. The mean value of the quality ratio is 0.99 for a window size of 5 and 6 nodes, with a standard deviation of 0.02. +In a configuration employing a \wide setting with three nodes, the average quality ratio for a window size of one is 0.93, with a standard deviation of 0.02; for a window size of two, it is 0.98 with a standard deviation of 0.01. When the number of nodes is increased to four, the average quality ratios observed are 0.88 for a window size of one (standard deviation = 0.04), 0.96 for a window size of two (standard deviation = 0.02), and 0.99 for a window size of three. Further increasing the node count to five yields average quality ratios of 0.76 for a window size of one (standard deviation = 0.09), 0.90 for a window size of two (standard deviation = 0.04), with further increments in window sizes of three and four maintaining standard deviations of 0.01. +For six nodes, the average quality ratios are as follows: 0.78 for a window size of one (standard deviation = 0.09); 0.87 for a window size of two (standard deviation = 0.04); 0.95 for a window size of three (standard deviation = 0.02); 0.97 for a window size of four (standard deviation = 0.02); and 0.99 for a window size of five. Notably, the range of quality ratios (maximum-minimum) narrows as the window size increases, from 0.261 for a window size of one to 0.14 for a window size of two, and further to 0.02 for a window size of five. +In a seven-node configuration, the average quality ratios are as follows: 0.73 for a window size of one (standard deviation = 0.08); 0.87 for a window size of two (standard deviation = 0.08); 0.95 for a window size of three (standard deviation = 0.02); 0.96 for a window size of four (standard deviation = 0.02); and 0.99 for both window sizes of five and six (standard deviation = 0.02 each). +In an \average setting with three nodes, the average quality ratios are 0.95 for a window size of one, with a standard deviation of 0.017, and 0.99 for a window size of two, with a standard deviation of 0.006. Expanding to four nodes, the quality ratios observed are 0.93 for a window size of one (standard deviation = 0.019), 0.97 for a window size of two (standard deviation = 0.006), and 0.99 for a window size of three (standard deviation = 0.007). Increasing the node count to five yields average quality ratios of 0.88 for a window size of one (standard deviation = 0.02), 0.93 for a window size of two (standard deviation = 0.03), 0.97 for a window size of three (standard deviation = 0.015), and 0.98 for a window size of four (standard deviation = 0.016). +For six nodes, the quality ratios are as follows: 0.87 for a window size of one (standard deviation = 0.047), 0.92 for a window size of two (standard deviation = 0.058), 0.96 for a window size of three (standard deviation = 0.021), 0.97 for a window size of four (standard deviation = 0.014), and 0.99 for a window size of five (standard deviation = 0.004). For seven nodes, the respective quality ratios are 0.83 for a window size of one (standard deviation = 0.068), 0.92 for a window size of two (standard deviation = 0.030), 0.98 for both window sizes of three and four (standard deviation = 0.007 and 0.017), and 0.99 for window sizes of five and six (standard deviation = 0.006 and 0.004). -When a \average setting is employed, in a configuration with three nodes and using a window size ranging from 1 to 2 the quality ratio is respectively: 0.95 and 0.99 with a standard deviation respectively of 0.017 and 0.006. For a configuration with four nodes and using a window size ranging from 1 to 3 the quality ratio is respectively: 0.93, 0.97 and 0.99 with a standard deviation respectively of 0.019, 0.006 and 0.007. For a configuration with five nodes and using a window size ranging from 1 to 4 the quality ratio is respectively: -0.88, 0.93, 0.97 and 0.98 with a standard deviation respectively of 0.02, 0.03, 0.015 and 0.016. For a configuration with six nodes and using a window size ranging from 1 to 5 the quality ratio is respectively: -0.87, 0.92, 0.96, 0.97 and 0.99 with a standard deviation respectively of 0.047, 0.058, 0.021, 0.014 and 0.004. For a configuration with seven nodes and using a window size ranging from 1 to 6 the quality ratio is respectively: -0.83, 0.92, 0.98, 0.98, 0.99 and 0.99 with a standard deviation respectively of 0.068, 0.030, 0.007, 0.017, 0.006 and 0.004. +\cref{fig:quality_window_wide_qualitative,fig:quality_window_average_qualitative} presents our results the qualitative metric in \cref{subsec:metrics} for the \wide and \average settings, respectively. +In a \wide setting with three nodes, average quality ratios were observed as 0.98 for a window size of one, with a standard deviation of 0.014, and 0.998 for a window size of two, with a standard deviation of 0.005. Increasing the node count to four, the quality ratios are 0.97 for a window size of one (standard deviation = 0.019), 0.99 for a window size of two (standard deviation = 0.004), and 0.996 for a window size of three (standard deviation = 0.004). -\cref{fig:quality_window_wide_qualitative,fig:quality_window_average_qualitative} presents our results the qualitative metric in \cref{subsec:metrics} for the \wide and \average settings, respectively. -Value are normalized to the optimal solution retrieved by the exhaustive approach. +For five nodes, the quality ratios extend as follows: 0.93 for a window size of one (standard deviation = 0.022), 0.97 for a window size of two (standard deviation = 0.015), 0.993 for a window size of three (standard deviation = 0.006), and 0.998 for a window size of four (standard deviation = 0.003). In a configuration with six nodes, the quality ratios are 0.92 for a window size of one (standard deviation = 0.028), 0.97 for a window size of two (standard deviation = 0.018), 0.995 for a window size of three (standard deviation = 0.004), 0.996 for a window size of four (standard deviation = 0.005), and 0.998 for a window size of five (standard deviation = 0.003). -When a \wide setting is employed, in a configuration with three nodes and using a window size ranging from 1 to 2 the quality ratio is respectively: 0.98 and 0.998 with a standard deviation respectively of 0.014 and 0.005. For a configuration with four nodes and using a window size ranging from 1 to 3 the quality ratio is respectively: 0.97, 0.99, and 0.996 with a standard deviation respectively of 0.019, 0.004, and 0.004. For a configuration with five nodes and using a window size ranging from 1 to 4 the quality ratio is respectively: -0.93, 0.97, 0.993, and 0.998 with a standard deviation respectively of 0.022, 0.015, 0.006, and 0.003. For a configuration with six nodes and using a window size ranging from 1 to 5 the quality ratio is respectively: -0.92, 0.97, 0.995, 0.996, and 0.998 with a standard deviation respectively of 0.028, 0.018, 0.004, 0.005, and 0.003. For a configuration with seven nodes and using a window size ranging from 1 to 6 the quality ratio is respectively: -0.90, 0.96, 0.97, 0.994, 0.997, and 0.999 with a standard deviation respectively of 0.016, 0.019, 0.010, 0.005, 0.003, and 0.003. +Finally, with seven nodes, the quality ratios are 0.90 for a window size of one (standard deviation = 0.016), 0.96 for a window size of two (standard deviation = 0.019), 0.97 for a window size of three (standard deviation = 0.010), 0.994 for a window size of four (standard deviation = 0.005), 0.997 for a window size of five (standard deviation = 0.003), and 0.999 for a window size of six (standard deviation = 0.003). +In an \average setting with three nodes, the average quality ratios are 0.99 for a window size of one, with a standard deviation of 0.003, and 1.00 for a window size of two, with a standard deviation of 0.000. When the configuration includes four nodes and window sizes from one to three, the quality ratios are 0.98 (standard deviation = 0.010), 0.99 (standard deviation = 0.003), and 1.00 (standard deviation = 0.003). -When \average setting is employed, in a configuration with three nodes and using a window size ranging from 1 to 2 the quality ratio is respectively: 0.99 and 1.00 with a standard deviation respectively of 0.003, and 0.000. -For a configuration with four nodes and using a window size ranging from 1 to 3 the quality ratio is respectively: -0.98, 0.99, and 1.00 with a standard deviation respectively of 0.010, 0.003 and 0.003. For a configuration with five nodes and using a window size ranging from 1 to 4 the quality ratio is respectively: -0.98, 0.99, 1.00 and 1.00 with a standard deviation respectively of 0.004, 0.003, 0.004, and 0.000. For a configuration with six nodes and using a window size ranging from 1 to 5 the quality ratio is respectively: -0.97, 0.98, 1.00, 1.00 and 1.00 with a standard deviation respectively of 0.006, 0.003, 0.000, 0.003 and 0.003. For a configuration with seven nodes and using a window size ranging from 1 to 6 the quality ratio is respectively: -0.96, 0.98, 0.99, 0.99, 1.00 and 1.00 with a standard deviation respectively of 0.019, 0.009, 0.005, 0.005, 0.003 and 0.000. +Expanding to five nodes, the quality ratios are 0.98 for a window size of one (standard deviation = 0.004), 0.99 for a window size of two (standard deviation = 0.003), 1.00 for both window sizes three and four (standard deviations = 0.004 and 0.000, respectively). In a configuration with six nodes and window sizes ranging from one to five, the quality ratios are 0.97 (standard deviation = 0.006), 0.98 (standard deviation = 0.003), and 1.00 for window sizes three through five (standard deviations = 0.000, 0.003, and 0.003). +For seven nodes, with window sizes extending from one to six, the quality ratios are 0.96 (standard deviation = 0.019), 0.98 (standard deviation = 0.009), 0.99 for both window sizes three and four (standard deviation = 0.005), and 1.00 for window sizes five and six (standard deviations = 0.003 and 0.000). % We note that the benefit of an increasing window size can be appreciated with lower numbers, reaching a sort of saturation around the average length (e.g., window of length 6 with a 7-vertex pipeline) where the quality ratio overlaps. The only exception is for 6-vertex pipeline where the overapping starts with window size 2. However, this might be due to the specific setting and therefore does not generalize. % %Thus because the heuristic has more services to choose from and can find a better combination.