-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
10 changed files
with
92 additions
and
180 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,107 +1,57 @@ | ||
| # Extend Metrics | ||
|
|
||
| To implement a new metric, create a new class that extends the abstract class `AbstractMetric`: | ||
| To implement a new metric, create a new class that extends the abstract class `Metric`: | ||
|
|
||
| ```java | ||
| package org.benchmark.metric; | ||
|
|
||
| @Shorthand("MyMetric") | ||
| public class MyMetric extends AbstractMetric{ | ||
| public class MyMetric extends Metric { | ||
|
|
||
| @Override | ||
| public void receiveData(Properties p) { | ||
| // ... | ||
| } | ||
|
|
||
| @Override | ||
| public void close() { | ||
| callbackClose(); | ||
| super.close(); | ||
|
|
||
| } | ||
|
|
||
| protected void callbackClose() { | ||
| // your close method | ||
| } | ||
| public MyMetric() { | ||
| super("name", "abbreviation", "description"); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Receive Data | ||
|
|
||
| This method will receive all the results during the benchmark. | ||
|
|
||
| You'll receive a few values regarding each query execution. Those values include the amount of time the execution took, if it succeeded, and if not, the reason why it failed, which can be either a timeout, a wrong HTTP Code or an unknown error. | ||
| Further on you also receive the result size of the query. | ||
|
|
||
| If your metric is a single value metric, you can use the `processData` method, which will automatically add each value together. | ||
| However, if your metric is query specific, you can use the `addDataToContainter` method. (Look at the [QPSMetric](https://github.com/dice-group/IGUANA/blob/master/iguana.resultprocessor/src/main/java/org/aksw/iguana/rp/metrics/impl/QPSMetric.java)) | ||
| You can then choose if the metric is supposed to be calculated for each Query, Worker | ||
| or Task by implementing the appropriate interfaces: `QueryMetric`, `WorkerMetric`, `TaskMetric`. | ||
|
|
||
| Be aware that both methods will save the results for each used worker. This allows the calculation of the overall metric, as well as the metric for each worker itself. | ||
| You can also choose to implement the `ModelWritingMetric` interface, if you want your | ||
| metric to create a special RDF model, that you want to be added to the result model. | ||
|
|
||
| We will stick to the single-value metric for now. | ||
|
|
||
|
|
||
| The following shows an example, that retrieves every possible value and saves the time and success: | ||
| The following gives you an examples on how to work with the `data` parameter: | ||
|
|
||
| ```java | ||
| @Override | ||
| public void receiveData(Properties p) { | ||
|
|
||
| double time = Double.parseDouble(p.get(COMMON.RECEIVE_DATA_TIME).toString()); | ||
| long tmpSuccess = Long.parseLong(p.get(COMMON.RECEIVE_DATA_SUCCESS).toString()); | ||
| long success = (tmpSuccess > 0) ? 1 : 0; | ||
| long failure = (success == 1) ? 0 : 1; | ||
| long timeout = (tmpSuccess == COMMON.QUERY_SOCKET_TIMEOUT) ? 1 : 0; | ||
| long unknown = (tmpSuccess == COMMON.QUERY_UNKNOWN_EXCEPTION) ? 1 : 0; | ||
| long wrongCode = (tmpSuccess == COMMON.QUERY_HTTP_FAILURE) ? 1 : 0; | ||
|
|
||
| if(p.containsKey(COMMON.RECEIVE_DATA_SIZE)) { | ||
| size = Long.parseLong(p.get(COMMON.RECEIVE_DATA_SIZE).toString()); | ||
| @Override | ||
| public Number calculateTaskMetric(StresstestMetadata task, List<QueryExecutionStats>[][] data) { | ||
| for (WorkerMetadata worker : task.workers()) { | ||
| for (int i = 0; i < worker.noOfQueries(); i++) { | ||
| // This list contains every query execution statistics of one query | ||
| // from the current worker | ||
| List<QueryExecutionStats> execs = data[worker.workerID()][i]; | ||
| } | ||
| } | ||
| return BigInteger.ZERO; | ||
| } | ||
|
|
||
| Properties results = new Properties(); | ||
| results.put(TOTAL_TIME, time); | ||
| results.put(TOTAL_SUCCESS, success); | ||
|
|
||
| Properties extra = getExtraMeta(p); | ||
| processData(extra, results); | ||
| } | ||
| ``` | ||
|
|
||
| ## Close | ||
|
|
||
| In this method you should calculate your metric and send the results. | ||
| An example: | ||
|
|
||
| ```java | ||
| protected void callbackClose() { | ||
| // create a model that contains the results | ||
| Model m = ModelFactory.createDefaultModel(); | ||
|
|
||
| Property property = getMetricProperty(); | ||
| Double sum = 0.0; | ||
|
|
||
| // Go over each worker and add metric results to model | ||
| for(Properties key : dataContainer.keySet()){ | ||
| Double totalTime = (Double) dataContainer.get(key).get(TOTAL_TIME); | ||
| Integer success = (Integer) dataContainer.get(key).get(TOTAL_SUCCESS); | ||
|
|
||
| Double noOfQueriesPerHour = hourInMS * success * 1.0 / totalTime; | ||
| sum += noOfQueriesPerHour; | ||
| Resource subject = getSubject(key); | ||
|
|
||
| m.add(getConnectingStatement(subject)); | ||
| m.add(subject, property, ResourceFactory.createTypedLiteral(noOfQueriesPerHour)); | ||
| @Override | ||
| public Number calculateWorkerMetric(WorkerMetadata worker, List<QueryExecutionStats>[] data) { | ||
| for (int i = 0; i < worker.noOfQueries(); i++) { | ||
| // This list contains every query execution statistics of one query | ||
| // from the given worker | ||
| List<QueryExecutionStats> execs = data[i]; | ||
| } | ||
| return BigInteger.ZERO; | ||
| } | ||
|
|
||
| // Add overall metric to model | ||
| m.add(getTaskResource(), property, ResourceFactory.createTypedLiteral(sum)); | ||
|
|
||
| // Send data to storage | ||
| sendData(m); | ||
| } | ||
| ``` | ||
|
|
||
| ## Constructor | ||
|
|
||
| The constructor parameters are provided the same way as for the tasks. Thus, simply look at the [Extend Task](../extend-task) page. | ||
| @Override | ||
| @Nonnull | ||
| public Model createMetricModel(StresstestMetadata task, Map<String, List<QueryExecutionStats>> data) { | ||
| for (String queryID : task.queryIDS()) { | ||
| // This list contains every query execution statistics of one query from | ||
| // every worker that executed this querys | ||
| List<QueryExecutionStats> execs = data.get(queryID); | ||
| } | ||
| } | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,47 +1,23 @@ | ||
| # Extend Result Storages | ||
|
|
||
| If you want to use a different storage other than RDF, you can implement a different storage solution. | ||
|
|
||
| The current implementation of Iguana is highly optimized for RDF, thus we recommend you to work on top of the `TripleBasedStorage` class: | ||
| If you want to use a different storage other than RDF, you can implement a different storage solution. | ||
|
|
||
| ```java | ||
| package org.benchmark.storage; | ||
|
|
||
| @Shorthand("MyStorage") | ||
| public class MyStorage extends TripleBasedStorage { | ||
|
|
||
| @Override | ||
| public void commit() { | ||
|
|
||
| } | ||
|
|
||
| @Override | ||
| public String toString(){ | ||
| return this.getClass().getSimpleName(); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Commit | ||
| public class MyStorage implements Storage { | ||
|
|
||
| This method should take all the current results, store them, and remove them from the memory. | ||
|
|
||
| You can access the results at the Jena Model `this.metricResults`. | ||
|
|
||
| For example: | ||
|
|
||
| ```java | ||
| @Override | ||
| public void commit() { | ||
| try (OutputStream os = new FileOutputStream(file.toString(), true)) { | ||
| RDFDataMgr.write(os, metricResults, RDFFormat.NTRIPLES); | ||
| metricResults.removeAll(); | ||
| } catch (IOException e) { | ||
| LOGGER.error("Could not commit to NTFileStorage.", e); | ||
| @Override | ||
| public void storeResults(Model m) { | ||
| // method for storing model | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The method `storeResults` will be called at the end of the task. The model from | ||
| the parameter contains the final result model for that task. | ||
|
|
||
| ## Constructor | ||
|
|
||
| The constructor parameters are provided the same way as for the tasks. Thus, simply look at the [Extend Task](../extend-task) page. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,53 +1,36 @@ | ||
| # Implemented Metrics | ||
| ## Global Metrics | ||
| The following metrics are calculated for each task and worker: | ||
|
|
||
| Every metric will be calculated globally (for one Experiment Task) and locally (for each Worker). | ||
| Hence, you are able to analyze the metrics of the whole benchmark or only of each worker. | ||
| | Metric | Description | | ||
| |---------|------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | NoQ | The number of successfully executed Queries. | | ||
| | QMPH | The number of successfully executed Query Mixes (amount of queries inside a query source) Per Hour. | | ||
| | NoQPH | The number of successfully executed Queries Per Hour. | | ||
| | AvgQPS | The average of the QPS metric value between all queries. | | ||
| | PAvgQPS | The average of the PQPS metric value between all queries. For this metric you have to set a value for the penalty (in milliseconds) (example below). | | ||
|
|
||
| ## NoQ | ||
|
|
||
| The number of successfully executed Queries | ||
|
|
||
| ## QMPH | ||
| ## Query Metrics | ||
| The following metrics are calculated for each query. | ||
|
|
||
| The number of executed Query Mixes Per Hour | ||
| | Metric | Description | | ||
| |-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | QPS | The number of successfully executed Queries per second. | | ||
| | PQPS | The number of executed Queries per second. Each failed query execution will receive a given time penalty instead of its execution duration. | | ||
| | EachQuery | Stores for each query executions its statistics. This includes the execution duration, response code, result size and a boolean value if the execution was successful. | | ||
| | AES | This metric aggregates the values of each query execution. | | ||
|
|
||
| ## NoQPH | ||
|
|
||
| The number of successfully executed Queries Per Hour | ||
|
|
||
| ## QPS | ||
|
|
||
| For each query, the `queries per second`, the `total time` in milliseconds (summed up time of each execution), the number of `succeeded` and `failed` executions, and the `result size` will be saved. | ||
|
|
||
| Additionally, Iguana will try to tell you how many times a query has failed and for what reason (`timeout`, `wrong return code`, e.g. 400, or `unknown`). | ||
|
|
||
| Further on the QPS metric provides a penalized QPS metric that penalizes queries that fail. | ||
| Some systems just return an error code, if they can't resolve a query, thus they can have a very high score, even though they were only able to handle a few queries. That would be rather unfair to the compared systems, therefore we introduced the penalty QPS. It is calculated the same as the QPS score, but for each failed query it uses the penalty instead of the actual time the failed query took. | ||
|
|
||
| The default penalty is set to the `timeOut` value of the task. However, you can override it as follows: | ||
|
|
||
| ### Configuration for PAvgQPS and QPS | ||
| An example for the configuration of both: | ||
| ```yaml | ||
| metrics: | ||
| - className: "QPS" | ||
| metrics: | ||
| - className: "PAvgQPS" | ||
| configuration: | ||
| #in MS | ||
| penalty: 10000 | ||
| ``` | ||
| ## AvgQPS | ||
| The average of all queries per second. | ||
| It also adds a penalizedAvgQPS metric. The default penalty is set to the `timeOut` value of the task, but it can be overwritten as follows: | ||
|
|
||
| ```yaml | ||
| metrics: | ||
| - className: "AvgQPS" | ||
| - className: "PQPS" | ||
| configuration: | ||
| # in ms | ||
| penalty: 10000 | ||
| ``` | ||
|
|
||
| ## EachQuery | ||
|
|
||
| Will save every query execution. (Experimental) | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters