Refurbishment #2

README

@@ -1 +1,21 @@
-Look in the "tpc" directory for source code; or see https://github.com/eishay/jvm-serializers/wiki for current results.
+(RuedigerMoeller =>)
+
+My current results:
+
+https://github.com/RuedigerMoeller/fast-serialization/wiki/TestPage
+
+My modifications:
+
+* The original project actually used the MINIMUM runtime for each benchmark thereby completely excluding GC related effects. 
+I changed it to measure the average results of all runs (excluding warmup)
+
+* I start a new VM for each benchmark run instead of running all in a single VM
+
+* I limit test runs by time not by number of iterations
+
+* I somewhat categorized serializers (full graph, generic, manually adoption, JSON/Binary, crossplatform ..). A lot of 
+comparing apples and oranges is going on there ..
+
+* beautified charts
+
+

tpc/README

@@ -1,16 +1,37 @@
 Requirements:
 - GNU Make 3.81+
-- JDK 1.5+
+- JDK 1.7+
+
+for windows install cygwin with make+bash features
 
 To compile:
-  make
+  make 
+  (if you run in trouble its usually because of CRLF dirt) E.g. scalac did not run because it was CRLF
 
-To run:
-  ./run -help
+To run
 
-To run and generate charts with all serializers: (For official results, add -trials=500)
-  ./run -chart -include=`cat serializers.txt | tr "\\n" ","`  data/media.1.cks
+  bash run-bench.sh
+
+Parameters of the default run are hardcoded in run-bench.sh
 
+To update results of subset of serializers
 
-To generate bindings for schema files, see top of "Makefile" for
-instructions on what to put into "Config.mk".
+  bash run-bench.sh serializer1,serializer2,..
+
+How this works:
+
+  the project was somewhat messy, i probably contributed to that .. but its easier to manage now.
+  I removed the config files to avoid problems when serializers rename or are un/commented. Additionally
+  each benchmark runs in a separate VM.
+
+  1) the run-bench.sh uses class BenchmarkExporter.java to get a ',' separated list of all serializers to run.
+     All benchmarks registered in the BenchMarkRunner.java class a run be default.
+  2) after each bench finished, the mk-stats.sh script aggregates the results found in ./results/tmp to ./results.txt
+     the the class StatsCruncher.java creates a Textile style report.
+     To modify the report output, directly edit the StatsCruncher.java . You are welcome to enhance this using
+     some templating/configuration files.
+     You can copy the report.textile directly to github wiki in textile-mode.
+  3) You can run the mk-stats.sh script again without having to re-run the benchmarks if you want to add
+     custom query charts.
+  4) to filter out certain libraries, do not uncomment them in BenchMarkRunner.java (except they fail), better
+     edit StatsCruncher.java to exclude them.

tpc/attic/README_old

@@ -0,0 +1,16 @@
+Requirements:
+- GNU Make 3.81+
+- JDK 1.7+
+
+To compile:
+  make
+
+To run:
+  ./run -help
+
+To run and generate charts with all serializers: (For official results, add -trials=500)
+  ./run -chart -include=`cat serializers.txt | tr "\\n" ","`  data/media.1.cks
+
+
+To generate bindings for schema files, see top of "Makefile" for
+instructions on what to put into "Config.mk".

tpc/data/media.1.cks-result.txt

@@ -0,0 +1 @@
+

tpc/mk-stats.sh

@@ -0,0 +1,21 @@
+#! /usr/bin/env bash
+
+echo "" > stats.txt
+
+FILES=./results/tmp/*-result.txt
+for f in $FILES
+do
+   awk '/./{line=$0} END{print line}' $f >> stats.txt
+done
+
+cpgen=$(cat build/gen-cp)
+cplib=$(cat build/lib-cp)
+sep=':' 
+# cygwin
+case "`uname`" in
+	CYGWIN*) sep=';' ;;
+esac
+
+cp=./build/bytecode/main$sep$cpgen$sep$cplib
+
+java -cp $cp serializers.StatsCruncher

tpc/run-bench.sh

@@ -0,0 +1,54 @@
+#! /usr/bin/env bash
+
+# added this, because the other runscripts did not work for me with recent cygwin installation
+
+mem=-Xmx256m
+clz=serializers.BenchmarkRunner
+
+cpgen=$(cat build/gen-cp)
+cplib=$(cat build/lib-cp)
+sep=':' 
+# cygwin
+case "`uname`" in
+	CYGWIN*) sep=';' ;;
+esac
+
+cp=./build/bytecode/main$sep$cpgen$sep$cplib
+
+# for low run-to-run jitter  (anyway expect ~3% run-to-run jitter)
+# testTime=60000
+# warmupTime=60000
+# turn off turbo boost and any other kind of dynamic clock scaling
+
+testTime=10000
+warmupTime=15000
+iter=2000
+
+mkdir ./results/tmp &> /dev/null
+
+if [ -n "$1" ]; then
+    sentence=$1
+else
+    rm ./results/tmp/*.txt
+    sentence=$(java -cp $cp serializers.BenchMarkExporter) # just grab all serializers
+fi
+
+sentence=${sentence//,/$'\n'}  # change the colons to white space
+for word in $sentence
+do
+    echo "running $word .."
+    file=$word-result.txt
+    file=./results/tmp/${file//\//-}  # change '/' to '-'
+    echo $word > $file
+    java $mem -cp $cp $clz -iterations=$iter -warmup-time=$warmupTime -testRunMillis=$testTime -include=$word data/media.1.cks >> $file    
+done
+
+# find files with no numbers => errors
+echo ""
+echo "====================================================================================="
+echo "errors:"
+find ./results/tmp/. -print -type f -name "*.txt" -exec tail -1 {} \; | grep -B 1 create
+echo "====================================================================================="
+echo ""
+
+exec ./mk-stats.sh

tpc/src/serializers/BenchMarkExporter.java

@@ -0,0 +1,70 @@
+package serializers;
+
+/**
+ * Copyright (c) 2012, Ruediger Moeller. All rights reserved.
+ * <p/>
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ * <p/>
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ * <p/>
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301  USA
+ * <p/>
+ * Date: 09.03.14
+ * Time: 10:09
+ * To change this template use File | Settings | File Templates.
+ */
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Set;
+
+/**
+ * tweak to generate a string containing all registered benchmarks and extract bench feature data. called by run script
+ */
+public class BenchMarkExporter extends BenchmarkRunner {
+
+    String alltests = ""; // ',' separated 
+    HashMap<String,SerFeatures> featureMap = new HashMap<>(); // have to map back after running .. sigh
+
+    public BenchMarkExporter() {
+        runBenchmark(new String[0]);
+    }
+
+    protected void runBenchmark(String[] args)
+    {
+        TestGroups groups = new TestGroups();
+        addTests(groups);
+        Set<String> media = groups.groupMap.get("media").entryNames;
+        for (Iterator<String> iterator = media.iterator(); iterator.hasNext(); ) {
+            String next = iterator.next().trim();
+            if ( ! next.equals("cks") && ! next.equals("cks-text") ) // used to read data, exclude
+                alltests += next+ (iterator.hasNext() ? "," : "");
+            SerFeatures features = groups.groupMap.get("media").getSerMap().get(next).getFeatures();
+//            System.out.println("serializer:"+next+" miscFeatures: "+miscFeatures);
+            featureMap.put(next, features);
+        }
+    }
+
+    public String getAlltests() {
+        return alltests;
+    }
+
+    public HashMap<String, SerFeatures> getFeatureMap() {
+        return featureMap;
+    }
+
+    public static void main(String arg[]) {
+        System.out.println(new BenchMarkExporter().getAlltests());
+    }
+
+}

tpc/src/serializers/BenchmarkBase.java

@@ -11,14 +11,14 @@
  */
 abstract class BenchmarkBase
 {
-    public final static int DEFAULT_ITERATIONS = 2000;
-    public final static int DEFAULT_TRIALS = 500;
+    public final static int DEFAULT_ITERATIONS = 2000; 
+    public final static int DEFAULT_TEST_RUN_MILLIS = 10000; // 10 seconds
 
     /**
      * Number of milliseconds to warm up for each operation type for each serializer. Let's
      * start with 3 seconds.
      */
-    final static long DEFAULT_WARMUP_MSECS = 3000;
+    final static long DEFAULT_WARMUP_MSECS = 10000;
 
     // These tests aren't included by default.  Use the "-hidden" flag to enable them.
     protected static final HashSet<String> HIDDEN = new HashSet<String>();
@@ -54,7 +54,7 @@ public enum measurements
     protected final static class Params
     {
         public int iterations = DEFAULT_ITERATIONS;
-        public int trials = DEFAULT_TRIALS;
+        public int testRunMillis = DEFAULT_TEST_RUN_MILLIS;
         public long warmupTime = DEFAULT_WARMUP_MSECS;
         public boolean prewarm = true;
         public Boolean filterIsInclude;
@@ -171,19 +171,19 @@ else if (option.equals("iterations")) {
                     System.exit(1);
                 }
             }
-            else if (option.equals("trials")) {
+            else if (option.equals("testRunMillis")) {
                 if (value == null) {
-                    System.err.println("The \"trials\" option requires a value.");
+                    System.err.println("The \"testRunMillis\" option requires a value.");
                     System.exit(1);
                 }
                 try {
-                    params.trials = Integer.parseInt(value);
+                    params.testRunMillis = Integer.parseInt(value);
                 } catch (NumberFormatException ex) {
-                    System.err.println("Invalid value for \"trials\" option: \"" + value + "\"");
+                    System.err.println("Invalid value for \"testRunMillis\" option: \"" + value + "\"");
                     System.exit(1);
                 }
-                if (params.trials < 1) {
-                    System.err.println("Invalid value for \"trials\" option: \"" + value + "\"");
+                if (params.testRunMillis < 1) {
+                    System.err.println("Invalid value for \"testRunMillis\" option: \"" + value + "\"");
                     System.exit(1);
                 }
             }
@@ -238,7 +238,7 @@ else if (option.equals("help")) {
                 System.out.println();
                 System.out.println("Options:");
                 System.out.println("  -iterations=n         [default=" + DEFAULT_ITERATIONS + "]");
-                System.out.println("  -trials=n             [default=" + DEFAULT_TRIALS + "]");
+                System.out.println("  -testRunMillis=n      [default=" + DEFAULT_TEST_RUN_MILLIS + "ms]");
                 System.out.println("  -warmup-time=millis   [default=" + DEFAULT_WARMUP_MSECS + "]");
                 System.out.println("  -skip-pre-warmup      (don't warm all serializers before the first measurement)");
                 System.out.println("  -chart                (generate a Google Chart URL for the results)");
@@ -483,18 +483,20 @@ protected <J> EnumMap<measurements, Map<String, Double>> runMeasurements(PrintWr
                                  * Should only warm things for the serializer that we test next: HotSpot JIT will
                                  * otherwise spent most of its time optimizing slower ones...
                                  */
-                                warmTest(runner, params.warmupTime, testCreate);
+                                warmTest(runner, params.warmupTime/3, testCreate);
 
                                 doGc();
-                                double timeCreate = runner.runTakeMin(params.trials, testCreate, params.iterations * 100); // do more iteration for object creation because of its short time
+                                // ruediger: turns out startup/init time is pretty equal for all tests. 
+                                // No need to spend too much time here
+                                double timeCreate = runner.runWithTimeMeasurement(params.testRunMillis / 3, testCreate, params.iterations); 
 
                                 warmTest(runner, params.warmupTime, testSerialize);
 
                                 doGc();
-                                double timeSerialize = runner.runTakeMin(params.trials, testSerialize, params.iterations);
-
+                                double timeSerialize = runner.runWithTimeMeasurement(params.testRunMillis, testSerialize, params.iterations);
+                            
                                 doGc();
-                                double timeDeserialize = runner.runTakeMin(params.trials, testDeserialize, params.iterations);
+                                double timeDeserialize = runner.runWithTimeMeasurement(params.testRunMillis, testDeserialize, params.iterations);
 
                                 double totalTime = timeSerialize + timeDeserialize;