Tutorial

In this tutorial, we will cover how to test a piece of code used for calculating volume curves with qspec and using qspec to help ensure the validity of the code as we change and refactor it to suit the changing capabilities that the code needs to support. Familiarity with q is assumed and the individual qspec functions are not covered in depth. Please refer to the reference pages for further details.

To start with, let’s take a look at the code currently being used for calculating the volume curves.

curve-v1.q

curve:{[syms;start;end];
  v:select avgBucket:sum[size]%count[date], total: sum size by sym, time.minute from trade where date within `date$(start;end), sym in syms, time within `time$(start;end);
  tv: exec sum total by sym from v;
  `sym`minute xasc select avgBucket, pctDaily:total%tv[first sym] by sym,minute from v
  }

This is pretty simple and might represent what we would have after a very preliminary pass to get something up and working. The results it produces look like this:

q)curve[`IBM`MSFT;2009.11.01T09:30;2009.11.30T16:00]                                                                                                                                
sym minute| avgBucket pctDaily    
----------| ----------------------
IBM 09:30 | 2828.571  0.00466508  
IBM 09:31 | 3309.524  0.005458301 
IBM 09:32 | 3654.545  0.006314351 
IBM 09:33 | 3117.391  0.005631082 
IBM 09:34 | 1308.333  0.002466053 
IBM 09:35 | 412.5     0.0007775134
IBM 09:36 | 3280.952  0.005411179 
IBM 09:37 | 1363.636  0.002356101 
IBM 09:38 | 484       0.0009502941
IBM 09:39 | 904.1667  0.001704246 
IBM 09:40 | 728       0.001429368 
IBM 09:41 | 422.7273  0.0007303913
IBM 09:42 | 1524      0.002992248 
IBM 09:43 | 2800      0.005277667

Now, ideally, you would actually be using somewhat real data to test against. Since I don’t have a contract with Reuters to republish marketdata or any generously donated sample data sets, I’ll make due with some random data that I cooked up.

If you would like to follow along exactly, here is the code I’m using to make the random data and the commands used to generate it:

gen.q

gen:{[syms;prices;nums;start;days];
  raze (enlist each flip `sym`date!flip syms cross start + til days) cross' {[num;price]
    invAbs:{(count[x]?1 -1)*x};
    n:`int$num + first invAbs 1?(1?1f)*num; / Adjust num to generate within 0-100% of base num up or down
    freqs:0f, sums .8 .1 .05 .01 .01 .01 .016 .001 .001 .001;
    sizes:100 200 300 400 500 1000 10000 20000 30000 40000 50000;
    ([]time:asc 09:30t + n?16t - 09:30t;price:price + invAbs n?first 1?.1;size:sizes freqs bin n?1f)
    } .' raze days#'enlist each nums,'prices
  }

q gen.q -S 200
KDB+ 2.6 2009.12.04 Copyright (C) 1993-2009 Kx Systems
q)`:trade set gen[`IBM`MSFT`AAPL;100 30 200f;1000 2000 10000;2009.11.01;30]
`:trade

Okay, so now that there is some data and some code, we have something that we can test. Let’s start with some pretty basic properties that the code should have. First, the column indicating the percentage of the daily volume should probably add up to 1 for each symbol. Next, if we were to calculate average daily volume and then sum the average bucket column, we should have matching values. Finally, dividing the average bucket values by the average daily volume should yield values equal to the percentage of the daily volume. We will begin by expressing these as @expectations@ inside of a @specification@:

.tst.desc["Volume curves"]{
  should["have percentage values that add to 1"]{};
  should["have average bucket volumes equal in sum to the average daily volume"]{};
  should["match the percentage of daily volume column to the average bucket column divided by ADV"]{};
  };

Note that we haven’t written any actual tests yet, just jotted down what we’re going to be testing and roughly what behaviors we expect it to conform to. Even this is enough to get some output from the testing tool however:

test_curve-vA.q

testq test_curve-vA.q --desc
Volume curves::
- should have percentage values that add to 1
- should have average bucket volumes equal in sum to the average daily volume
- should match the percentage of daily volume column to the average bucket column divided by ADV

Now, let’s move on to defining one of the tests.

test_curve-vB.q

.tst.desc["Volume curves"]{
  should["have percentage values that add to 1"]{
    fixture `trade;
    c:curve[`IBM`MSFT;2009.11.01T09:30;2009.11.30T16:00];
    (value exec first sum pctDaily by sym from c) musteq 1f; 
    c:curve[`IBM`MSFT;2009.11.01T12:00;2009.11.15T15:00];
    (value exec first sum pctDaily by sym from c) musteq 1f; 
    };  
  should["have average bucket volumes equal in sum to the average daily volume"]{};
  should["match the percentage of daily volume column to the average bucket column divided by ADV"]{};
  };  

We’ve established some simple tests that the outputs of the curve function match the expectation. These are pretty much happy path scenarios, but they’ll be good enough for now. Also note that we have used the @fixture@ function to load in the test data that we generated earlier.

Running this test yields the following:

testq curve-v1.q test_curve-vA.q
...
For 1 specifications, 3 expectations were run.

3 passed, 0 failed.  0 errors.

Currently, empty specifications will report as having passed. For the time being, please be careful that you ensure not to leave a test empty before committing. I apologize for the inconvenience.

Moving on, we’ll fill in the last two tests:

test_curve-vC.q