Integrate bitemporal SQL reporting service (Tech Sprint 2024)

.github/workflows/license-scanning-node.yml

@@ -16,7 +16,7 @@ jobs:
       matrix:
         node-version: ['20.x']
         module-folder: [
-          'reference-data',
+          'reference-service',
           'trade-feed',
           'web-front-end/angular',
           'web-front-end/react'

.github/workflows/security.yml

@@ -22,7 +22,7 @@ jobs:
     strategy:
         matrix:
           module-folder: [
-            'reference-data',
+            'reference-service',
             'trade-feed',
             'web-front-end/angular',
             'web-front-end/react']
@@ -154,7 +154,7 @@ jobs:
             'trade-service',
             'web-front-end/angular',
             'people-service',
-            'reference-data']
+            'reference-service']
     steps:
       - uses: actions/checkout@v4
       - uses: docker-practice/actions-setup-docker@master

README.md

@@ -7,10 +7,10 @@
 
 <img align="right" src="./docs/img/2023_TraderX_Vertical.png" alt="TraderX Logo" width="250"/>
 
-TraderX is a Sample Trading Application, designed to be a distributed reference application 
-in the financial services domain which can serve as a starting point for experimentation 
+TraderX is a Sample Trading Application, designed to be a distributed reference application
+in the financial services domain which can serve as a starting point for experimentation
 with various techniques and other open source projects.  It is designed to be simple
-and accessible to developers of all backgrounds, with minimal pre-assumptions, and it 
+and accessible to developers of all backgrounds, with minimal pre-assumptions, and it
 can serve as a starting point for educational and experimentation purposes.
 
 It is designed to be runnable from any developer workstation with minimal assumptions
@@ -23,14 +23,18 @@ messaging systems and are able to showcase a wide range of technical challenges
 More detailed information about this project can be found in the website which is generated
 from the code under the `docs` directory of this project.
 
-
 ## Project Demo and Overview Presentation
 
-Learn more about the project - including a brief demo, in the Keynote Demo session 
+Learn more about the project - including a brief demo, in the Keynote Demo session
 that was presented at the [Open Source in Finance Forum 2023](https://events.linuxfoundation.org/open-source-finance-forum-new-york/)
 
 [![TraderX Overview Video - OSFF 2023](./docs/img/2023_osff_video_thumb.png)](https://youtu.be/tSKDJlRYkm0?list=PLmPXh6nBuhJueQS5q-5IU3-0vmZEIUbz0&t=400)
 
+### Application Logic Changes
+
+The application has been modified so that it starts with a seed of only `Buy` trades so that we can open stock positions. The naive rationale is that you should not be able to sell securities that you don't have. Therefore if you do not have an open position with given security - you cannot sell it - and if you do have the security - you can only sell as much as you currently have. After all the securities have been sold - a position is closed.
+
+Trades and Positions are additionally saved to XTDB so that we can show a history of changes and consecutive trades. We introduced Prices (as discussed above - randomly generated rather than taken from some Exchange feed - for demonstration running locally they are a good enough approximation of 'live market').
 
 ## Project Components
 
@@ -41,28 +45,29 @@ The project consists of multiple moving parts, and you can see how things hang t
 | :--- | :--- | :--- |
 | [docs](docs) | markdown | Architecture and Flow Diagrams are here! |
 | [database](database) | java/h2 | A simple self-contained SQL database |
-| [reference-data](reference-data) | node/nestjs | REST service (off a flat file) for querying ticker symbols |
+| [reference-service](reference-service) | clojure | REST service (off a flat file) for querying ticker symbols, providing stock prices and also storing trades and positions in XTDB in order to provide history for Reports |
 | [trade-feed](trade-feed) | node/socketio | Message bus used for trade flows, as well as streaming to the GUI |
 | [people-service](people-service) | .Net core | Service for looking up users, for account mangement |
 | [account-service](account-service) | java/spring | Service for querying and validating accounts |
 | [position-service](position-service) | java/spring | Position service for looking up positions and trades by the blotter |
 | [trade-service](trade-service) | java/spring | Service for submitting trade/order requests for further processing |
 | [trade-processor](trade-processor) | java/spring | Trade Feed consumer which processes trade/orders |
 | [web-front-end](web-front-end) | html/angular or react | Interactive UI for executing trades and viewing blotter. Note: the AngularJS GUI was an initial contribution and contains account management capabilities. The React GUI was contributed during a hack day and may not work for managing accounts, but it does work for executing trades and viewing the blotter |
+[xtdb](./docker-compose.yml#25) | [bitemporal database](https://github.com/xtdb/xtdb) | open-source immutable database with comprehensive time-travel. XTDB has been built to simplify application development and address complex data compliance requirements. XTDB can be used via SQL and [XTQL](https://docs.xtdb.com/tutorials/introducing-xtql.html) |
 
-## Installation  
+## Installation
 
 This is installed locally through normal git clone operations.
 
 ## Usage example (Manual)
 
-In order to get things working together, it is recommended to select a range of ports to provide all running processes with, so that the pieces can interconnect as needed.  To run this all up 'by hand' here are default ports which are used, and you can easily export these variables to your favorite shell. 
+In order to get things working together, it is recommended to select a range of ports to provide all running processes with, so that the pieces can interconnect as needed.  To run this all up 'by hand' here are default ports which are used, and you can easily export these variables to your favorite shell.
 
 ```bash
 export DATABASE_TCP_PORT=18082
 export DATABASE_PG_PORT=18083
 export DATABASE_WEB_PORT=18084
-export REFERENCE_DATA_SERVICE_PORT=18085
+export REFERENCE_SERVICE_PORT=18085
 export TRADE_FEED_PORT=18086
 export ACCOUNT_SERVICE_PORT=18088
 export PEOPLE_SERVICE_PORT=18089
@@ -77,7 +82,8 @@ The recommended starting sequence to let everything find what it needs is:
 
 ```bash
 database
-reference-data
+xtdb
+reference-service
 trade-feed
 people-service
 account-service
@@ -89,17 +95,17 @@ web-front-end
 
 ## Usage (Docker + Docker Compose)
 
-The easiest way to run up the entire system is using Docker Compose. This should work on your local computer using Docker Desktop / Docker Compose (tested on Mac Silicon) and also in Github Codespaces. 
+The easiest way to run up the entire system is using Docker Compose. This should work on your local computer using Docker Desktop / Docker Compose (tested on Mac Silicon) and also in Github Codespaces.
 
 ### Codespaces
-If using Github Codespaces it is recommended you select an 8-core type machine with 32GB RAM to ensure all the components have the required resources to start. 
+If using Github Codespaces it is recommended you select an 8-core type machine with 32GB RAM to ensure all the components have the required resources to start.
 
-To do this 
+To do this
 * Select the Green Code menu at the top of this page
 * Select the Codespace tab then click the three dots '...' and select 'New with options...'.
 * Change the machine type to '8-core' and click 'Create codespace'
 
-As of writing, personal Github accounts receive 120 free core hours per month for using Codespaces, see the most recent details [here](https://docs.github.com/en/billing/managing-billing-for-github-codespaces/about-billing-for-github-codespaces#monthly-included-storage-and-core-hours-for-personal-accounts) 
+As of writing, personal Github accounts receive 120 free core hours per month for using Codespaces, see the most recent details [here](https://docs.github.com/en/billing/managing-billing-for-github-codespaces/about-billing-for-github-codespaces#monthly-included-storage-and-core-hours-for-personal-accounts)
 
 Once you have cloned the repository locally or once your Codespace has started, from the root traderX directory run
 ```

database/initialSchema.sql

@@ -1,70 +1,70 @@
 Drop Table Trades IF EXISTS;
 
-Drop Table AccountUsers IF EXISTS; 
+Drop Table AccountUsers IF EXISTS;
 
-Drop Table Positions IF EXISTS; 
+Drop Table Positions IF EXISTS;
 
-Drop Table Accounts IF EXISTS; 
+Drop Table Accounts IF EXISTS;
 
 Drop Sequence ACCOUNTS_SEQ IF EXISTS;
 
-CREATE TABLE Accounts ( ID INTEGER PRIMARY KEY, DisplayName VARCHAR (50) ) ; 
+CREATE TABLE Accounts ( ID INTEGER PRIMARY KEY, DisplayName VARCHAR (50) ) ;
 
-CREATE TABLE AccountUsers ( AccountID INTEGER NOT NULL, Username VARCHAR(15) NOT NULL, PRIMARY KEY (AccountID,Username));  
+CREATE TABLE AccountUsers ( AccountID INTEGER NOT NULL, Username VARCHAR(15) NOT NULL, PRIMARY KEY (AccountID,Username));
 
-ALTER TABLE AccountUsers ADD FOREIGN KEY (AccountID) References Accounts(ID); 
+ALTER TABLE AccountUsers ADD FOREIGN KEY (AccountID) References Accounts(ID);
 
-CREATE TABLE Positions ( AccountID INTEGER , Security VARCHAR(15) , Updated TIMESTAMP, Quantity INTEGER, Primary Key (AccountID, Security) );  
+CREATE TABLE Positions ( AccountID INTEGER , Security VARCHAR(15) , Updated TIMESTAMP, Quantity INTEGER, PValue INTEGER, Primary Key (AccountID, Security) );
 
-Alter Table Positions ADD FOREIGN KEY (AccountID) References Accounts(ID) ; 
+Alter Table Positions ADD FOREIGN KEY (AccountID) References Accounts(ID) ;
 
-CREATE TABLE Trades ( ID Varchar (50) Primary Key, AccountID INTEGER, Created TIMESTAMP, Updated TIMESTAMP, Security VARCHAR (15) ,  Side VARCHAR(10) check (Side in ('Buy','Sell')),  Quantity INTEGER check Quantity > 0 , State VARCHAR(20) check (State in ('New', 'Processing', 'Settled', 'Cancelled'))) ;  
+CREATE TABLE Trades ( ID Varchar (50) Primary Key, AccountID INTEGER, Created TIMESTAMP, Updated TIMESTAMP, Security VARCHAR (15) ,  Side VARCHAR(10) check (Side in ('Buy','Sell')),  Quantity INTEGER check Quantity > 0, UnitPrice INTEGER, State VARCHAR(20) check (State in ('New', 'Processing', 'Settled', 'Cancelled'))) ;
 
-Alter Table Trades Add Foreign Key (AccountID) references Accounts(ID); 
+Alter Table Trades Add Foreign Key (AccountID) references Accounts(ID);
 
 CREATE SEQUENCE ACCOUNTS_SEQ start with 65000 INCREMENT BY 1;
 
 --- SAMPLE DATA ---
 
-INSERT into Accounts (ID, DisplayName) VALUES (22214, 'Test Account 20'); 
-INSERT into Accounts (ID, DisplayName) VALUES (11413, 'Private Clients Fund TTXX'); 
-INSERT into Accounts (ID, DisplayName) VALUES (42422, 'Algo Execution Partners'); 
-INSERT into Accounts (ID, DisplayName) VALUES (52355, 'Big Corporate Fund'); 
-INSERT into Accounts (ID, DisplayName) VALUES (62654, 'Hedge Fund TXY1'); 
-INSERT into Accounts (ID, DisplayName) VALUES (10031, 'Internal Trading Book'); 
-INSERT into Accounts (ID, DisplayName) VALUES (44044, 'Trading Account 1'); 
-
-INSERT into AccountUsers (AccountID, Username) VALUES (22214, 'user01'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (22214, 'user03'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (22214, 'user09'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (22214, 'user05'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (22214, 'user07'); 
-
-INSERT into AccountUsers (AccountID, Username) VALUES (62654, 'user09'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (62654, 'user05'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (62654, 'user07'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (62654, 'user01'); 
-
-INSERT into AccountUsers (AccountID, Username) VALUES (10031, 'user01'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (10031, 'user03'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (10031, 'user09'); 
-
-INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user09'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user05'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user07'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user04'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user01'); 
-INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user06'); 
- 
-
-INSERT into Trades(ID, Created, Updated, Security, Side, Quantity, State, AccountID) VALUES('TRADE-22214-AABBCC', NOW(), NOW(), 'IBM', 'Sell', 100, 'Settled', 22214); 
-INSERT into Trades(ID, Created, Updated, Security, Side, Quantity, State, AccountID) VALUES('TRADE-22214-DDEEFF', NOW(), NOW(), 'MS', 'Buy', 1000, 'Settled', 22214); 
-INSERT into Trades(ID, Created, Updated, Security, Side, Quantity, State, AccountID) VALUES('TRADE-22214-GGHHII', NOW(), NOW(), 'C', 'Sell', 2000, 'Settled', 22214); 
-
-INSERT into Positions (AccountID, Security, Updated, Quantity) VALUES(22214, 'MS',NOW(), 1000); 
-INSERT into Positions (AccountID, Security, Updated, Quantity) VALUES(22214, 'IBM',NOW(), -100); 
-INSERT into Positions (AccountID, Security, Updated, Quantity) VALUES(22214, 'C',NOW(), -2000); 
-
-
-INSERT into Trades(ID, Created, Updated, Security, Side, Quantity, State, AccountID) VALUES('TRADE-52355-AABBCC', NOW(), NOW(), 'BAC', 'Sell', 2400, 'Settled', 52355); 
-INSERT into Positions (AccountID, Security, Updated, Quantity) VALUES(52355, 'BAC',NOW(), -2400); 
+INSERT into Accounts (ID, DisplayName) VALUES (22214, 'Test Account 20');
+INSERT into Accounts (ID, DisplayName) VALUES (11413, 'Private Clients Fund TTXX');
+INSERT into Accounts (ID, DisplayName) VALUES (42422, 'Algo Execution Partners');
+INSERT into Accounts (ID, DisplayName) VALUES (52355, 'Big Corporate Fund');
+INSERT into Accounts (ID, DisplayName) VALUES (62654, 'Hedge Fund TXY1');
+INSERT into Accounts (ID, DisplayName) VALUES (10031, 'Internal Trading Book');
+INSERT into Accounts (ID, DisplayName) VALUES (44044, 'Trading Account 1');
+
+INSERT into AccountUsers (AccountID, Username) VALUES (22214, 'user01');
+INSERT into AccountUsers (AccountID, Username) VALUES (22214, 'user03');
+INSERT into AccountUsers (AccountID, Username) VALUES (22214, 'user09');
+INSERT into AccountUsers (AccountID, Username) VALUES (22214, 'user05');
+INSERT into AccountUsers (AccountID, Username) VALUES (22214, 'user07');
+
+INSERT into AccountUsers (AccountID, Username) VALUES (62654, 'user09');
+INSERT into AccountUsers (AccountID, Username) VALUES (62654, 'user05');
+INSERT into AccountUsers (AccountID, Username) VALUES (62654, 'user07');
+INSERT into AccountUsers (AccountID, Username) VALUES (62654, 'user01');
+
+INSERT into AccountUsers (AccountID, Username) VALUES (10031, 'user01');
+INSERT into AccountUsers (AccountID, Username) VALUES (10031, 'user03');
+INSERT into AccountUsers (AccountID, Username) VALUES (10031, 'user09');
+
+INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user09');
+INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user05');
+INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user07');
+INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user04');
+INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user01');
+INSERT into AccountUsers (AccountID, Username) VALUES (44044, 'user06');
+
+
+INSERT into Trades(ID, Created, Updated, Security, Side, Quantity, UnitPrice, State, AccountID) VALUES('TRADE-22214-AABBCC', NOW(), NOW(), 'IBM', 'Buy', 100, 123, 'Settled', 22214);
+INSERT into Trades(ID, Created, Updated, Security, Side, Quantity, UnitPrice, State, AccountID) VALUES('TRADE-22214-DDEEFF', NOW(), NOW(), 'MS', 'Buy', 1000, 88, 'Settled', 22214);
+INSERT into Trades(ID, Created, Updated, Security, Side, Quantity, UnitPrice, State, AccountID) VALUES('TRADE-22214-GGHHII', NOW(), NOW(), 'C', 'Buy', 2000, 321, 'Settled', 22214);
+
+INSERT into Positions (AccountID, Security, Updated, Quantity, PValue) VALUES(22214, 'MS',NOW(), 1000, -88000);
+INSERT into Positions (AccountID, Security, Updated, Quantity, PValue) VALUES(22214, 'IBM',NOW(), 100, -12300);
+INSERT into Positions (AccountID, Security, Updated, Quantity, PValue) VALUES(22214, 'C',NOW(), 2000, -642000);
+
+
+INSERT into Trades(ID, Created, Updated, Security, Side, Quantity, UnitPrice, State, AccountID) VALUES('TRADE-52355-AABBCC', NOW(), NOW(), 'BAC', 'Buy', 2400, 20, 'Settled', 52355);
+INSERT into Positions (AccountID, Security, Updated, Quantity, PValue) VALUES(52355, 'BAC',NOW(), 2400, -48000);

docker-compose.yml

@@ -1,5 +1,3 @@
-version: '3.4'
-
 services:
   database:
     restart: always
@@ -24,23 +22,41 @@ services:
     command: ./run.sh
     networks:
       - localnet
-  reference-data:
+  xtdb:
+    image: ghcr.io/xtdb/xtdb:latest
+    pull_policy: always
+    expose:
+      - 5432
+      - 3000
+    ports:
+      - 18099:5432
+      - 18100:3000
+    volumes:
+      - xtdb-data:/var/lib/xtdb
+    networks:
+      - localnet
+  reference-service:
     restart: always
-    image: reference-data
     build:
-      context: reference-data
+      context: reference-service
       dockerfile: Dockerfile
-    working_dir: /reference-data
     expose:
       - 18085
     ports:
       - 18085:18085
-    volumes:
-      # Mount the root folder that contains .git
-      - .:/workspace:cached
-    command: npm run start
+    environment:
+      - XTDB_HOST=xtdb
+      - XTDB_PORT=5432
+      - WEB_PORT=18085
+      - TRADE_FEED_ADDRESS=http://trade-feed:18086
+      - PRICE_UPDATE_INTERVAL_MS=10000
     networks:
       - localnet
+    depends_on:
+      - xtdb
+      - trade-feed
+    volumes:
+      - ./reference-service:/reference-service
   trade-feed:
     restart: always
     image: trade-feed
@@ -134,7 +150,7 @@ services:
       - DATABASE_TCP_HOST=database
       - PEOPLE_SERVICE_HOST=people-service
       - ACCOUNT_SERVICE_HOST=account-service
-      - REFERENCE_DATA_HOST=reference-data
+      - REFERENCE_SERVICE_HOST=reference-service
       - TRADE_FEED_HOST=trade-feed
     volumes:
       # Mount the root folder that contains .git
@@ -146,7 +162,7 @@ services:
       - database
       - people-service
       - account-service
-      - reference-data
+      - reference-service
       - trade-feed
   trade-processor:
     restart: always
@@ -213,8 +229,16 @@ services:
       - position-service
       - trade-service
       - trade-processor
+      - reference-service
       - web-front-end-angular
 
 networks:
   localnet:
     driver: bridge
+
+volumes:
+  xtdb-data:
+    driver: local
+
+
+

docs/code.md

@@ -22,7 +22,7 @@ You can find the coordinates of the codebase here. Their relationship is explain
 * Python / Flask
 * [Source Code](../position-service)
 
-### `reference-data`
+### `reference-service`
 
 * NodeJS
 * [Source Code](../refdata-service)

ingress/nginx.traderx.conf.template

@@ -5,8 +5,8 @@
         location /db-web/ {
             proxy_pass http://database:18084/;
         }
-        location /reference-data/ {
-            proxy_pass http://reference-data:18085/;
+        location /reference-service/ {
+            proxy_pass http://reference-service:18085/;
         }
 
         location /ng-cli-ws {
@@ -28,7 +28,7 @@
 
         }
 
-    location /socket.io/ {
+        location /socket.io/ {
             proxy_set_header X-Forwarded-For ${DOLLAR}proxy_add_x_forwarded_for;
             proxy_set_header Host ${DOLLAR}http_host;