Adding idManagement features + several corrections in the code

packages/matrix-client-server/HOW_TO_CONTINUE.md

@@ -0,0 +1,35 @@
+**What has been done in Client Server**
+
+
+__Client Authentication__ : @Amine Chraibi
+
+__Capabilities__ :
+
+__Events__ : The complete management of events has not been fully implemented. The different GET APIs have been implemented but the mecanism behind sending events requires to be developed. The redaction algorithm can be found in utils/event.ts. The /sync API which is central to the project is still being developed - it requires many additional features that were partially implemented.
+
+__Rooms__ : The management of rooms is closely intertwined with the mecanism of sending events so it is still in progress.
+
+__Modules__ : Many API's concerning different modules have been implemented. Beware that some TODO may remain if the complete implementation relied on features that were not developed at the time.
+
+__Encryption and Key management__ : nothing has been done concerning this for the moment
+
+
+**How to continue from there ?**
+
+__Read fully the Matrix Client Server spec__ : reading and fully understanding the spec is absolutely necessary to start developing the missing features in Client Server. Many of the remaining tasks are closely intertwined and require a precise understanding of the general functionning in order to be implemented.
+
+__Check out and understand the synapses implementation__ : since the main constraint of this project is to be able to reuse the database schema from the synapses project, it is necessary to study deeply how the spec is implemented in their project. 
+1) each time you want to use a table in their db - go checkout all the calls done to this table in their repository (with searching for "table_name" or "table='table_name") to understand how this table is intended to work
+2) for every component of their code, all the db interactions are developed in the folder synapse/storage/databases/main so it is necessary to see how they use their db to avoid missing key components
+3) for the more general behaviour of their code, go checkout the different handlers they implemented in the folder synapse/handlers 
+
+__How to implement the sync API__ : implementing the /sync api resquires to implement all the general behaviour of sending events /notifiyng people when new events have been sent / ensure the coherence of the different processes etc. A basis for the use of filters can be found in the utils/filter.ts file. A small basis for events can be found in the utils/events.ts file. 
+More generally it is necessary to understand precisely how synapses chose to function to be able to reproduce some behaviours we need to copy in order to use their db. 
+
+__Concerning existing APIs__ : the APIs that have been implemented may not be complete for the moment. They fully correspond to the expected behaviour described by the spec but they may not achieve internal actions that could be required for the correct implementation of further APIs (like /sync for instance). Therefore while developing the remaining behaviours it will be necessary to adapt and fix the behaviour of certain existing APIs.
+
+__Concerning the use of multiple workers in synapses__ : synapses uses several different workers that each do a part of the work but this is not the case in our architecture. Therefore we do not have to ensure coherence between the different workers like synapses does. Nevertheless since the server-server (federation service) is not yet implemented, it is important to keep in mind that ensuring coherence and a correct circulation of informations between all the servers is a big issue that must be tackled in order to finish the implementation.
+
+__Concerning the use of multiple writers for streams in synapses__ : synapses uses different writers to handle event streams and always needs to ensure that each writer is correctly following the stream and that each worker shares coherent data with other writers. For the moment the premisse of stream management that has been implemented relies on only one writer and does not tackle these issues. This is not a final choice (cf. utils/idManager.md) and surely a multi-writer stream management must be considered in the future.
+
+

packages/matrix-client-server/HOW_TO_USE_DB.md

@@ -0,0 +1,11 @@
+**How to use the synapses database**
+
+__Database schema__ : go check the schema in the synapses element-hq repository on Github (insert link). 
+NB : This schema includes deprecated tables for backwart compatibility. One should search in the synapses repository any occurence of 'a_table_name' or 'table=a_table_name' in order to see how synapses uses the table that might be useful. 
+
+__List of used tables for the moment__ : given the current development of the server, not all tables from synapses have been added to our project. The list of tables used for the moment can be found in the type Collections in src/matrixDb/index.ts.
+
+__How to add a new table__ : when it is necessary to use a new table that is not currently used, one must add it to the type Collections in src/matrixDb/index.ts. For test purposes, one must also add the line used to create the table in the matrixDbQueries in src/testData/buildUserDb.ts.
+
+__How to run queries on the db (get, insert, upsert, delete etc...)__ : there are many methods that achieve actions on the db. They are described in src/matrixDb/index.ts. They have to be both implemented in sqlite.ts and pg.ts. Some methods may be defined in the folder  matrix-identity server/src/db.
+

packages/matrix-client-server/HOW_TO_WRITE_TESTS.md

@@ -0,0 +1,5 @@
+**How to write tests**
+
+__Tests organisation__ :  tests are organized relatively to their prefix in the api url. 
+
+__Config management__ : in order to avoid conflicts between calls to the database, we chose to use one db per test file. The config to use is the registerConf.json and then in the first beforeAll it is necessary to give a new name to the db used. An example for the config management can be found in files like src/presence/presence.test.ts or src/rooms/rooms.test.ts.

packages/matrix-client-server/package.json

@@ -51,13 +51,14 @@
     "build:example": "rollup -p @rollup/plugin-typescript -e express,@twake/matrix-client-server -m -o example/client-server.js example/client-server.ts",
     "build:lib": "rollup -c",
     "start": "node server.mjs",
-    "test": "jest --passWithNoTests"
+    "test": "LOG_TRANSPORTS=File LOG_FILE=/dev/null jest"
   },
   "dependencies": {
     "@twake/config-parser": "*",
     "@twake/logger": "*",
     "@twake/matrix-identity-server": "*",
     "@twake/utils": "*",
+    "async-mutex": "^0.5.0",
     "express": "^4.19.2",
     "libphonenumber-js": "^1.11.4",
     "node-fetch": "^3.3.0"

packages/matrix-client-server/src/__testData__/buildUserDB.ts

@@ -39,7 +39,7 @@ const matrixDbQueries = [
   'CREATE TABLE IF NOT EXISTS "user_threepids" ( user_id TEXT NOT NULL, medium TEXT NOT NULL, address TEXT NOT NULL, validated_at BIGINT NOT NULL, added_at BIGINT NOT NULL, CONSTRAINT medium_address UNIQUE (medium, address) )',
   'CREATE TABLE IF NOT EXISTS threepid_validation_session (session_id TEXT PRIMARY KEY,medium TEXT NOT NULL,address TEXT NOT NULL,client_secret TEXT NOT NULL,last_send_attempt BIGINT NOT NULL,validated_at BIGINT)',
   'CREATE TABLE IF NOT EXISTS threepid_validation_token (token TEXT PRIMARY KEY,session_id TEXT NOT NULL,next_link TEXT,expires BIGINT NOT NULL)',
-  'CREATE TABLE IF NOT EXISTS presence (user_id TEXT NOT NULL, state VARCHAR(20), status_msg TEXT, mtime BIGINT, UNIQUE (user_id))',
+  'CREATE TABLE IF NOT EXISTS presence_stream( stream_id BIGINT, user_id TEXT, state TEXT, last_active_ts BIGINT, last_federation_update_ts BIGINT, last_user_sync_ts BIGINT, status_msg TEXT, currently_active BOOLEAN , instance_name TEXT)',
   'CREATE TABLE IF NOT EXISTS open_id_tokens ( token TEXT NOT NULL PRIMARY KEY, ts_valid_until_ms bigint NOT NULL, user_id TEXT NOT NULL, UNIQUE (token) )',
   'CREATE TABLE IF NOT EXISTS user_threepid_id_server ( user_id TEXT NOT NULL, medium TEXT NOT NULL, address TEXT NOT NULL, id_server TEXT NOT NULL )',
   'CREATE TABLE IF NOT EXISTS "access_tokens" (id BIGINT PRIMARY KEY, user_id TEXT NOT NULL, device_id TEXT, token TEXT NOT NULL,valid_until_ms BIGINT,puppets_user_id TEXT,last_validated BIGINT, refresh_token_id BIGINT REFERENCES refresh_tokens (id) ON DELETE CASCADE, used INTEGEREAN,UNIQUE(token))',
@@ -50,6 +50,8 @@ const matrixDbQueries = [
   'CREATE TABLE IF NOT EXISTS ui_auth_sessions_credentials(session_id TEXT NOT NULL, stage_type TEXT NOT NULL,  result TEXT NOT NULL, UNIQUE (session_id, stage_type),FOREIGN KEY (session_id) REFERENCES ui_auth_sessions (session_id))',
   'CREATE TABLE IF NOT EXISTS ui_auth_sessions_ips(session_id TEXT NOT NULL,ip TEXT NOT NULL,user_agent TEXT NOT NULL,UNIQUE (session_id, ip, user_agent), FOREIGN KEY (session_id)REFERENCES ui_auth_sessions (session_id))',
   'CREATE TABLE IF NOT EXISTS current_state_events (event_id text NOT NULL,room_id text NOT NULL,type text NOT NULL,state_key text NOT NULL,membership text)',
+  'CREATE TABLE IF NOT EXISTS stream_positions (stream_name TEXT NOT NULL,instance_name TEXT NOT NULL, stream_id BIGINT NOT NULL)',
+  'CREATE TABLE IF NOT EXISTS current_state_events (event_id text NOT NULL,room_id text NOT NULL,type text NOT NULL,state_key text NOT NULL,membership text)',
   'CREATE TABLE IF NOT EXISTS users_in_public_rooms ( user_id TEXT NOT NULL, room_id TEXT NOT NULL )',
   'CREATE TABLE IF NOT EXISTS users_who_share_private_rooms ( user_id TEXT NOT NULL, other_user_id TEXT NOT NULL, room_id TEXT NOT NULL )',
   'CREATE UNIQUE INDEX users_who_share_private_rooms_u_idx ON users_who_share_private_rooms(user_id, other_user_id, room_id)',