v0.9.5

[Mimetis]

version V 0.95

Introduction

Beta nuget packages available : https://www.nuget.org/packages/Dotmim.Sync.Core/0.9.5-beta-0788

This new version v0.95 is a major DMS update.
The main purpose of this version is to allow any setup to be managed by DMS.

• Upgrading from v0.94 to v0.95
• Immutablity & stateless objects
• Migration
• Removing triggers columns comparison
• WebServerAgent replace WebServerOrchestrator. WebRemoteOrchestrator replace WebClientOrchestrator
• Multiple Interceptors
• Overriding current operation from Server side: OnGettingOperation

[Mimetis]

Upgrading from v0.94 to v0.95

Upgrading from v0.94 to v0.95 is now supported.

• Client Side : No need to call UpgradeAsync, since it's handled by default by DMS
• Server Side : You need to create a RemoteOrchestrator and call the UpgradeAsync method

// Upgrade to last version
if (await remoteOrchestrator.NeedsToUpgradeAsync())
   await remoteOrchestrator.UpgradeAsync(progress:progress);

When making the upgrade of your system (server / clients), be sure to remember that:

• A client upgraded is not compatible with a non upgraded server.
• A server upgraded will be compatible with non upgraded clients

That being said, be sure to firstly upgrade your server to the last version, call the UpgradeAsync (as explained before), then upgrade your clients.

[Mimetis]

Immutablity & stateless objects

DMS main objects (LocalOrchestrator, RemoteOrchestrator, SyncAgent) become all agnostic to the setup instance.

That's why a lot of code have been refactored to be able to extract any dependencies in these objects.
Some parameters from some methods have been moved to maintain the methods agnostics and independant.

For instance, let's see how we called SynchronizeAsync in v0.94 and now in v0.95:

// v0.94
// ------------------
var serverProvider = new SqlSyncProvider(serverConnectionString);
var clientProvider = new SqliteSyncProvider(clientConnectionString);
var agent = new SyncAgent(clientProvider, serverProvider, tables);
var s1 = await agent.SynchronizeAsync();

// v0.95
// ------------------
var serverProvider = new SqlSyncProvider(serverConnectionString);
var clientProvider = new SqliteSyncProvider(clientConnectionString);
var agent = new SyncAgent(clientProvider, serverProvider);
var s1 = await agent.SynchronizeAsync(tables);

As you can see the tables are not part of the SyncAgent instance, but are passed to the method SynchronizeAsync() instead.

A new feature has been added to allow calling SynchronizeAsync() without any parameters (SyncSetup or tables String[]), if you have already done it before :

// v0.95
// ------------------
// Already done a sync with the correct setup before
var s1 = await agent.SynchronizeAsync();

Working with scopes becames way more easily:

// v0.95
// ------------------
var agent = new SyncAgent(client.Provider, Server.Provider, options);

var setup = new SyncSetup("Product", "ProductCategory");
var s = await agent.SynchronizeAsync(setup);

var setup = new SyncSetup("Employee");
var s2 = await agent.SynchronizeAsync(setup, "v2");

And as said earlier, on the next call, you do not have to pass the setup instance:

// v0.95
// ------------------
var agent = new SyncAgent(clientProvider, serverProvider);

var s1 = await agent.SynchronizeAsync();
var s2 = await agent.SynchronizeAsync("v2");

Following this pattern, a lot of methods from LocalOrchestrator and RemoteOrchestrator have been refactored:

• You can call a public method with the scopeName as an argument.
• You can call a public method with the corresponding SyncSetup instance.

For example the method ProvisionAsync on a RemoteOrchestrator was previously called like this:

// v0.94
// ------------------
var setup = new SyncSetup(new string[] { "ProductCategory", "Product" });
var remoteOrchestrator = new RemoteOrchestrator(serverProvider, setup);
await remoteOrchestrator.ProvisionAsync();

And now the new method signature is:

// v0.95
// ------------------
var setup = new SyncSetup(new string[] { "ProductCategory", "Product" });
var remoteOrchestrator = new RemoteOrchestrator(serverProvider);
var serverScope = await remoteOrchestrator.ProvisionAsync(setup);

[Mimetis]

Migration

First of all, DMS will not update or migrate your schema anymore
This is a major (intended) regression from previous version.

• The Method MigrationAsync has disappeared.
• Automatic schema update has been disabled.

The only timeframe where DMS has an interaction with your schema is during the first setup.
DMS will continue to create the schema on the client side, if the database is empty.
Once it's done, all updates must be handled by your code, using tools like:

• EF Migrations
• Manual sql scripts updates
• Any others tools you know

Why removing the automatic schema update ?

Well basically, automatic updates on clients and server will force to update all clients at the same time.
And we may have some scenarios where some clients can't update their schema or do not want to.

We need to be able to serve and sync clients :

• With the previous version of the schema until the schema is updated
• With the next version of the schema when schema update have been applied.

How to migrate from one version to the next one

You have an in depth example here : https://github.com/Mimetis/Dotmim.Sync/blob/v0.9.5/Samples/Migration/Program.cs

First of all, it's your responsibility to ensure your database can handle several versions.
For instance:

• Do not add a column in a table that does not accept Null values (all clients database that are not updated will fail when sync will happen)
• Do not remove a table that was previously needed and where constraints will make the sync failing.
• Any others examples you can think of

Here is a full example on how to handle multiple clients where:

• A first sync topology is in place: We are going to create a scope called v1.
• The two clients are using the version v1 of the database.
• Server will update the schema with a new column and a new table
• A new scope called v2 will be added to the sync topology to handle clients updated and clients not updated.
• First client will make an update of its schema and once it's done will use the new scope v2.
• Second client will stay on version v1.

We are using a direct TCP RemoteOrchestrator, but HTTP WebClientOrchestrator works the same way

// v0.95
// ------------------
var serverProvider = new SqlSyncProvider(serverDbConnectionString);
var client1Provider = new SqlSyncProvider(clientDbConnectionString);
var client2Provider = new SqlSyncProvider(clientDbConnectionString));

// first setup will contain 1 table with 4 columns
var setup = new SyncSetup(new string[] { "ProductCategory" });
setup.Tables[productCategoryTableName].Columns.AddRange(
       new string[] { "ProductCategoryId", "Name", "rowguid", "ModifiedDate" });

// Assuming we have two clients that are syncing this first setup on a scope named "v1":
var agent1 = new SyncAgent(client1Provider, serverProvider);
await agent1.SynchronizeAsync( setup, "v1");

var agent2 = new SyncAgent(client2Provider, serverProvider);
await agent2.SynchronizeAsync( setup, "v1");

Now, we are updating the server schema:

// v0.95
// ------------------
// assuming we are adding a column to ProductCategory
// this method will add a new column : "CreatedDate" using a sql stmt:
// ALTER TABLE dbo.ProductCategory ADD CreatedDate datetime NULL;
await AddColumnsToProductCategoryAsync(serverProvider);

// Now we are creating a new scope called V2:
// ProductCategory with 1 more column
// a new table Product
var setupV2 = new SyncSetup(new string[] { "ProductCategory", "Product" });
setupV2.Tables["ProductCategory"].Columns.AddRange(
    new string[] { "ProductCategoryID", "ParentProductCategoryID", "Name", "rowguid", "ModifiedDate", "CreatedDate" });

// We are provisioning this new scope on the server
var serverScope = await remoteOrchestrator.ProvisionAsync("v2", setupV2);

We have here two scopes running in parrallel on the server :

• v1 with 1 table : ProductCategory with 4 columns.
• v2 with 2 tables : ProductCategory with 5 columns and Product table.

At this point, you need to update your client schema as it's not made automatically by DMS anymore:

// v0.95
// ------------------
// Add the column CreatedDate to client 1
await AddColumnsToProductCategoryAsync(clientProvider1);

// Add product table to client 1
await CreateTableProductAsync(clientProvider1);

Note that we are not updating the schema of client 2.

We are provsioning the client 1 with the new scope v2:

// v0.95
// ------------------
var serverScope = await remoteOrchestrator.GetServerScopeInfoAsync("v2")
// Provision the "v2" scope on the client 1 with the new setup
await localOrchestrator.ProvisionAsync(serverScope);

// makeing a sync on the client 1 with the v2 scope
Console.WriteLine(await agent.SynchronizeAsync("v2", SyntType.Reinitialize));

Client 1 updated to v2 will be able to sync and Client 2, still on v1 is still able to sync.

[slagtejn]

The solution for the server works good, but the client implementation needed to be changed to throw an exception and it's still a strange solution. The client scope info data would be overwritten when sync finishes because the agent has the old object. It updates the scope with the schema & setup, but it's the old object. So provision will update the table correctly, but when sync finishes it will update it back. Also this solution depends on the events firing in that specific order. I know they will now, but in the future that might not be the case. I still think it would be nice to have some sort of interceptor that contains both the Client&Server scope info and be able to update them.

Updated client code:

var remoteOrchestrator = new WebRemoteOrchestrator(uri);
var localOrchestrator = new LocalOrchestrator(provider);

ClientScopeInfo clientScopeInfo = null;
ServerScopeInfo serverScopeInfo = null;

localOrchestrator.OnClientScopeInfoLoaded(args =>
{
    clientScopeInfo = args.ClientScopeInfo;
});

// I assume you are using a WebRemoteOrchestrator
// with a classic RemoteOrchestrator, you can use OnServerScopeInfoLoaded() instead
remoteOrchestrator.OnHttpGettingScopeResponse(async args =>
{
    serverScopeInfo = args.ServerScopeInfo;

    if (clientScopeInfo != null && !clientScopeInfo.IsNewScope && !clientScopeInfo.Setup.EqualsByProperties(serverScopeInfo.Setup))
    {
        var cs = await localOrchestrator.ProvisionAsync(serverScopeInfo, overwrite: true);
        throw new Exception("Re-provisioned scope"); //Need to short circuit because the agent still has the old ClientScopeInfo
    }
});

[Mimetis]

Interesting.
Let me find a better solution

The client scope info data would be overwritten when sync finishes because the agent has the old object.

That should not happens I guess.
If you are changing the scope_info in an interceptor, it should be changed as well in the sync agent.

[Mimetis]

Ok, I think I get a solution, that is not just some weird edges interception.
Regarding a little bit closer, we may have a lot of situations, if we want to stick to your single scope migration.

First of all, I have simplified the RemoteOrchestrator call, to make it simplier:

Updating an existing scope

var setup = new SyncSetup("ProductCategory", "Product", "Customer");

// NOT NEEDED ANYMORE
// var schema = await remoteOrchestrator.GetSchemaAsync(setup);
// serverScope.Schema = schema;
// serverScope.Setup = setup;

// Using the overwrite property to ovewrite an existing scope the setup
var serverScope = await remoteOrchestrator.ProvisionAsync(setup, overwrite: true);

On client side, I've added a new interceptor in the pipeline to intercept a client setup conflicting with a server setup, and let the choice to the dev handling the situation.

First scenario : You are handling the migration of the setup and you know when and how to call the routine:

var serverScope = await remoteOrchestrator.GetServerScopeInfoAsync();
var localOrchestrator = new LocalOrchestrator(client.Provider);
await localOrchestrator.ProvisionAsync(serverScope, overwrite: true);

Second Scenario : You are upgrading the schema, but don't want to handle something on the client side, relaying on a new interceptor

Here is a piece of code we may eventually have.
As you can see, when a conflict occurs on the setup (this code is running on the client) you can do something and finally returning an enumeration value ConflictingSetupAction

agent.LocalOrchestrator.OnConflictingSetup(async args =>
{
    if (args.ServerScopeInfo != null)
    {
        args.ClientScopeInfo = await localOrchestrator.ProvisionAsync(args.ServerScopeInfo, overwrite: true);

        // this action will let the sync continue
        args.Action = ConflictingSetupAction.Continue;

        return;
     }
    // if we raise this step, just and the sync without raising an error
    args.Action = ConflictingSetupAction.Abort;

    // The Rollback Action will raise an error
    // args.Action = ConflictingSetupAction.Rollback;
    }
});

Once you have intercept the conflict, and if you are returning a ConflictingSetupAction.Continue value, the LocalOrchestrator will make a final check, comparing the two scopes (ClientScopeInfo and ServerScopeInfo) and if they are still conflicting, the sync will raise an error.

Thought ?

[slagtejn]

This is a much better solution! I don't have any other thoughts to improve it.

[Mimetis]

I'm just checking everything is fine, and I will merge the PR in the master branch today I guess
This new feature will be part of the next version v0.9.6

[Mimetis]

Removing triggers columns comparison

Triggers are not making a comparison on each column anymore
Since we need to handle multiple scopes, update trigger will make an update in the tracking table, even if the update is coming from a column not present in any registered scopes.
It will basically reproduce the same behavior as what we have actually with the change tracking provider.

Here is a comparison before and after this change:

-- v0.94
-- ------------------

UPDATE [side] 
SET 	[update_scope_id] = NULL -- since the update if from local, it's a NULL
	,[last_change_datetime] = GetUtcDate()

FROM [ProductCategory_tracking] [side]
JOIN INSERTED AS [i] ON [side].[ProductCategoryID] = [i].[ProductCategoryID]
JOIN DELETED AS [d] ON [d].[ProductCategoryID] = [i].[ProductCategoryID]
WHERE (
	ISNULL(NULLIF([d].[ParentProductCategoryID], [i].[ParentProductCategoryID]), NULLIF([i].[ParentProductCategoryID], 
       [d].[ParentProductCategoryID])) IS NOT NULL
	 OR ISNULL(NULLIF([d].[Name], [i].[Name]), NULLIF([i].[Name], [d].[Name])) IS NOT NULL
	 OR ISNULL(NULLIF([d].[rowguid], [i].[rowguid]), NULLIF([i].[rowguid], [d].[rowguid])) IS NOT NULL
	 OR ISNULL(NULLIF([d].[ModifiedDate], [i].[ModifiedDate]), NULLIF([i].[ModifiedDate], [d].[ModifiedDate])) IS NOT NULL

-- v0.95
-- ------------------
UPDATE [side] 
SET 	[update_scope_id] = NULL -- since the update if from local, it's a NULL
	,[last_change_datetime] = GetUtcDate()

FROM [ProductCategory_tracking] [side]
JOIN INSERTED AS [i] ON [side].[ProductCategoryID] = [i].[ProductCategoryID]

[VagueGit]

This seems to be working better than 0.94. We have this table

CREATE TABLE [OrderDetails] (
	"OrderDetailId"		integer NOT NULL,
	"OrderId"		integer NOT NULL,
	"JobId"		integer,
	"ItemCode"		nvarchar(15) NOT NULL COLLATE NOCASE CHECK (length(ItemCode) <= 15),
	"ItemDescription"		nvarchar(35) NOT NULL COLLATE NOCASE CHECK (length(ItemDescription) <= 35),
	"UnitCost"		numeric NOT NULL CHECK (UnitCost BETWEEN -9999998.99 AND 9999999.99),
	"NrUnits"		smallint NOT NULL CHECK (NrUnits BETWEEN 0 AND 32767),
	"UnitNet"		numeric NOT NULL CHECK (UnitNet BETWEEN -9999998.99 AND 9999999.99),
    PRIMARY KEY ([OrderDetailId])
,
    FOREIGN KEY ([OrderId])
        REFERENCES [Orders]([OrderId])
);

In 0.94 inserts on the client would sync to the server, but an update to NrUnits would not sync.

In 0.95 updates to this table sync as expected. Thank you!

[Mimetis]

Ah ah I dont know why its working better in the new version but its cool !

[Mimetis]

WebServerAgent

The Dotmim.Sync.Web.Client has been refactored to replace WebClientOrchestrator with a more appropriate WebRemoteOrchestrator.

Since this orchestrator is basically used on the client side as a "remote web proxy orchestrator" it's more appropriate to rename it to WebRemoteOrchestrator.

Just renaming WebClientOrchestrator to WebRemoteOrchestrator will do the trick:

// v0.94
// ------------------
var proxyOrchestrator = new WebClientOrchestrator("https://localhost:44342/api/sync");
var clientProvider = new SqlSyncProvider(clientConnectionString);
var agent = new SyncAgent(clientProvider, serverOrchestrator);

// v0.95
// ------------------
var remoteOrchestrator = new WebRemoteOrchestrator("https://localhost:44342/api/sync");
var clientProvider = new SqlSyncProvider(clientConnectionString);
var agent = new SyncAgent(clientProvider, remoteOrchestrator);

The Dotmim.Sync.Web.Server has been refactored to replace WebServerOrchestrator with a more appropriate class WebServerAgent.

It basically changes the controller, but not too much.

// v0.94
// ------------------
private WebServerOrchestrator orchestrator;

// Injected thanks to Dependency Injection
public SyncController(WebServerOrchestrator webServerOrchestrator)
    => this.orchestrator = webServerOrchestrator;

/// <summary>
/// This POST handler is mandatory to handle all the sync process
/// </summary>
/// <returns></returns>
[HttpPost]
public Task Post()
    => orchestrator.HandleRequestAsync(this.HttpContext);

/// <summary>
/// This GET handler is optional. It allows you to see the configuration hosted on the server
/// The configuration is shown only if Environmenent == Development
/// </summary>
[HttpGet]
public Task Get()
    => WebServerOrchestrator.WriteHelloAsync(this.HttpContext, orchestrator);

// v0.95
// ------------------

private WebServerAgent webServerAgent;
private readonly IWebHostEnvironment env;

// Injected thanks to Dependency Injection
public SyncController(WebServerAgent webServerAgent, IWebHostEnvironment env)
{
    this.webServerAgent = webServerAgent;
    this.env = env;
}

/// <summary>
/// This POST handler is mandatory to handle all the sync process
/// </summary>
/// <returns></returns>
[HttpPost]
public Task Post() 
    => webServerAgent.HandleRequestAsync(this.HttpContext);

/// <summary>
/// This GET handler is optional. It allows you to see the configuration hosted on the server
/// </summary>
[HttpGet]
public async Task Get()
{
    if (env.IsDevelopment())
    {
        await this.HttpContext.WriteHelloAsync(webServerAgent);
    }
    else
    {
        var stringBuilder = new StringBuilder();

        stringBuilder.AppendLine("<!doctype html>");
        stringBuilder.AppendLine("<html>");
        stringBuilder.AppendLine("<title>Web Server properties</title>");
        stringBuilder.AppendLine("<body>");
        stringBuilder.AppendLine(" PRODUCTION. Write Whatever You Want Here ");
        stringBuilder.AppendLine("</body>");
        await this.HttpContext.Response.WriteAsync(stringBuilder.ToString());
    }
}

[maxrodro]

Hi @Mimetis
Thank you very much for this great contribution, many in this community appreciate it.

I'm going to put my team to test and see how this great improvement helps us.

[Mimetis]

Interceptors

The interceptors are basically the objects in charge of all the On methods, like OnTableChangesApplying, OnTableChangesSelected, OnDatabaseChangesApplied and so on...

Previously these interceptors were unique:
Once you called a On method, it was not possible to call the same method again during a sync process.

For instance, this code will raise the OnSessionBegin only one time:

// v0.94
// ------------------

// Creating an agent that will handle all the process
var agent = new SyncAgent(clientProvider, serverProvider, options);

agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 1"));
agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 2"));
agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 3"));
agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 4"));

var s = await agent.SynchronizeAsync(scopeName, setup, progress);*
Console.WriteLine("Synchronization done.");

The output from this code will be:

Sync start
Session begins 4
Synchronization done.

Today, you can have multiple interceptors of the same type.
Think like an event handler where you subscribe multiple times on the same event.
The same code will have a different behavior in the next release.

Note that we have a Guid returned each time you call an interceptor through a On method:

// v0.95
// ------------------

var id1 = agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 1"));
var id2 = agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 2"));
var id3 = agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 3"));
var id4 = agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 4"));

var s = await agent.SynchronizeAsync(scopeName, setup, progress);*
Console.WriteLine("Synchronization done.");

The output from this code will be:

Sync start
Session begins 1
Session begins 2
Session begins 3
Session begins 4
Synchronization done.

As said, each call to the OnSessionBegin will return a unique Guid you can use to remove an interceptor:

// v0.95
// ------------------

var id1 = agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 1"));
var id2 = agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 2"));
var id3 = agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 3"));
var id4 = agent.LocalOrchestrator.OnSessionBegin(sba => Console.WriteLine("Session begins 4"));

agent.LocalOrchestrator.ClearInterceptors(id3);

var s = await agent.SynchronizeAsync(scopeName, setup, progress);*
Console.WriteLine("Synchronization done.");

The output from this code will be:

Sync start
Session begins 1
Session begins 2
Session begins 4
Synchronization done.

Obviously you can remove all interceptors from certain type:

// v0.95
// ------------------

agent.LocalOrchestrator.ClearInterceptors<SessionBeginArgs>();

Or remove all interceptors:

// v0.95
// ------------------

agent.LocalOrchestrator.ClearInterceptors();

[Mimetis]

Overriding current operation from Server side: OnGettingOperation

You can now override the current synchronization in progress (called here operation) from your client, using the OnGettingOperation interceptor, from the server side.
Basically, you can, for a particular client (identified by its ClientScopeId) override the sync in progress, with different values:

[Flags]
public enum SyncOperation
{
    /// <summary>
    /// Normal synchronization
    /// </summary>
    Normal = 0,

    /// <summary>
    /// Reinitialize the whole sync database, applying all rows from the server to the client
    /// </summary>
    Reinitialize = 1,
    
    /// <summary>
    /// Reinitialize the whole sync database, applying all rows from the server to the client, after trying a client upload
    /// </summary>
    ReinitializeWithUpload = 2,

    /// <summary>
    /// Drop all the sync metadatas even tracking tables and scope infos and make a full sync again
    /// </summary>
    DropAllAndSync = 4,

    /// <summary>
    /// Drop all the sync metadatas even tracking tables and scope infos and exit
    /// </summary>
    DropAllAndExit = 8,

    /// <summary>
    /// Deprovision stored procedures & triggers and sync again
    /// </summary>
    DeprovisionAndSync = 16,
}

You can override the operation using the OnGettingOperation, from the server side:
As an example, the Post method in your sync controller where you want to force a ReinitializeAsync from a particular client:

[HttpPost]
public async Task Post()
{
    // Get scope name called by the client
    var scopeName = HttpContext.GetScopeName();

    // Get client scope id
    var clientScopeId = context.GetClientScopeId();

    // In a multi scopes scenario, get the correct WebServerAgent instance
    var webserverAgent = webserverAgents.FirstOrDefault(c => c.ScopeName == scopeName);

   webServerAgent.RemoteOrchestrator.OnGettingOperation(operationArgs=>
   {
      SyncOperation syncOperation = SyncOperation.Normal;

      // Let's imagine you have a service class that get the operation needed
      syncOperation = yourServiceClass.GetOperationForClientId(clientScopeId);
      // if hard coded, can be :
      // syncOperation = SyncOperation.Reinitialize;

      // this operation will be applied for the current sync
      operationArgs.Operation = syncOperation; 
   });
    

    // Handle request
    await webserverAgent.HandleRequestAsync(HttpContext).ConfigureAwait(false);

}