v0.9.3

[Mimetis]

v0.9.3

Packages are available as pre-release packages on nuget.
You can grab the packages here : https://www.nuget.org/packages?q=Dotmim.Sync

I'm sorry, there is no version v0.9.2 .... :)
Why ? Well... I just increment the version to .3 instead of .2, without any reason ;)
Hope you are not disappointed !

Upgrade to version v0.9.3

Do not forget to call the UpgradeAsync method to upgrade your current schema version the last version.

• Client Side : No need to call UpgradeAsync, since it's handled by default by DMS
• Server Side : You need to create a RemoteOrchestrator with the correct SyncSetup and call the UpgradeAsync method

// Upgrade to last version
if (await remoteOrchestrator.NeedsToUpgradeAsync())
   await remoteOrchestrator.UpgradeAsync(progress:progress);

When making the upgrade of your system (server / clients), be sure to remember that:

• A client upgraded is not compatible with a non upgraded server.
• A server upgraded will be compatible with non upgraded clients

That being said, be sure to firstly upgrade your server to the last version, call the UpgradeAsync (as explained before), then upgrade your clients.

BatchSize improvements

This new version removes the in memory sync process.

• This option is usually never used when sync architecture is based on HTTP transport.
• This option increases dramatically the memory consumption, and could be a big bottleneck on both side (Server / Client)
• The overall cost of using batchsize (and serialization on disk) is minimal and this new version brings a lot of improvements on disk usage.

All sync will rely now on the file system to serialize rows using the BatchSize option.

Calculation of batch files is now greatly improved and will match the BatchSize option. (When you are specifying 2000, you will have file where size will be <= 2 Mo. Previously this value was an approximation)

• BatchSize is mandatory now and default value is fixed to 5000 (5 Mo)
• BatchSize min value is fixed to 100 (100 ko)

Backward compatibility

If you are not using the batchsize mode on your client and do not have upgrade your client to version v0.9.3 (ie you are still on version <= v0.9.1), you are still compatible with this version deployed on the server.

We are still continuing to improve the DMS SDK, and trying to preserve the backward compatibility (Well .... we try at least !)

Performances

• Performances are better, especially when reading large databases.
• Since we are not relying on memory batches anymore, and with a new way of reading / writing rows, memory consumption have decreased by 80% in the best scenarios

Here is a comparison, on the first sync, between v0.9.3 and v0.9.1.
The database used has 120 tables (some tables have more than 20 columns) and we have more than 1M rows.

The overall reading performance increase is due to the use of Multiple Active ResultSets property of SQL Server.
Do not forget to add it in your connection string if you want to benefit from this improvement.
For MySql, performances have increased, but not that much, since MySql does not support MARS.

Elapsed time

Overall first synchronization elapsed time

Before :

After:

Memory consumption

Memory consumption has been greatly improved, due to an new algorithm to read writes rows on disk (where we are not using in memory collections anymore, if not needed)

Before:

After:

Snapshot creation

Snapshot is not meant to be transactional, since it's only a pre select.
We can use parallels select to create a snapshot on every providers.

Before:

After:

SyncProgressLevel

This version introduces a new option called SyncProgressLevel. It will allow you to have more details during the sync process.
Default level (equals to what we had before) is the Information level.
Debug level will get you more information on each events (before events and after events)
Trace level will get you the events before and after they have did their job plus events about metadata (triggers / tracking tables / stored procedures) creation
Sql level will get you all the sql statement executed.

var options = new SyncOptions { ProgressLevel = SyncProgressLevel.Information };

Information level :

As you can see you have all relevant information about read and writes tables

Debug Level:

Trace level:

Sql level:

Improvement on SqlSyncChangeTracking

Thanks to @kevindost we have now a better management of column filter, when using SqlSyncChangeTracking.
Now if a column is not part of your syncsetup table, and that column is updated, the row will not be synced anymore.

[Tuytje]

I'm trying to upgrade from 0.9.1 to 0.9.3 and I noticed 2 changes that affect my code on the web side:

1. WebServerOrchestrator.OnTableChangesBatchApplying
   This does not exist anymore. I used this method to set the userId on each incoming row. I take it that it has been replaced by another method, could you point me to that method?

2. WebServerOrchestrator.OnTableChangesSelected, args doesn't have 'Changes'
   I'm using the args.Changes to loop over all rows to set the userId to null before the data goes to the client. This 'Changes' method is available anymore. Where can I access this?

Thanks!

[Mimetis]

Indeed, due to performance issues, I've reworked the serialization and all the batch stuff

You can use this:

WebServerOrchestrator.OnTableChangesApplyingSyncRows(args =>
{
    foreach (var syncRow in args.SyncRows)
    {
    }
});

And

webServerOrchestrator.OnTableChangesSelectedSyncRow(args =>
{
    Console.WriteLine(args.SyncRow);
});

let me know if it's working for you

[Tuytje]

Am I correct in thinking that "OnTableChangesSelectedSyncRow" will be called for each row, where "OnTableChangesSelected" gave me a collection of rows to go over?
Thank you

[Mimetis]

Yes
Two majors differences:

• OnTableChangesSelectedSyncRow is called for each row before the row is added to the batch and serialized on disk
• OnTableChangesSelected is called after all rows are read, and the batch is created on the disk

So far, once the batch is serialized on the disk, you can open it and make some changes on it:

 // The table is read. The batch parts infos are generated and already available on disk
 webServerOrchestrator.OnTableChangesSelected(async tcsa =>
 {
     foreach (var bpi in tcsa.BatchPartInfos)
     {
         var table = await tcsa.BatchInfo.LoadBatchPartInfoAsync(bpi);

         foreach (var row in table.Rows.ToArray())
         {
             Console.WriteLine(row);
         }

         await tcsa.BatchInfo.SaveBatchPartInfoAsync(bpi, table);
     }
 });

[matevzcesar]

Hello.

We have upgraded our server from 0.9.1 to 0.9.4 and noticed that clients using version 0.9.1 synced to the upgraded server "successfully", as in the sync call completed without errors. However, all subsequent uploads did not resolve in changes on the server but they did get new changes from the server. After upgrading the clients to 0.9.4, new changes successfully uploaded to the server, but changes created when the clients versions were not the same as the server version did not.

We have solved this by updating these records locally, after the clients were updated and the changes showed up on the server.

So my question is, is this expected behavior? Should we have made sure all clients were upgraded to the new version, before making new changes?