Changes uploaded but not applied on the server?

[MaTToX3]

Hello,

we ran into a critical issue, where changes are selected on the client and uploaded to the server however on the server, they are not applied and no error is reported back leading to data loss. I have been wondering what could cause such behavior? Below is the JSON result of a sync we log. It shows total uploaded changes as 1060 (which is a sum of all the changes selected on the client), however only the very last table is actually being applied and this is the only data we have. We have around 70 users syncing data multiple times per day and only twice has this happen so far however I fear more of those issues will appear. Any ideas or tips where to look for an issue?

{
    "SessionId": "e6852570-e39e-4a87-851b-baf96e56d077",
    "StartTime": "2022-06-06T12:33:16.354775Z",
    "ScopeName": null,
    "CompleteTime": "2022-06-06T12:37:07.885707Z",
    "TotalChangesApplied": 2363,
    "TotalChangesDownloaded": 2363,
    "TotalChangesUploaded": 1060,
    "TotalResolvedConflicts": 297,
    "TotalSyncErrors": 0,
    "ChangesAppliedOnServer": {
        "tca": [
            {
                "tn": "PROD_Dobavnica_History",
                "sn": "",
                "st": 16,
                "rc": 0,
                "a": 331,
                "f": 0,
                "trc": 1060,
                "tac": 331
            }
        ]
    },
    "ChangesAppliedOnClient": {
        ... removed ...
    },
    "SnapshotChangesAppliedOnClient": null,
    "ClientChangesSelected": {
        "tcs": [
            {
                "n": "CL_CrtnaKoda",
                "s": "",
                "u": 134
            },
            {
                "n": "PROD_Dobavnica",
                "s": "",
                "u": 10
            },
            {
                "n": "PROD_DobavnicaArtikel",
                "s": "",
                "u": 136
            },
            {
                "n": "PROD_RealiziraneKolicineMerilecStaging",
                "s": "",
                "d": 4,
                "u": 9
            },
            {
                "n": "PROD_SifraDobavnice",
                "s": "",
                "u": 1
            },
            {
                "n": "CL_CrtnaKoda_History",
                "s": "",
                "u": 279
            },
            {
                "n": "PROD_SifraDobavnice_History",
                "s": "",
                "u": 11
            },
            {
                "n": "PROD_DobavnicaArtikel_History",
                "s": "",
                "u": 145
            },
            {
                "n": "PROD_Dobavnica_History",
                "s": "",
                "u": 331
            }
        ]
    },
    "ServerChangesSelected": {
        ... removed ... 
    }
}

[Mimetis]

Which version are you running ?

[Mimetis]

Hard to tell. Not enough info.
Lost rows are new rows ? I mean, Are these rows mostly Insert (that do not appears on server) or more Update rows (or even Delete rows)?

Do you have a reproduction sample to share ? (I can imagine that must be complicated ...)

[MaTToX3]

Lost rows are new rows.

We are trying to reproduce the issue ourselves, no luck so far. For a quick fix we have implemented the "touch record" feature for our clients, which will update all the records automatically and then request the new sync to see if data will be applied then - since our clients still have the data and we don't want to loose that. However this is only temporary fix (if it is even going to work of course).

Will keep you updated regarding that.

[Mimetis]

Ok, thanks.
Is it working when you are applying the "touch record" ?

[MaTToX3]

Hello,

I can confirm that touching the records and syncing again helps - the missing data has now been saved. Unless one of the clients makes reinit sync (which can happen), we seem to have a backup plan for it, which is great.

[MaTToX3]

Could it be related to MARS being enabled?

[MaTToX3]

I was wondering; a couple weeks ago we implemented "CANCEL" option to our sync progress dialog - since sometimes sync took a while, we gave the user an option to cancel it so he/she could continue working. Could this somehow lead to weird behavior around that? We do nothing special, just pass CancellationTokenSource around and cancel it, however I did see it throws different Exception for cancelation, based on how long you wait before you cancel it (it can be SyncException, OperationCanceledException).

Since applying is, as far as I know, done in transaction, this could probably be nothing, but I am taking a look at all of the features we implemented to see if there is a bug somewhere. Any info/details around that? :)

[matevzcesar]

This issue occurs when syncing over HTTP on version 0.9.4.

Issue: Only tables from the last request are applied, agent reports successful sync.

This issue seemed to be connected to cleaning the batch folder, before the client got the response from the last request, could be a timeout or some network issue.

To reproduce this issue, we have set a breakpoint inside the OnDatabaseChangesApplied interceptor, which is invoked after cleaning the changes folder and waited for the request to timeout, then resumed the application. When a retry of the last request is sent from the client, only the changes from that request are applied and the sync finishes with no errors. However it is evident from the sync result data, that more changes were selected than applied, as shown in OP.

Workaround: Set CleanFolder to false on server side SyncOptions. The issue has not reoccurred after this workaround.

I have tried to reproduce this issue on version 0.9.6 the same way. It seemed to be fixed, but I did not check if it was because the interceptor was invoked earlier in this version.

@Mimetis do you have any insight if this could be an issue in later versions?

[Mimetis]

Yes, I think I made a correction on this version (don't remember exactly which version tbh)