Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Cannot sync due to invalid modification time" #4378

Open
ThatCoffeeGuy opened this issue Mar 22, 2022 · 152 comments
Open

"Cannot sync due to invalid modification time" #4378

ThatCoffeeGuy opened this issue Mar 22, 2022 · 152 comments

Comments

@ThatCoffeeGuy
Copy link

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Expected behaviour

It should be able to sync files with no issues

Actual behaviour

Complains about invalid modification time while basically everything else works perfectly with the same files

Steps to reproduce

  1. Fill up your nextcloud instance with terabytes of files
  2. Wait a few months, it eventually starts corrupting your files

Client configuration

Version 3.4.4 (Windows).

Operating system: Windows 10

OS language: US ENG

Qt version used by client package (Linux only, see also Settings dialog):

Client package (From Nextcloud or distro) (Linux only):

Installation path of client:
default

Server version

22.2.5

I can see that it had ruined many people's files back in 2021 https://syncloud.discourse.group/t/disaster-with-nexcloud-3-4-0-client/141/4

But I am not sure if I was using the same instance back then. It seems this issue is still present. I am spending hours on trying to move files, rescan them, whatever. It is really making me stop recommending this (potentially) amazing product to anyone.

@liquidbase
Copy link

liquidbase commented Mar 23, 2022

Identical problem with Windows 11.

Client version is current 3.4.4
Server version is currently 23.0.3

Trying to fix the problem by setting the date of the last modification via Linux command touch to the current date does not interest the client. Even after a clean reinstallation of the client, the sync errors reappear right after the first sync.

Is there a possibility that the sync errors are stored in the database, preventing the client from starting a proper sync?

edit
A manual fix of the sync is possible depending on the size of the Nextcloud and would work as follows:

  1. Find all files with wrong modification date with the command
    find /path/to/data/directory/ -type f -name "*" -newermt 1970-01-01 ! -newermt 1970-01-02
  2. Scan all files to correct the database values with the command
    sudo -u www-data php occ files:scan --all
  3. Restart sync

If the sync continues to generate errors, you will need to manually clean up the user's deleted files on the server.

@hwangjunjie
Copy link

hwangjunjie commented Mar 24, 2022

Thanks, this method works and I have fixed it.
Here is the full command:
find /path/to/data/directory/ -type f -name "*" -newermt 1970-01-01 ! -newermt 1970-01-02 -print0 | xargs -0 touch -m

@Sieboldianus
Copy link

Sieboldianus commented Mar 24, 2022

nextcloud-client: 3.4.4
server: 22.2.5

Saw this error today on my main Documents folder sync, the first time after using Nextcloud 3 years (folder size: 40GB, quite a few files).

find /nectloud/server/data -type f -name "*" -newermt 1970-01-01 ! -newermt 1970-01-02

gives no results.

Are there any instructions/steps what else could be done/how to solve this issue?

Inside Documents, all files show green checkmarks and it doesn't say which files this applies to. If I right-click on the error in NC client and on "View activity", I get an empty window.

[Edit]
I can see that the Documents folder in the Web-App shows a wrong date (modified in the future, 86 years from now):
nc

On the server drive (ZFS), the folder has the correct date:

ls -alh --full-time
drwxr-xr-x 39 www-data www-data 49 2021-11-22 08:31:30.803407858 +0100  Documents
  • Rescanning the whole folder (sudo -u www-data php occ files:scan --path "user/files/Documents") does not work (error persists), same with files:scan --all
  • Deleting trash on the server for this user does not work

Atm. I assume this happened because I deleted a single photo through the web app today. I usually never do this (always remove the files on harddisk and let it sync). However, I don't know why this resulted in a broken timestamp on Nextcloud's database, and I don't know why it cannot get the correct one from the filesystem, since it is all there and correct.

[Edit2]

Also tested solvable_files.sh with list argument, but no output/no files detected.

[Edit3]

No way to solve this error with the sh files provided by Nextcloud so far.

Leaving this at it today, perhaps tomorrow someone will find a solution.

Here're the log lines from Desktop Client that appear to point to the issue:

click
2022-03-24 18:43:48:692 [ info nextcloud.sync.propagator libsync\owncloudpropagator.h:212 ]:	Starting CSyncEnums::CSYNC_INSTRUCTION_ERROR propagation of "Documents" by OCC::PropagateIgnoreJob(0x19f124febe0)
2022-03-24 18:43:48:692 [ debug nextcloud.sync.database.sql common/ownsql.h:145 ]	[ OCC::SqlQuery::bindValue ]:	SQL bind 1 "Documents"
2022-03-24 18:43:48:692 [ warning nextcloud.sync.propagator libsync\owncloudpropagator.cpp:281 ]:	Could not complete propagation of "Documents" by OCC::PropagateIgnoreJob(0x19f124febe0) with status OCC::SyncFileItem::NormalError and error: "Cannot sync due to invalid modification time"
2022-03-24 18:43:48:692 [ debug nextcloud.sync.statustracker libsync\syncfilestatustracker.cpp:274 ]	[ OCC::SyncFileStatusTracker::slotItemCompleted ]:	Item completed "Documents" OCC::SyncFileItem::NormalError CSyncEnums::CSYNC_INSTRUCTION_ERROR
2022-03-24 18:43:48:692 [ warning nextcloud.gui.activity gui\tray\usermodel.cpp:561 ]:	Item  "Documents"  retrieved resulted in  "Cannot sync due to invalid modification time"
2022-03-24 18:43:48:692 [ warning nextcloud.gui.activity gui\tray\usermodel.cpp:538 ]:	Item  "Documents"  retrieved resulted in error  "Cannot sync due to invalid modification time"
2022-03-24 18:43:48:692 [ info nextcloud.gui.activity gui\tray\activitylistmodel.cpp:402 ]:	Error successfully added to the notification list:  "Cannot sync due to invalid modification time"
2022-03-24 18:43:48:692 [ info nextcloud.sync.propagator libsync\owncloudpropagator.cpp:1179 ]:	PropagateDirectory::slotFirstJobFinished emit finished OCC::SyncFileItem::NormalError
2022-03-24 18:43:48:692 [ info nextcloud.sync.propagator.root.directory libsync\owncloudpropagator.cpp:1306 ]:	OCC::SyncFileItem::NormalError slotSubJobsFinished OCC::PropagatorJob::Running pending uploads 0 subjobs state OCC::PropagatorJob::Finished
2022-03-24 18:43:48:692 [ info nextcloud.sync.propagator libsync\owncloudpropagator.cpp:1320 ]:	PropagateRootDirectory::slotSubJobsFinished emit finished OCC::SyncFileItem::NormalError
2022-03-24 18:43:48:694 [ debug nextcloud.sync.database.sql common\ownsql.cpp:295 ]	[ OCC::SqlQuery::exec ]:	SQL exec "SELECT path FROM conflicts"
2022-03-24 18:43:48:694 [ debug nextcloud.sync.database.sql common\ownsql.cpp:295 ]	[ OCC::SqlQuery::exec ]:	SQL exec "DELETE FROM flags WHERE path != '' AND path NOT IN (SELECT path from metadata);"
2022-03-24 18:43:48:694 [ debug nextcloud.sync.database.sql common\ownsql.cpp:328 ]	[ OCC::SqlQuery::exec ]:	Last exec affected 0 rows.
2022-03-24 18:43:48:695 [ warning nextcloud.gui.folder gui\folder.cpp:971 ]:	SyncEngine finished with ERROR

@Sieboldianus
Copy link

Solved it for the moment:

choco uninstall nextcloud-client
choco install nextcloud-client --allow-downgrade --version 3.4.3

Now the desktop client syncs again and does not error on Modification date. The wrong (future) modification date on Documents folder is still there in the Nexctloud App/Database, but at least I can sync again. This is a workaround, not a solution.

@AndyXheli
Copy link

AndyXheli commented Mar 24, 2022

@Sieboldianus

Hi. That’s how I managed to solve this

First of all you need to find all the files with destroyed timestamp

find yournextcloudfilelocation/ ! -newermt 1970-01-02 > damaged.txt
then use this script to touch and modify the date. You can set whatewer the date you want just change 15 jan

#!/bin/bash

file="damaged.tx"

while read -r line; do
touch -d '15 jan' "$line"
done <$file

Then If you use docker

sudo docker exec --user www-data your containerID php occ files:scan --all
if you don’t use docker then

sudo -u www-data php occ files:scan --all

after that you need to sync all your clients from scratch

https://help.nextcloud.com/t/desktop-client-3-4-0-destroys-local-time-stamp-and-keeps-uploading-data-to-server/128512/96

@Sieboldianus
Copy link

Sieboldianus commented Mar 24, 2022

@AndyXheli Thanks andy. I tested these steps. find yournextcloudfilelocation/ ! -newermt 1970-01-02 > damaged.txt and similar commands did not return any files, same with the Nextcloud provided files.

There was only one folder affected: My main Documents folder, which had a date modified set to 2107 (86 years from now in the future). On the filesystem (server), the date was correct. On the client, the date was also correct.

⚠️ The wrong date was only in the nextcloud database. ⚠️

Here is how I solved this (just now).

cd /var/www/nextcloud/data/user/files/
touch -m Documents # Update only the last modified date to today
occ files:scan --path "user/files/Documents"' # scan again

tada:
image

What wasn't clear to me is that files:scan will only pick up files that changed.

Also updated nextcloud-client to 3.4.4 again and can confirm that the error disappeared.

@exchange12rocks
Copy link

Also if you have a Windows client, you can fix broken files from the client-side with this PowerShell one-liner (run it in the NextCloud folder in your user profile):

Get-ChildItem -Recurse | Where-Object -FilterScript {$_.LastWriteTime -eq ([datetime]::new(621355968000000000))} | ForEach-Object -Process {$_.LastWriteTime = Get-Date}

@Sieboldianus
Copy link

Also if you have a Windows client, you can fix broken files from the client-side with this PowerShell one-liner (run it in the NextCloud folder in your user profile):

Get-ChildItem -Recurse | Where-Object -FilterScript {$_.LastWriteTime -eq ([datetime]::new(621355968000000000))} | ForEach-Object -Process {$_.LastWriteTime = Get-Date}

I doubt this would work, since the client refused to sync to the server (at all) - because the wrong date was on the server (database only, fs not affected), not on the client side.

@exchange12rocks
Copy link

Also if you have a Windows client, you can fix broken files from the client-side with this PowerShell one-liner (run it in the NextCloud folder in your user profile):

Get-ChildItem -Recurse | Where-Object -FilterScript {$_.LastWriteTime -eq ([datetime]::new(621355968000000000))} | ForEach-Object -Process {$_.LastWriteTime = Get-Date}

I doubt this would work, since the client refused to sync to the server (at all) - because the wrong date was on the server (database only, fs not affected), not on the client side.

Works for me, mate 🤷‍♂️ - all green and happy now

@Maxplosion
Copy link

Also if you have a Windows client, you can fix broken files from the client-side with this PowerShell one-liner (run it in the NextCloud folder in your user profile):

Get-ChildItem -Recurse | Where-Object -FilterScript {$_.LastWriteTime -eq ([datetime]::new(621355968000000000))} | ForEach-Object -Process {$_.LastWriteTime = Get-Date}

Thanks, that worked for me to adjust the filedates, just had to change the timestamp to [datetime]::new(621356004000000000) as mine were 01.01.1970 01:00 instead of 00:00.

@pwnorbitals
Copy link

Also if you have a Windows client, you can fix broken files from the client-side with this PowerShell one-liner (run it in the NextCloud folder in your user profile):

Get-ChildItem -Recurse | Where-Object -FilterScript {$_.LastWriteTime -eq ([datetime]::new(621355968000000000))} | ForEach-Object -Process {$_.LastWriteTime = Get-Date}

Didn't work for me, it looks like the problem lies in the cloud database on my end too

@frankhoff
Copy link

Apparently the 01.01.1970 is not the only problematic date, with the update to version 3.4.4 on Linux Mint 20.3 some files suddenly got the modified at 07.02.2106.

Fixing on the client side works with find . -newermt 2106-02-06 -ls and find . -newermt 2106-02-06 -exec touch -m {} on a Linux client, but the error seems to remain on the server or database for some reason.

@lars-sh
Copy link

lars-sh commented Apr 3, 2022

We also had some cases of "Cannot sync due to invalid modification time", but instead of the other solutions, which work on the client-side, we solved this on the server-side, which might be a good idea in case you've no access to some clients yourself.

Solution

I found some entries in the data base table oc_filecache with either an unexpectedly low mtime or an unexpectedly high one (way larger than current timestamp). You can use the following query for that.

In a second step I replaced the mtime value using the storage_mtime value.
That's it.

SELECT *
FROM oc_filecache
WHERE oc_filecache.mtime <> oc_filecache.storage_mtime
  AND oc_filecache.mtime NOT BETWEEN 1000000000 AND UNIX_TIMESTAMP()
LIMIT 100
;

@pwnorbitals
Copy link

I found some entries in the data base table oc_filecache with either an unexpectedly low mtime or an unexpectedly high one (way larger than current timestamp). You can use the following query for that.

In a second step I replaced the mtime value using the storage_mtime value. That's it.

Can confirm this fixed the issue on my end

@jreed1701
Copy link

We also had some cases of "Cannot sync due to invalid modification time", but instead of the other solutions, which work on the client-side, we solved this on the server-side, which might be a good idea in case you've no access to some clients yourself.

Solution

I found some entries in the data base table oc_filecache with either an unexpectedly low mtime or an unexpectedly high one (way larger than current timestamp). You can use the following query for that.

In a second step I replaced the mtime value using the storage_mtime value. That's it.

SELECT *
FROM oc_filecache
WHERE oc_filecache.mtime <> oc_filecache.storage_mtime
  AND oc_filecache.mtime NOT BETWEEN 1000000000 AND UNIX_TIMESTAMP()
LIMIT 100
;

I wanted to try this out but I'm not an SQL expert. I was able to get into my PSQL database, connect to the database with oc_filecache table, however, running the command caused issues.

ERROR: function unix_timestamps() does not exists.   

Is this SQL statement for mysql instead? If so, can someone help me undertand the error message and/or tweak it for PSQL?

@jreed1701
Copy link

jreed1701 commented Apr 11, 2022

I ended up using:

UPDATE oc_filecache SET mtime = storage_mtime WHERE mtime < 86400;

However, it didn't work for me because BOTH mtime and storage_mtime for those entries were 0.

I ended up getting the current unix timstamp manually, and then replacing the text "1234567890" with it.

UPDATE oc_filecache SET mtime = 1234567890 WHERE mtime < 86400;

@cramaboule
Copy link

cramaboule commented Apr 14, 2022

@AndyXheli Thanks andy. I tested these steps. find yournextcloudfilelocation/ ! -newermt 1970-01-02 > damaged.txt and similar commands did not return any files, same with the Nextcloud provided files.

There was only one folder affected: My main Documents folder, which had a date modified set to 2107 (86 years from now in the future). On the filesystem (server), the date was correct. On the client, the date was also correct.

⚠️ The wrong date was only in the nextcloud database. ⚠️

Here is how I solved this (just now).

cd /var/www/nextcloud/data/user/files/
touch -m Documents # Update only the last modified date to today
occ files:scan --path "user/files/Documents"' # scan again

tada: image

What wasn't clear to me is that files:scan will only pick up files that changed.

Also updated nextcloud-client to 3.4.4 again and can confirm that the error disappeared.

This resolved my issue:
find /path/to/data/directory/ -type f -newermt "1900-01-01" ! -newermt "1980-01-01" -exec touch {} +

I checked all files and dir up to 1980 THAN

find /path/to/data/directory/ -type f -newermt "2023-01-01" ! -newermt "2200-01-01" -exec touch {} +

From 2023 up to 2200 as I had files modified in futur (86 years from now)
Even in files_trashbin !!!
then:
occ files:scan --path "user/files/Documents" --verbose
or
occ files:scan --all --verbose

You should not have any error

Now fixed !!!

Those files were on a Mac going to Windows by USB key and sync some years ago to my nexcloud server (Linux Nexcloud)

Hope this help !

C.

@lars-sh
Copy link

lars-sh commented Apr 14, 2022

However, it didn't work for me because BOTH mtime and storage_mtime for those entries were 0.

@jreed1701, I just remember problems with the desktop client version 3.4.0, that might be related to your problem and that's further explained othere here: https://github.com/nextcloud/desktop/wiki/How-to-fix-the-error-invalid-or-negative-modification-date

@jreed1701
Copy link

However, it didn't work for me because BOTH mtime and storage_mtime for those entries were 0.

@jreed1701, I just remember problems with the desktop client version 3.4.0, that might be related to your problem and that's further explained othere here: https://github.com/nextcloud/desktop/wiki/How-to-fix-the-error-invalid-or-negative-modification-date

Thanks @lars-sh - I followed the procedure and running ./solvable_files.sh just returns and does nothing. I'm sure I have the right arguments because ./unsolvable_files.sh works just fine.

@vvvinny
Copy link

vvvinny commented Apr 20, 2022

Thanks, this method works and I have fixed it. Here is the full command: find /path/to/data/directory/ -type f -name "*" -newermt 1970-01-01 ! -newermt 1970-01-02 -print0 | xargs -0 touch -m

Resolved with this command, but expanded date to 1969-12-31.

@jreed1701
Copy link

I got fed up, wiped my nextcloud server to start over. I ended up using the 3.3.6 client on Windows and that worked fine. I don't plan to upgrade until there's some definitive fix in the next release. Hopefully the admins communicate that clearly.

@benklop
Copy link

benklop commented Apr 21, 2022

in my case i think this occurred when I changed the folders being synced in the client while a sync was trying to pull down a bunch of files. After checking an additional folder and pressing apply, I started receiving these errors and the mtime in the database doesn't seem correct any longer.

@shibacomputer
Copy link

This bug burnt me severely in 2021 as mentioned in the OP, and its amazing to see this regression emerge again, first on Windows and now on Linux (Fedora 35). I literally have no real way of handling this with my users except to blanket ban the desktop clients, which is wild.

@vasyugan
Copy link

vasyugan commented May 4, 2022

I see this even though there are no actual invalid modification times, neither on the server nor on the client.

@Edukarl
Copy link

Edukarl commented May 9, 2022

I see this even though there are no actual invalid modification times, neither on the server nor on the client.

I have the same problem. No invalid modification time on the files. Seems I just moved some files to a different folder on the client at the wrong time (during upload?).

EDIT: Turns out, the did have the invalid 01.01.1970 modification time. Just not on the server itself, but only on the client. So the have been uploaded but syncing to other clients did not work.

@skito
Copy link

skito commented May 11, 2022

I'm having the same problem on desktop client running on Win10. MacOS and mobile clients are running without issues.

Tried to fix the mtimes with server side commands suggested above and also do occ files:scan --all, but actually everything on the server was fine.

So I started reverting back old client versions one by one and the the first version to fix that problem was version 3.3.6... No issues there. This is definitely client side issue and I can't believe there've been 12 new releases since, that still didn't fix the issue including the latest 3.5.0

Nextcloud team please address this bug with more serious attention. It's obvious that a change between 3.3.6 and 3.4.0 which is further replicated in newer versions is causing this issue.

I'm ready to assist and provide log data if needed. Just let me know.

@shibacomputer
Copy link

Why won't the Nextcloud team respond to this?
The original issue has nearly 200 👍's, and the best we've got is an automated script trying to close the issue every couple of months.

@gbakeman
Copy link

I'm yet another user who just reinstalled their system, installed NC Desktop 3.9.0, and was faced with this exact error for over 100+ files. @ErikUden latest answer helped me fix it, but hopeful there's a more systemic fix in the works.

@rene-gomez
Copy link

@ErikUden #4378 (comment) helped me fix it, but hopeful there's a more systemic fix in the works.

@github-actions
Copy link

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

@github-actions github-actions bot added the stale label Sep 20, 2023
@ThatCoffeeGuy
Copy link
Author

Can anyone confirm if it's fixed now? I completely stopped using all nextcloud clients so I can't test it.

@mia-riezebos
Copy link

Can anyone confirm if it's fixed now? I completely stopped using all nextcloud clients so I can't test it.

I haven't encountered this issue in at least a year of using it after it happened to me initially. The "hotfix" I used to fix it when I did encounter the issue was to manually set the modification date of all of my files to a valid date in the near past.
This means that I actually lost a lot of valuable DateTime information on files that were stored on my server before I applied the hotfix (upwards of 500G), but it was better than not being able to sync my libraries.

It is unclear (to me, at least) whether current clients have solved this issue, because i no longer have files with invalid modification times on my server...

Hoping someone else will be able to test this & confirm for you cuz my answer is probably not satisfactory.

@github-actions github-actions bot removed the stale label Sep 21, 2023
@ccoenen
Copy link

ccoenen commented Sep 21, 2023

This is one of those cases where a stale-bot is not just passive-aggressive, but may even be doing harm. This issue affected quite a few people and it may lead to data loss (far future data not being synced to your computer without a notice, a user relying on the correct working of a sync client may overlook data missing on their computer)

I would be way more comfortable if we had some info from the developers about this before this gets closed. Did anyone find out how these wrong timestamps could have happened in the first place? Are there now any new checks in place to prevent something like this in the future?

I have to admit that I don't recall every single one of the 100+ comments on this issue, so if I missed something, please let me know.

@EarthSalamander42
Copy link

Still happening. Server hosted on Nextcloud AIO v7.0.0 - Ubuntu 22.04
client 3.9.0 running on windows server 2022 standard (10.0.20348 Build 20348)

@BrickMyPhone
Copy link

Issue still persists. I have a folder whose last modified is 2477 years in the future(new record?). On the server itself the date is correct.

@jdaviescoates
Copy link

I've now started hitting this again too. I'm getting all the errors on my Nextcloud Android client which keeps asking me to choose which file to keep (local or server) for 49 files that have mysteriously now got the wrong date (1970 as so many others in this thread). Although whilst I was at first able to choose which one I wanted to keep (I went for the server ones which still had the correct date), now if I click on any of the notifications I just get an error Error creating conflict dialogue!

@szamanr
Copy link

szamanr commented Oct 21, 2023

i just set up nextcloud on a new linux machine (v3.10.1 from snap) and got this again. only some files are affected.

@dreua
Copy link

dreua commented Oct 21, 2023

Okay this is now with 212 upvotes by far (next is 66) the most upvoted issue on nextcloud/desktop. I've no idea when those votes appeared (i.e. whether there was a recent development in this bug hitting more people) but at any rate it would be appreciated by a ton of people (assuming that not everyone hit by this actually comes here and has an account to upvote) if someone from the dev team could acknowledge this bug and give some feedback.

I know this is open source, but Nextcloud is also a bigger piece of software which is not that easy to work on without prior knowledge (at least that is what I think) so if you need funding to work on this DO let us know.

@ThatCoffeeGuy
Copy link
Author

ThatCoffeeGuy commented Nov 3, 2023

I believe it had like 100 upvotes on the first 3 days of posting it. It's a long-running issue that's still not resolved, even worse, it directly corrupts your files.

Also, I don't think they are ignoring it, a few weeks back there was an update:

@jancborchardt jancborchardt moved this to from 🧭 Planning evaluation / ideas to 🏗️ At engineering in 🖍 Design team on Aug 31

But yeah, some official response, in a form of a comment at least, would be nice.

@laudrup
Copy link

laudrup commented Dec 2, 2023

I had a similar issue and inspired by this comment I figured out I had some entries in the nextcloud database where mtime had somehow ended up being -3600.

The same files on the server had a mtime of Unich epoch (1970-01-01) while the files on the client side had a timestamp sometime in the future (2106-02-07, not something recognizable to me at least).

I ended up deleting the entries with the bogus mtime from the database:

delete * from oc_filecache where mtime=-3600;

found the files one the server with an mtime of Unix epoch:

find . -type f ! -newermt 1970-01-01

and gave them a more reasonable timestamp.

Deleted the affected files from the client and ran:

sudo -u www-data php /var/www/nextcloud/occ files:scan --all

on the server.

The files end up being synced and I haven't seen the problem.

The above steps are more or less from memory as I cannot recall exactly what I did in which order.

Some steps might not be needed and you might end up losing data by trying this. Just sharing this in case someone might find it helpful and in case it might help getting to the root cause of this issue.

@vasyugan
Copy link

vasyugan commented Jan 2, 2024

I had a similar issue and inspired by this comment I figured out I had some entries in the nextcloud database where mtime had somehow ended up being -3600.

The same files on the server had a mtime of Unich epoch (1970-01-01) while the files on the client side had a timestamp sometime in the future (2106-02-07, not something recognizable to me at least).

I ended up deleting the entries with the bogus mtime from the database:

delete * from oc_filecache where mtime=-3600;

found the files one the server with an mtime of Unix epoch:

find . -type f ! -newermt 1970-01-01

and gave them a more reasonable timestamp.

Deleted the affected files from the client and ran:

sudo -u www-data php /var/www/nextcloud/occ files:scan --all

on the server.

The files end up being synced and I haven't seen the problem.

The above steps are more or less from memory as I cannot recall exactly what I did in which order.

Some steps might not be needed and you might end up losing data by trying this. Just sharing this in case someone might find it helpful and in case it might help getting to the root cause of this issue.

Been there, done that. Didn't help. Tried on multiple machines.

@shibacomputer
Copy link

This issue has returned for my instance just in time for 2024 🥳
Nextcloud 27.1.1 on an Ubuntu 22.04.1 server and S3 backend. User is running Fedora 39 with the latest client. But mainly bumping this so it doesn't get auto closed.

@aaronaxvig
Copy link

Similar to the comments above with the Powershell script, I was hitting this issue when trying to sync a bunch of files to Nextcloud for the first time. Some of those files didn't have a good Last Modified date, so I needed to fix that. I found that changing -eq (equal) to -lt (less than) in the script was quite helpful since my timestamps didn't exactly match 621355968000000000.

Get-ChildItem -Recurse | Where-Object -FilterScript {$_.LastWriteTime -lt ([datetime]::new(621355968000000000))} to show all the files with the issue.

Add | ForEach-Object -Process {$_.LastWriteTime = (Get-Date).AddYears(-10)} on the end of that previous command and rerun it to set that date to something nicer. 10 years ago from today seemed like a nice choice for me.

Full command:
Get-ChildItem -Recurse | Where-Object -FilterScript {$_.LastWriteTime -lt ([datetime]::new(621355968000000000))} | ForEach-Object -Process {$_.LastWriteTime = (Get-Date).AddYears(-10)}

My files did successfully sync to Nextcloud after this.

I suppose a past version of the client didn't mind these messed up dates, and would successfully sync them to Nextcloud. And then Nextcloud changed to not accept the messed up dates but people already had some on the server, which is what seems to be afflicting 90% of the people in this comment section.

@sdcampbell
Copy link

This happens to me on Mac. Works great on Ubuntu client. Then every once in a while I need to sync the files to a Macbook. The Mac is running the latest client version. I get a sync error and the file timestamp is 422 years in the future.

@ThatCoffeeGuy
Copy link
Author

The bug seems to be present for almost 2 years - Can we get a developer response?

@KarimGeiger
Copy link

Nah. Who cares about 240 confirmed cases of file corruption?

@didyouexpectthat
Copy link

My files' modified timestamp is set to when it was initially created, which predates 1970 in over 10,000 cases. Before officially supporting it, it would be nice to be able to ignore these errors. However, supporting them would be more ideal.

@hardwareadictos
Copy link

My files' modified timestamp is set to when it was initially created, which predates 1970 in over 10,000 cases. Before officially supporting it, it would be nice to be able to ignore these errors. However, supporting them would be more ideal.

That would be nice.

Unbeliebable that 2 years later there is no fix for this rather than "playing" with the database and with the data directory without warranties....

@Snuupy
Copy link

Snuupy commented Jun 23, 2024

For Windows, if you need to find the exact timestamp to look for (#4378 (comment)), you can get it by running:

$filePath = "brokenfile.ext"
$file = Get-ChildItem -Path $filePath

# Get LastWriteTime in .NET DateTime ticks
$lastWriteTimeTicks = $file.LastWriteTime.Ticks
$lastWriteTimeTicks

Then replace the above command.

@KronosTheLate
Copy link

I did some Julia scripting to update the problematic modification times. I used the run function with a command of the form touch $filename instead of the built-in touch function because the latter raised an error on directories for me.

Initially, I called mtime on one of the problematic files. The date displayed as 1970-01-01 and 1 hour I believe, which corresponded to a mtime of -3600, which is what I then looked for.

invalid_date_dir = "directory/contining/your/problematic/files/and_or/folders"
problematic_mtime = -3600
for (root, dirs, files) in walkdir(invalid_date_dir)
    for dir in dirs
        if mtime(joinpath(root, dir)) == problematic_mtime
            try
                cmd = `touch $(joinpath(root, dir))`
                run(cmd)
                @info "Updated modification time of directory $dir"
            catch e
                @warn "Encountered error while touching directory $dir. Continuing"
            end
        end
    end
    for file in files
        if mtime(joinpath(root, file)) == problematic_mtime
            try
                cmd = `touch $(joinpath(root, file))`
                run(cmd)
                @info "Updated modification time of file $file"
            catch e
                @warn "Encountered error while touching file $file. Continuing"
            end
        end
    end
end

I realize that this is done by others previously in this thread, but not in a way I could read and understand. I feel like this is rather clear, if you happen to have Julia installed.

@netpalantir
Copy link

I downloaded a folder from Google Drive, unzipped it, and copied it into the NextCloud sync folder. Almost all the files had this error!

I did not need to preserve the modification time of the files, so I just killed them all with find . -exec touch \{\} \;.

However, for a non-technical user, this is a showstopper bug.

(Also, I had a file with a very long name. The sync client got stuck in a loop inundating the desktop with the notification that the file has invalid characters...)

@ted423
Copy link

ted423 commented Sep 30, 2024

image
it's wrong before. so could you just sync those file?

https://drive.google.com/file/d/1BxhpYyyVDNbsGdlh7DjhVnK4doswGAEQ/view?usp=sharing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: 📄 To do (max 2 entries / member)
Status: 🏗️ At engineering
Development

No branches or pull requests