Have Been Trying to Import 300,000 Photos for Weeks! Request to start in middle? Chron job? :)

[ElliotMcGucken]

Greetings All!

The import from flickr stalls after a few days. Sometimes it stalls at over 100,000 photos, sometimes under. To date I have imported around 150,000 photos out of 300,000. But it's been a week since the import has even reached beyond 150,000 photos, after numerous stalls and restarts in the browser.

It may be that flickr times out, or the internet connection to the browser times out, but the timeout always happens in different places.

For this reason, I would like to request a feature where one could elect to just start at 100,000 photos, or at some arbitrary number. So basically the first 100,000 photos in the flickr stream could be skipped, and flickr2piwigo would just immediately start at 100,001 photos.

This is because it can take a day or two or three just to GET to the place where it stalled at 100,000 photos, and sometimes it stalls before it get there. So three days can be spent just verifying the photos which I know I already have anyway, until I have to restart the import from the browser, whence after a few days it stalls again.

Also, a server-side chron job would at least eliminate the browser network timeout.

Ideas? Suggestions?

Thank you! :)

[samwilson]

A CLI tool is #13 (I've sort of started it, but not really).

Hopefully I'll find time soon! I've been rather out of action for a couple of weeks.

I wonder if there's JS memory leak that's causing things to bomb out after this long? I'll have a look. But yeah, it's a bit much to expect a browser session to run for days and weeks! A cron-jobbable script would be great for other reasons too (e.g. easy sync from mobile to Flickr then to Piwigo without any action required).

The other thing could be to track where we're up to in the import, and kick it off from there when it's restarted. But of course, there would be issues with that if other new photos have been added on the Flickr side (or some deleted too, which could mean that there would be some skipped if we were re-starting from a remembered point). Hmm.

[ElliotMcGucken]

Greetings Sam!

One idea here:

When importing photos from flickr, Piwigo records the date uploaded to Flickr and saves it in the Piwigo database.

Flickr2Piwigo always begins by importing the most recently uploaded photos to Flickr. Thus before beginning the import, Flickr2Piwigo could search the Piwigo database for the Flickr photo with the oldest upload date and note its Flickr ID number as OLDEST_PHOTO.

Then, could Flickr2Piwigo begin by only importing the photos that were uploaded before the photo OLDEST_PHOTO?

Also, a simple toggle/option could be provided in Flickr2Piwgo, which the user could select:

1. Begin import anew
2. Start import beyond OLDEST_PHOTO

Might this be simple enough to do?

I think I could get the browser to complete the entire import with this option!

The great thing is that Flickr2Piwigo can sometimes run for days, so now the only problem is that it takes days to reach beyond the 150,000 photos it has already imported, so that it can import the next 150,000 photos. Thank you! :)