-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Flickr2Piwigo sees only 190,168 photos and presents errors. It chops up long album titles. #12
Comments
Thanks for reporting this; I'll hopefully get to examining things this weekend. And no, if you start again it'll do it all again (let's aim to fix that though! and maybe add a cronjob possibility, for on-going syncing). |
Thanks Sam! So if I start it again, will it overwrite the images and albums, or will it add the same images again under different names and in different albums? Will it take up new disk space with the exact same images, as it downloads them again? |
It seems that if it doesn't download every single image in one go, then one must delete all the photos and databases and reinstall and start over. I guess this means that each time one needs to update it with some new flickr photos, one must delete the installation, database, and all the images downloaded so far, and reinstall and start over? Thanks! |
No, if it's not working correctly now, we'll fix it so it does work correctly. There will be no need to delete the database! :) |
Thanks! After downloading 8,000 photos and closing one browser, I opened a new one and logged in. Flickr2Piwigo has put me in an endless loop for the last few hours. I press "Import All my Pictures," it says "processing" for ten or fifteen minutes, and then takes me back to the beginning, and no photos are added. I have done this a couple dozen times over the last few hours, so I will probably just reinstall and start the whole thing over. |
Greetings! So I deleted the old installation last night and started with a brand new one. I left the browser running overnight, and after it transferred a few thousand photos from flickr to piwigo, the browser crashed. So now I am trying to start the transfer again, and again I am in a loop. I press "Import All my Pictures," it says "processing" for ten or fifteen minutes, and then takes me back to the beginning, and no photos are added. I can list all my albums, but there is no way to transfer photos. It seems that if Flickr2Piwigo does not successfully transfer every photo in one single browser-based run, then there is no way to transfer the photos it failed to transfer? Thanks! P.S. Each time I get to the "Begin Transfer" screen, I hit the button to begin the transfer, and it takes me back to "Import all my Pictures" screen. I hit the "Import all my Pictures" and after ten minutes it takes me to the "Begin Transfer" screen. I hit the button to begin the transfer, and it takes me back to "Import all my Pictures" screen and again and again. This is the loop i have been stuck in in two different installations. |
A simple, and great, solution would be a simple linux shell command to transfer all the photos for flickr. It would likely simplify your work, and solve 90%+ of the transfer needs! :) And if the transfer died, the script could just pick up where it last left off? Thanks! :) |
Yes, I'm also thinking that a shell command would be a great addition, and perhaps will work on such a thing before getting some of these other bugs ironed out. You're right, it wouldn't be too hard. |
Thanks! Yes the shell command would make Flickr2Piwigo even more popular, and after it works, it could easily become a browser button used during the setup of Piwigo too. :) Regarding learning about object-oriented programming for php and the Flickr API, what resources/books might you recommend? Thanks! :) |
Hello Sam! Might you have any idea when a command-line script for Flickr2Piwigo would be available? It would be great for new installations! Simple functionality would rock:
And too, the above functionality would also be able to update the Piwigo installation, via a regular chron job or just running the script now and then. Any idea on how long that might take? Thanks! |
The CLI tool is being tracked in #13. As for books, I'm not really sure. Just start coding something you need! And go from there. Developing a library is a good learning experience, because you're thinking from the point of view of programmers, rather than users, and so you make nicer APIs. :-) |
This switches the import-all function to work with any number of images and without timing out, by paging through a user's photos and importing each photo. The API endpoints are renamed (as a new one is introduced). DO NOT MERGE YET. Refs: #12
@ElliotMcGucken I have started work on this, and would love any feedback. The PR is not yet complete, but is working for the "import all" function (it has broken some other things, that I'll fix tomorrow). If you could test it it'd be great! You just need to checkout the |
Thanks Sam! I installed your import-all branch in a brand new installation of Piwigo! It now sees all my photos, but it doesn't seem to be transferring them. After about 15 minutes of "processing," it still shows this: There is no sign of the new Piwigo database being populated, nor of the photos actually being transferred. Should I just wait longer for the transfer to begin? Also, if I stop it and restart it, should it still work just the same? Thanks! Elliot |
Hello Sam! I did another Fresh Install of your import-all branch in a brand new installation of Piwigo!! It is able to list all the albums after a couple tries/reloads and some errors. But then, when I try to import the photos, it doesn't seem to do anything--again it just hangs on this page, saying that it is "Processing:" Thanks! |
Hm, bother. :) Yep you should be able to stop and start all you like. Could you try opening the javascript console (pressing F12 usually does it) and reloading the page and seeing if there are any errors output there? I'll keep working on it. |
I think this is because of the deprecated function warning you're getting.
Thanks! |
The local config stuff is explained here: http://piwigo.org/doc/doku.php?id=user_documentation:use:features:conf_locale — basically it involves creating that file if it doesn't exist and putting the following in it: <?php
$conf['show_php_errors'] = 0; (Or you can do the equivalent setting in your php.ini file by setting the The problem is that the Oh and "Developer tools" is the name of that sidebar that appears when you hit F12. It's jolly useful, if a little confusing! |
This switches the import-all function to work with any number of images and without timing out, by paging through a user's photos and importing each photo. The API endpoints are renamed (as a new one is introduced). DO NOT MERGE YET. Refs: #12
Thanks Sam! Adding the file config.inc.php makes it work now:
|
Update! It froze after importing 275 photos and hung for ten minutes or so. So I restarted it, and it looks like it is starting the import all over from the very beginning? Is it actually re-downloading all the files again? Or is it skipping the ones it already has? If I stop it and hit refresh again, it starts all over again. Does this look right? P.S. It DOES look like it is re-downloading everything again, as it is eating up new disk space. |
it shouldn't re-download: it checks the file name of each photo before downloading. The Flickr ones are given names like I've updated the patch; you should redownload it. It will start the download from the beginning each time, note. But it'll go a lot quicker for the ones it's already got. Also, make sure to purge your browser cache when reloading the import page (ctrl-shift-r) to make sure it's running with the latest javascript. |
Oh, just to clarify, if you're running this from a git checkout, you'll need to:
because I'm doing a single-commit patch. |
This switches the import-all function to work with any number of images and without timing out by paging through a user's photos and importing each photo. The API endpoints are renamed (as a new one is introduced). Tag importing is fixed to use the 'raw' value rather than the spaceless normalized one. Refs: #12
Thanks Sam! Just to be clear, just now I made a brand new, fresh installation of Piwigo and your Flickr2Piwigo import-all branch. In order to install your Flickr2Piwigo import-all branch, I just now entered:
Is that enough to give me the latest version, on a brand new installation made just now? Or must I also enter:
And if I need to enter these commands too, can you please provide the full text of these commands for this case, without abstraction? Thanks! Elliot |
This switches the import-all function to work with any number of images and without timing out by paging through a user's photos and importing each photo. The API endpoints are renamed (as a new one is introduced). Tag importing is fixed to use the 'raw' value rather than the spaceless normalized one. Checksums of files are also checked before import, along with filenames. Refs: #12
Right, I think I've found and fixed a few issues. Your username has an ampersand in it, which was confusing the checking; the reporting back of checksum-matched already-imported photos was not working; and there was a chance that long album names were being truncated prior to creation, and so the creation was failing. The other thing I'd like you to check is the 'orphan' photos that get imported. If there's no album on Flickr, then the photo will be imported as an orphan. This is fine, but I think you won't be able to view it in the front end till you've given it an album. Could you have another crack with the latest code, and see how you go? :-) Thanks!! |
Thanks Sam! Here are the log files for today's install: http://66.45.225.82/20logs/ I created a brand new fresh install of everything today, and I'm seeing some of the same errors as yesterday (I think the errors are due to characters such as apostrophes and/or periods in the photo titles). Again, when it gets to the album:
It fails to create the album again. Just like yesterday, it creates the thumbnails: http://66.45.225.82/20pw/piwigo/index.php?/recent_pics/start-480 But when one clicks on a thumbnail, one gets the same error again: The titles of the photos are the same as the album title:
This character string has 251 characters in it. Might that be a problem after addslashes and all? Are the apostrophes ' in "St. Patty's" and "St. Patrick's Day" problems perhaps? Or are the periods in "St." a problem? I have long titles, but once your code is debugged for the long titles, it should be great! Orphan Photo Test but there is still no way to view these photos in Piwigo--they show up nowhere. ONLY photos in albums are seen anywhere: http://66.45.225.82/20pw/piwigo/index.php?/categories The orphan photos appear nowhere in "recent_pics" either: I then added the photos to a brand new album on Flickr, and I refreshed Flickr2Piwigo, but Flickr2Piwigo failed to create the new album. It still stated, "already imported" for all the photos, skipping them, but again, there is no way to view these photos at my Piwigo install. Here are the log files for today's install: http://66.45.225.82/20logs/ Hope this helps! Thanks, Elliot :) |
Greetings Sam! Flickr2Piwigo has been running fairly well for a few hours now! Here are the logfiles for up until this moment: It has been skipping many albums. Here are the Piwigo albums: And here are the flickr albums: It seems that many of the albums that are skipped are albums that were created some time after the photos were uploaded to flickr? It seems to do better with photos that are uploaded into albums, and/or at the same time that the albums are created. And too, it still gives errors: Here are the most recent logfiles which are up to date: I just stopped Flickr2Piwigo for the day. We're getting close! Thanks Sam! Elliot :) |
This switches the import-all function to work with any number of images and without timing out by paging through a user's photos and importing each photo. The API endpoints are renamed (as a new one is introduced). Tag importing is fixed to use the 'raw' value rather than the spaceless normalized one. Checksums of files are also checked before import, along with filenames. Logging is added throughout the import process. Refs: #12
This switches the import-all function to work with any number of images and without timing out by paging through a user's photos and importing each photo. The API endpoints are renamed (as a new one is introduced). Tag importing is fixed to use the 'raw' value rather than the spaceless normalized one. Checksums of files are also checked before import, along with filenames. Logging is added throughout the import process. Refs: #12
You can see orphans at I've added some more logging. I'm really at a bit of a loss as to what is going wrong! :-( It should be logging the titles of the albums it's creating, could you check to see if they're what you expect? It doesn't do anything about the creation time of the album, so that's not the problem. It only creates the albums as it encounters the photos in them, so it may not do them in any particular order. Sorry I'm not very helpful this evening. :-) I'll keep thinking and debugging. |
Thanks Sam! So basically Flickr2Piwigo imports photos in chronological order, as opposed to albums in chronological order? That means that if I uploaded photos a year ago, but just put them in an album created today, the album won't be created by Flickr2Piwigo until the photos are encountered during the chronological import? Were you able to fix this bug from the previous install:
What do you think about this question--"Are the apostrophes ' in "St. Patty's" and "St. Patrick's Day" problems perhaps? Or are the periods in "St." a problem?" I know you said the ampersand posed a problem in names/titles. What about the ' apostrophes and . periods? Thanks Sam! |
Yep, it goes in chronological order (of upload date), or at least it goes in whatever order Flickr gives us for the "people.getPhotos" API call. So albums will not be populated in order, but will be by the end. And no, it's a good thought, but slashes, apostrophes, quotation marks, periods, etc. are all fine in photo and album titles. I've tested with https://www.flickr.com/photos/153362204@N03/39172591000/ with no problems. Maybe there's some rate-limiting or something going on? :( |
Tags! Maybe it's tags with funny characters. :-)
|
This switches the import-all function to work with any number of images and without timing out by paging through a user's photos and importing each photo. The API endpoints are renamed (as a new one is introduced). Tag importing is fixed to use the 'raw' value rather than the spaceless normalized one. Checksums of files are also checked before import, along with filenames. Logging is added throughout the import process. Refs: #12
Yep, seems to have been one problem at any rate. Could you please test again with the latest patch? :-) |
Greetings Sam! Thanks! I just installed a brand new fresh install of everything, and here is the result: I didn't see any glaring errors, but when Flickr2Piwigo showed 500 photos imported in the "Processing 500," it hung and stopped importing. So I refreshed the browser, and it seemed to work fine, skipping all the photos that it had already imported. But then when it hit "Processing 500" again, it stopped again, so it hangs here at "Processing 500:" I refreshed the browser one more time, and again, it stopped at "Processing 500." I just copied the latest log files here: I feel we're getting close! Thanks so much! Elliot :) P.S. Have you seen these funny characters in your Flickr2Piwigo install? I've noticed them before too: |
This switches the import-all function to work with any number of images and without timing out by paging through a user's photos and importing each photo. The API endpoints are renamed (as a new one is introduced). Tag importing is fixed to use the 'raw' value rather than the spaceless normalized one. Checksums of files are also checked before import, along with filenames. Logging is added throughout the import process. A button is added to stop the import before it's finished. Refs: #12
Oops, I was checking "totalPages < currentPage" rather than "currentPage < totalPages"! :) Fixed now. Try again. I also made the stop button work a bit better (so it gets its own little popup message saying "stopped"). No idea about the funny characters I'm afraid. Do they appear when no plugins are turned on? If so, I reckon you should raise a bug against core Piwigo. |
Thanks Sam! I did a fresh install of everything, and it seems to be working well! I did notice just a few errors, but it keeps on going and keeps transferring images and all, despite these errors which seem isolated so far: Here are the log files after transferring around 2500 photos, and as it is still running: Flickr2Piwigo is still running & transferring--just one issue here: The Home/Recent Photos pages seems stuck at 2248 photos: Even though the photos are continuing to load and it says 2633 under the albums:
Here are the logs: Thanks! P.S.: Now and then the counter at the top of the page jumps to 2249 photos, but then it quickly jumps back down to 2248 photos: http://66.45.225.82/25pw/piwigo/index.php?/recent_pics/start-2235 Meanwhile, the number under the albums keeps growing as photos are imported: |
Hello Sam! It seems that the above problem is still centered about the same problematic album from earlier attempts, as Flickr2Piwigo is still unable to create this album: "Irish Lass St. Patty's Day! Cowgirl Swimuit Bikini Model! Beautiful Golden Ratio Composition Photography Surf Goddess! Athletic Action Portraits of St. Patrick's Day Bikini Model! Irish Athena, Artemis, and Aphrodite! Sexy Hot dx4/dt=ic! Cowboy Boots!" Note that Flickr2Piwigo imports the albums just before and after this problematic album circrled below at my flickr page: Note how the circled album above is missing here, while the other albums around it are present: And this time, after it hits the album during import, it won't add any more photos to the Home / Recent photos [2248] which is now stuck at 2248, as described above. Here is the album on flickr: Here is one of the photos: Are there any characters in the album title, or photo title, descriptions, or keywords which could cause a bug? Is the album or photo title or description too long? When I run the title through addslashes (https://www.tools4noobs.com/online_php_functions/addslashes/), I get:
So are the apostrophes a problem? Does the addslashes make the string too long? Could it be a problem that there are two apostrophes to handle in the above string? Is the title too long, or does it have too many apostrophes, periods, or exclamation marks? Are the apostrophes the problem somewhere in the code?
Is the slash a problem in the keywords in dx4/dt=ic? Is the album title too long perhaps?
This same album seems to keep breaking Flickr2Piwigo, and now it doesn't even look like it creates the thumbnails anymore for the album? So it seems like now the album and all its photos are skipped? Other than this album, things are looking great! :) Here are the latest logs: Thanks Sam! Elliot :) |
I think the errors being displayed might be something to do with being rate limited from Flickr's end. I'll add some better checking for that. The recent photos thing only shows photos posted within a day before the lastest-posted photo. So if you're importing posted dates from Flickr, the recent-photos view should max out at however many you uploaded within a day... but of course, that will vary as the latest-posted date changes (as other photos are uploaded). I'll look into the album naming a bit more. I think it shouldn't mind the length or funny characters, but will do some better checking. Thanks so much for helping debug all this! |
I wanted to test what Flickr did when I deleted an album (need to make sure that it does not delete what is in it, even though they tell you that in the help).
When I went to Flickr2Piwigo, the plugin says "No photo in the current set." Thank you, |
This may be due to the 10 minute cache time. Do you get the same error after either waiting a bit or deleting the `_data/flickr_cache` directory?
…--
Sent from my field telephone with K-9 Mail. Please excuse brevity.
On 26 March 2018 12:44:15 pm AWST, biffhero ***@***.***> wrote:
I wanted to test what Flickr did when I deleted an album (need to make
sure that it does not delete what is in it, even though they tell you
that in the help).
1. Uploaded one image.
2. Created an album.
3. Put that image in the new album.
4. Deleted the album.
-- At this point, my test with Flickr was finished. So I had a new
photo in my Flickr photostream that I needed to import with the
import-all branch of Flickr2Piwigo.
When I went to Flickr2Piwigo, the plugin says "No photo in the current
set."
Thank you,
Rob
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#12 (comment)
|
Thanks Sam! You write:
It is interesting to note that the album title is the same as the photo titles in this case, and that Flickr2Piwigo is both unable to create the album and unable to successfully display the full-size photos. Album name & photo title:
In order to see the thumbnails, do a search on "Irish" here: This is the page of thumbnails one sees: Now when one clicks on any one thumbnail, one gets this error: Hope this helps! Here are the latest logs again from the most recent import: Thanks, Elliot P.S. I searched for the word "Irish" in the log file (http://66.45.225.82/25logs.2/), and it does look like the album is being created? But then there is no such album here: I think that the album name is giving Flickr2Piwigo a problem somewhere:
Perhaps if you created an album and photos with this same title at your flickr, you'd see the error too? Here is the album on flickr: Here is one of the photos: Thanks! Elliot :) |
This switches the import-all function to work with any number of images and without timing out by paging through a user's photos and importing each photo. The API endpoints are renamed (as a new one is introduced). Tag importing is fixed to use the 'raw' value rather than the spaceless normalized one. Checksums of files are also checked before import, along with filenames. Logging is added throughout the import process. A button is added to stop the import before it's finished. Refs: #12
Ah! Yes, indeed, the apostrophes (or rather, my lack of understanding of various bits of the Piwigo codebase) were to blame. Fixed now, and I've tested with your erroring album title and it's worked. So, perhaps we're closer now! :-) Give it a test and see how it goes. |
Thanks Sam! I ran a brand new install today, and it now works for this "Irish" gallery too! http://66.45.225.82/28pw/piwigo/index.php?/category/21 It's working great! I have seen some random errors, which may be flickr timing out or something, but they seem fairly minor--here are today's logs: http://66.45.225.82/28pw/piwigo/_data/logs/ And here is what the minor errors look like: Looking great! One small request would be to put any photo which is not in an album into an album titled, "Not in an official album."! It's not a huge deal, but it seems it may be simple and useful to people on down the line, as sometimes flickr won't add every photo to an album, even though it makes it look like it did. Also, if two albums have the same name, can we please make sure it creates both albums? Perhaps it already does this? Thanks Sam! I am going to stop Flickr2Piwigo now and restart it from home, and let it run overnight. Will let you know how it goes. Thanks for all your work on this! It's looking great. Best, Elliot |
Hello Sam! Would it be possible to import and store the safety status of the photo from flickr, such as safe, moderate, or restricted? If this isn't possible or easy, would it be possible to import moderate and restricted photos as private, while importing the safe photos as public? Thanks! |
Tracked in #18.
No, it doesn't, and actually it's sort of tricky to do that because it checks to see if the album already exists by looking for its name. I think the easiest solution would be to rename one of the albums on Flickr prior to import. Do you have many like this?
Tracked as #17. Let's carry on discussion over on #15, given that we're pretty close to solving the problem at hand. :-) |
This switches the import-all function to work with any number of images and without timing out by paging through a user's photos and importing each photo. The API endpoints are renamed (as a new one is introduced). Tag importing is fixed to use the 'raw' value rather than the spaceless normalized one. Checksums of files are also checked before import, along with filenames. Logging is added throughout the import process. A button is added to stop the import before it's finished. Refs: #12
Thank you! I have just set piwigo up on a brand new Centos server! I hope to help you debug Flickr2Piwigo. At the moment, it is transferring files, with some issues:
Here is a screenshot!
Some things I've noticed:
Here is the site:
http://66.45.225.82/pw/piwigo/
For instance, it takes an album title from flickr which has 85 photos, chops it up, and creates five albums, each which has 85 photos:
http://66.45.225.82/pw/piwigo/
Albums:
**- and Beautiful! Athena! [85]
The good news is that it is transferring photos! Thanks!
Also, a question:
Is there a way to run Flickr2Piwigo from the linux shell or command line, rather than from a browser?
I have a dedicated server where I would like to leave it running so as to transfer all my images.
A related question to my above question is that if one closes the browser running Flickr2Piwigo and then opens a new browser on a new computer and runs Flickr2Piwigo, will Flickr2Piwigo continue where it last left off when the first browser was closed? Or will Flickr2Piwigo begin all over again, and download every gallery starting from the very first gallery again?
Thanks! :)
The text was updated successfully, but these errors were encountered: