Download starting with oldest posts and messages to newest?

[Chrdvr]

I notice that the files seem to download, especially in the messages, from the newest messages to the oldest. The titles help with sorting, but if I sort by date, the newest are oldest and oldest are newest based on their download order. Is it possible to allow adjusting download order?

[Avnsx]

All you have to do is click on the name column in windows explorer, so that it either sorts by oldest to newest, or newest to oldest.

Is that what you're asking?

[frederick-fraser-the-non-artificial-ai]

I was just about to make a post with nearly the exact same title. However, my question was about the duplicate protection feature.

After downloading ~500 of 700 videos, a storm knocked out my power. The duplicate protection feature is great, but I would prefer to not have to download all those videos again.

Would it be possible to have an option in the config for downloading starting from the oldest post first or start from a specific date?

Regardless of the answer, thank you for responding to issues and for all the time you've put into this project.

[Avnsx]

After downloading ~500 of 700 videos, a storm knocked out my power. The duplicate protection feature is great, but I would prefer to not have to download all those videos again.

You do not have to re-download all the videos again anyways.

Fansly-downloader will check the previous download folder for any previously downloaded content, recognise their hashes and then when you start it again, it will already upfront recognise that it previously downloaded X content and just skip it until it gets to the portion of the media that you've actually not already downloaded.

For your specific use case it would be benefitial to set utilize_duplicate_threshold to False though, since you've already downloaded a big part of the content and are trying to get the rest of the content. Read more about that config option here (option 12.)

[Avnsx]

Would it be possible to have an option in the config for downloading starting from the oldest post first or start from a specific date?

Also regarding this; no it would literally not be possible, simply due to the way the fansly API works. Basically the only way to download content from fansly, is by opening the latest page of contents from e.g. Timeline and then find a specific identifier string within it, which tells us what the next older page is. Only like this we're able to even know, which API endpoints even serve the content. To my knowledge it's not possible to somehow find the oldest "identifier string" first and then iterate from the oldest API endpoint, which servers the content, to the newest.

You can think of this similar to how you're browing the website, it's not possible for you to open the oldest content on the website first and then scroll up without having previously loaded all the more recent media already.

[frederick-fraser-the-non-artificial-ai]

Fansly-downloader will check the previous download folder for any previously downloaded content, recognise their hashes and then when you start it again, it will already upfront recognise that it previously downloaded X content and just skip it until it gets to the portion of the media that you've actually not already downloaded.

For your specific use case it would be benefitial to set utilize_duplicate_threshold to False though, since you've already downloaded a big part of the content and are trying to get the rest of the content. Read more about that config option here (option 12.)

Thank you so much for the quick reply!

I get the same result whether utilize_duplicate_threshold is set to True or False in the config.ini file, both result in the application stopping after ~40 declined duplicates.

Also, I forgot to mention that I'm on the latest Windows executable, v0.4.1, if that makes any difference.

[Avnsx]

If you set that option to "False" and saved the config.ini file with your changes, afterwards started fansly-downloader again, it's programmatically not possible for it to stop due to having received too many duplicates.

In-fact it would say something about Already downloaded all possible .... content! ! [Duplicate threshold exceeded ...] in the console, if it truly stopped because of too many encountered duplicates.

Is that the case for you?

[frederick-fraser-the-non-artificial-ai]

If you set that option to "False" and saved the config.ini file with your changes, afterwards started fansly-downloader again, it's programmatically not possible for it to stop due to having received too many duplicates.

In-fact it would say something about Already downloaded all possible .... content! ! [Duplicate threshold exceeded ...] in the console, if it truly stopped because of too many encountered duplicates.

Is that the case for you?

No, I get the same behavior either way.

These are the steps I took.

1. With the line utilise_duplicate_threshold = False in the config.ini,
2. the config.ini file is saved,
3. the config.ini file is then closed from the text editor, and
4. the Windows executable application started,

the end of output includes,

• Info | 20:10 || Deduplication is automatically enabled for;
E:\...\Fansly_Downloader_0.4.1-Windows\..._fansly
• Info | 20:10 || Deduplication process is complete! Each new download will now be compared
against a total of 28 photo & 568 video hashes and corresponding media IDs.
• ...
• WARNING | 20:10 || Low amount of Videos scraped. Creators total Videos: 770 | Downloaded: 0 and
• Finished Normal type, download of 0 pictures & 0 videos! Declined duplicates: 40,

which is the exact output I get with the option utilise_duplicate_threshold = True.

I realize this is fairly long-winded, but wanted you to know each step so that you can hopefully see where I screwed up.

Windows is weird sometimes. So, I'm about to restart computer and try after that.

If that doesn't work, I might just switch over to the Python version and take a crack at the code myself so as not to bother you any more.

EDIT: Restarting computer and trying again yielded the same result.

EDIT: Well, I'm dumb. Found that the issue isn't utilise_duplicate_threshold option at all. Played with the code by adding outputs for sanity checking and found a specific timeline_cursor that happens to yield 0 media after 40 duplicates had been found, stopping afterwards. The JSON returned has no objects in the returned fields, i.e. {'posts': [], 'accountMediaBundles': [], ....

EDIT(Final): Fixed the issue by adding a call to the sleep function at the end of sort_download if any duplicates were found in a given batch of accessible_media.

[Avnsx]

EDIT(Final): Fixed the issue by adding a call to the sleep function at the end of sort_download if any duplicates were found in a given batch of accessible_media.

Can you show a short code snippet with the change you did, for this to stop happening?

Also did I get it right, the reason that this was happening for you, is because when you already had downloaded so many duplicates previously, this made fansly-downloader skip over a entire timeline_cursor, pretty much instantly and repeadetly, so at some point you got basically rate limited and the API would entirely stop responding with anything at all? If that is the case can you remove your slowing down sleep timer and try to check if the base request has any response headers notifying you of being rate limited, once you no longer get any json object returns?

It should be timeline_req that may contain anything about being rate limited in its response headers:

fansly-downloader/fansly_downloader.py

Lines 1406 to 1427 in dd7104e

	timeline_req = sess.get(f"https://apiv3.fansly.com/api/v1/{endpoint}/{creator_id}?before={timeline_cursor}&after=0&wallId=&contentSearch=&ngsw-bypass=true", headers=headers)
	break # break if no errors happened; which means we will try parsing & downloading contents of that timeline_cursor
	except Exception:
	if itera == 0:
	continue
	elif itera == 1:
	output(2, '\n WARNING', '<yellow>', f"Uhm, looks like we\'ve hit a rate limit ..\
	\n{20 * ' '}Using a VPN might fix this issue entirely.\
	\n{20 * ' '}Regardless, will now try to continue the download infinitely, every 15 seconds.\
	\n{20 * ' '}Let me know if this logic worked out at any point in time\
	\n{20 * ' '}by opening an issue ticket, please!")
	print('\n' + traceback.format_exc())
	else:
	print(f"Attempt {itera} ...")
	s(15)
	try:
	if timeline_req.status_code == 200:
	accessible_media = None
	contained_posts = []
	post_object = timeline_req.json()['response']

you just do print(timeline_req.headers) and see if there's anything regarding rate limit or maybe print(timeline_req.content) to see what is in the responses body

[frederick-fraser-the-non-artificial-ai]

I just added a variable called prev_dupe_count to the other variables you declare a couple lines above the beginning of sort_download, added it to the global scoping inside of sort_download, and added an if block right below the for loop and above the ending comment in sort_download to sleep 5 seconds if any duplicates were found in the batch of media being processed.

pic_count, vid_count, duplicate_count, prev_dupe_count = 0, 0, 0, 0

...

def sort_download(accessible_media: dict):
    # global required so we can use them at the end of the whole code in global space
    global pic_count, vid_count, save_dir, recent_photo_media_ids, recent_video_media_ids, recent_audio_media_ids, recent_photo_hashes, recent_video_hashes, recent_audio_hashes, duplicate_count, prev_dupe_count

...

    if duplicate_count > prev_dupe_count:
        s(5)
        prev_dupe_count = duplicate_count

    # all functions call sort_download at the end; which means we leave this function open ended, so that the python executor can get back into executing in global space @ the end of the global space code / loop this function repetetively as seen in timeline code

There's probably a better way to do this, but it's been almost a year since I last did any programming. Sleep of 5 seconds seems to work as far as the limiting and took less than 5 minutes to churn through the 500+ declines.

Thank you again for the responses and this neat piece of software you made!

EDIT: Just tried it, and sleep 5 seconds isn't working for me right now.

[Avnsx]

If you want to continue talking about this topic, please head over to #148