Support creating .torrent with larger piece size #19535
Conversation
It seems the latest Transmission, Deluge, BiglyBT, and uTorrent support 64MB along with qBitorrent so it's really just about older versions not supporting it. It's up to leechers to keep their clients up to date if they want to continue participating in swarms. |
|
PR updated, I've added an experimental commit that would allow creating .torrent files up to 1 GiB piece size. This is the max limit of libtorrent. I just thought that why don't we provide every options at once and let users take the responsibility (about the creation result). |
Totally support this and I will ping other devs to support this. |
Chocobo1
changed the title
|
lt 1.2: When creating a torrent with a piece size of 256 MB or more, a memory leak occurs. The client crashed several times. In cases where the client did not crash, the log writes |
I didn't observe it on my machine. Wild guess, would it be possible that it exhausted all your system memory?
I checked the created .torrent file and it seems to be valid. It is the torrent client (or specifically, the code responsible for loading the .torrent file) that is faulty. And this can be resolved later (probably with some help from libtorrent, not sure). Also, the error message is slightly improved. |
|
The client crashes after a leak, 16 GB RAM + 10 GB pagefile. There is no memory leakage on 128 MB, memory is used at a minimum. |
Just curious, how do you know it is a leak and not other problems? |
Perhaps applications like dottorrent only creates torrents up to 64MB for that very reason? |
Actually, I do not know this, but it is very similar to a memory leak.
|
|
CI libt 1.2 build on Windows. I created a torrent with 1G 512M and 256M pieces sizes. No crash for 512M and 1G. First time with 256 it crashed (qBt closed) but second time it didn't. But because i didn't get a stacktrace, the crash may have been caused because i had virtualbox running which causes some programs to crash for me sometimes. |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Yes, it was necessary to put in quotation marks from the very beginning. I do not know how to explain this and why it happens only with lt 1.2 and only from 256 MB+ |
I found an issue within libtorrent 2.x, I'll submit a fix. This issue prevents torrents with 1 GiB piece size from being added to transfer list. Smaller piece size should already be fine and libtorrent 1.2.x isn't affected.
Not sure but I find it unlikely.
OK, I retest it with a larger data and now I see that libtorrent 1.x is not efficient with memory usage. It sucks away all the memory. I'm going to limit the max piece size to 128 MiB when using libtorrent 1.x. Only libtorrent 2.x will be able to use 1 GiB piece size. |
Chocobo1
changed the title
Warning: users are at their own discretion to create .torrent with >= 64 MiB piece size as not every torrent client supports it. Larger piece sizes are only available when using libtorrent 2.x. libtorrent 1.x is not efficient with memory usage and in order to avoid user complaints it is limited to 128 MiB. Also note that, as of this writing, libtorrent 2.0.9 has an internal limitation that only allows loading maximum 256 MiB piece size. And therefore > 256 MiB size options are forbidden for now. Closes qbittorrent#19527.
|
After more investigations, it seems that libtorrent 2.0.9/1.2.19 has an internal limitation that only allows loading maximum 256 MiB piece size. And therefore > 256 MiB size options are forbidden for now. |
Probably for the best. This is already a huge step up from the 32MB piece size we mostly see in the wild. Even 64MB is still rare to see. |
Warning: users are at their own discretion to create .torrent with >= 64 MiB piece size as not every torrent client supports it. Larger piece sizes are only available when using libtorrent 2.x. libtorrent 1.x is not efficient with memory usage and in order to avoid user complaints it is limited to 128 MiB. Also note that, as of this writing, libtorrent 2.0.9 has an internal limitation that only allows loading maximum 256 MiB piece size. And therefore > 256 MiB size options are forbidden for now. Closes qbittorrent#19527. PR qbittorrent#19535.
Exactly, if you look at the torrents that were supplied with 256.0 MiB piece sizes - they were created with a program called py3createtorrent v1.1.0 What needs to be done is a fix for libtorrent 1.2.x branch or reject torrents of this type from being loaded via libtorrent 1.2.x based builds. |
Yeah, but at least something... |
@Chocobo1 Do you mind pointing me in the direction of where this limitation is in libtorrent code & is there something similar for libtorrent 1.2.19? |
|
Can we compromise and just make the maximum piece size 128MB? That seems like enough considering almost all torrents in the wild are 32MB or below. |
|
Warning: users are at their own discretion to create .torrent with >= 64 MiB piece size as not every torrent client supports it.
Larger piece sizes are only available when using libtorrent 2.x. libtorrent 1.x is not efficient with memory usage and in order to avoid user complaints it is limited to 128 MiB.
Also note that, as of this writing, libtorrent 2.0.9 has an internal limitation that only allows loading maximum 256 MiB piece size. And therefore > 256 MiB size options are forbidden for now.
Closes Support Creation of torrents with 64 MB piece size #19527.