Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix for #15 for compressing large files #16

Merged
merged 1 commit into from
May 6, 2016

Conversation

lesthaeghet
Copy link

This is a potential quick fix for Issue #15 which occurs when attempting to compress files larger than ZIP64_LIMIT. zinfo.file_size is never initialized to the correct file size and thus the determination of whether zip64 is required is made based on the file size of 0. This later results in an exception being raised as though the file size increased during compression, since the file size is actually counted during compression and later saved over zinfo.file_size.

It is important to note that this fix may not be cross platform. Different versions of Python do different things on Windows with the st_size parameter in the os.stat call. So, that may be worth investigating further. However, in the short term, this will fix the problem on Linux, Mac, and some Windows platforms without making it worse where it still doesn't work.

I would leave it to the maintainers to make a broader decision on whether this fix is appropriate or if a better solution would be desired. I'm happy to help.

This is a potential quick fix for Issue allanlei#15 which occurs when attempting to compress files larger than ZIP64_LIMIT.  `zinfo.file_size` is never initialized to the correct file size and thus the determination of whether zip64 is required is made based on the file size of 0.  This later results in an exception being raised as though the file size increased during compression, since the file size is actually counted during compression and later saved over `zinfo.file_size`. 

It is important to note that this fix may not be cross platform.  Different versions of Python do different things on Windows with the `st_size` parameter in the `os.stat` call.  So, that may be worth investigating further.  However, in the short term, this will fix the problem on Linux, Mac, and some Windows platforms without making it worse where it still doesn't work.

I would leave it to the maintainers to make a broader decision on whether this fix is appropriate or if a better solution would be desired.  I'm happy to help.
@lesthaeghet
Copy link
Author

Alternatively, replacing st[6] with os.path.getsize(filename) is a cross-platform solution. I'm happy to make that change to the commit if you would prefer. I'm just not sure if there aren't any corner cases where non-existent files or anything like that are handled or expected to be handled. So, I'll defer to the judgment of the maintainers. Let me know if you'd like me to make any changes to my commit.

@rmeyers4
Copy link

rmeyers4 commented May 4, 2016

This fix works if st[6] is replaced with os.path.getsize(filename), as shown above, and allowZip64=True is passed as an argument. I was able to download and decompress a 2.5GB file that was previously failing to unzip.

@allanlei allanlei merged commit 2ebe493 into allanlei:master May 6, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants