Rewrite the extractor to use 7zip instead of zipfile

[msm-code]

Issue: we don't support newer encryption methods, because Python's zipfile can't handle them:

karton.archive_extractor.sflock.exception.UnpackException: Unknown zipfile error: That compression method is not supported

Solution: use 7z to extract files. The downside is that we lose sandboxing provided by sflock, but 7z exploit is IMO highly unlikely (I know, famous last words).

Other solutions: find a secure pure python library for zip files? Something else?

[conitrade-as]

The following change set works with pyzipper to support more recent compression methods and even AES-encrypted archives (e.g. from bazaar.abuse.ch/).

diff --git a/karton/archive_extractor/sflock/unpack/zip.py b/karton/archive_extractor/sflock/unpack/zip.py
index af08deb..2bcf407 100644
--- a/karton/archive_extractor/sflock/unpack/zip.py
+++ b/karton/archive_extractor/sflock/unpack/zip.py
@@ -5,7 +5,7 @@
 
 import os.path
 import six
-import zipfile
+import pyzipper as zipfile
 import zlib
 
 from ..abstracts import File, Unpacker
@@ -58,6 +58,8 @@ class ZipFile(Unpacker):
                 return
             if "compression type" in msg:
                 return
+            if "requires a password" in msg:
+                return
 
             raise UnpackException("Unknown zipfile error: %s" % e)
 
@@ -69,8 +71,24 @@ class ZipFile(Unpacker):
             self.f.error = e
             return []
 
+        # check if the archive is using AES encryption
+        use_aes = False
+        for entry in archive.infolist():
+            if entry.compress_type == 99:
+                use_aes = True
+                break
+
+        if use_aes:
+            try:
+                archive = zipfile.AESZipFile(self.f.stream)
+            except (zipfile.BadZipfile, IOError) as e:
+                self.f.mode = "failed"
+                self.f.error = e
+                return []
+
         entries, directories, total_size = [], [], 0
         for entry in archive.infolist():
+
             if entry.filename.endswith("/") or entry.file_size < 0:
                 continue

[psrok1]

Our sflock changes were just merged into @doomedraven's fork (CAPESandbox/sflock#7) that uses 7z+zipjail for zip files.

I think that problem will be solved as we switch to the https://pypi.org/project/SFlock2/ package.

[conitrade-as]

Even better! 👍

[doomedraven]

Hello, all, I know, this is not related but just headups, just updated sflock2, 0.3.16 and Commit

• I found bug in lzip implementation, it was giving error(see the bottom) but file was extracted, but as ret was empty it was returning empty list, so i moved that to temp folder, extracted and then listed files, removing original file from temporal folder and listen only new files. This exception is also reported to @jbremer.

• I have added tests for lzip and win files

"""
>>> from sflock import unpack
>>> q = unpack(b"test.lz")
(b'/usr/local/lib/python3.8/dist-packages/sflock/data/zipjail.elf', b'test.lz', b'', '-c=1', '--', '/usr/bin/lzip', '-d', b'test.lz')
b'' b'Blocked system call occurred during sandboxing!\nip=0x7f8e08e904fb sp=0x7ffe8e081698 abi=0 nr=93 syscall=fchown\n\x1b[1;34mKilling child 415983\x1b[0m\n' here
"""

[psrok1]

Released 1.2.0 that includes SFlock2. Problem with unknown compression methods should be fixed.