Is there a proper way files are zipped to be put on s3, for Haystack to read them in?

[asharm0662]

Hi,

I am trying to zip and read in my own files from s3 bucket using the Tutorial 1 for Haystack here. I can read in the zip file that is specified in the tutorial, no problem on my local setup: s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt.zip"

When I zip my own files and move to s3 bucket, and try reading in the zip file using this code, I get error:

from haystack.preprocessor.utils import convert_files_to_dicts, fetch_archive_from_http
doc_dir = "Desktop/data/article_txt_data"
s3_url = "https://s3.us-east-1.amazonaws.com/example_bucket/example_zipfile.zip"

Error: `BadZipFile: File is not a zip file`

Machine specs:
Mac osx Big sur, M1 chip (if that makes a difference)
Python3

The way I zipped the file was:

1. create a directory on my desktop.
2. put non zip file into directory
3. open terminal
4. cd into directory
5. run zip -r data.zip . -x ".*" -x "__MACOSX" -x ".DS_Store" (remove any hidden files and output 'data.zip' to same directory)
6. upload data.zip to s3.
7. Test and get error BadZipFile

Then I ran this function, i found here

def fixBadZipfile(zipFile):  
 f = open(zipFile, 'r+b')  
 data = f.read()
 pos = data.find(b'\x50\x4b\x05\x06') # End of central directory signature  
 if (pos > 0):
     print('yes')
     #self._log("Trancating file at location " + str(pos + 22)+ ".")  
     f.seek(pos + 22)   # size of 'ZIP end of central directory record' 
     f.truncate()  
     f.close()  
 else:  
     print('not working')
     
fixBadZipfile('path/to/zip/file.zip')

8. and reuploaded the new zip, reran the same code but same error again.

Regardless of what I try, I get the same error when I try to read in my own zip files. Is there something specific I need to do to zip my own files to be put on s3 so I dont get the BadZipFile error for Haystack specifically?

Thank you in advance.

[bogdankostic]

Hey @asharm0662! My guess is that your file is not publicly available by everyone and therefore, the zip-file is not downloaded. If you open the s3-url in your browser, will it download the file or do you get an "AccessDenied" error?

[asharm0662]

that worked , thank you!