Handling Uploaded Files

Uploading files to the app server is accomplished with the jQuery-File-Upload plugin.

• The pre-/post-upload templates for adding a file are located in add_file.html and those for updating a file in fileupload_update_templates.js.html

• The binding/implementation is located in fileupload.js.

During Request Creation

Only 1 file can be uploaded during Request creation with a size limited to 20 MB (only by Parsley in new-request-agency.js).

Upon submitting the New Request form with a chosen file the following occurs:

1. the file's mime type is validated against the system's default, and possibly a custom, magic file
2. the file is saved to quarantine as a named temporary file
3. the temporary file is scanned for viruses

If any errors are encountered during 1-3,

4. the server will respond with updated form field errors

otherwise

4. the upload is considered valid and its file metadata is stored in redis
   • this metadata is supposed to later be used when creating the associated File response record, but the mime type, size, and hash are currently being fetched from the file itself
   • while retrieving the metadata from the redis store rather than the file is a faster operation, the difference is negligible due to the 20 mb upload size limit
5. the file is moved from quarantine to the uploads directory, with its original name restored, under a directory named after the associated FOIL Request ID
   • e.g. uploads/FOIL-2017-860-00001/the_uploaded_file.ext

Through the "Add Files" Action

1. jQuery-File-Upload is bound to the "#fileupload" form.
2. Upon initiating an upload or group of uploads, POST requests containing up to 512 kb of file data (each pertaining to one file) are sent to the upload/<request_id> endpoint.
3. The uploads directory for the associated request is checked for any existing files with the same name as an uploaded file and, if any matching files are found, the upload is cancelled.

If the upload is smaller that the 512 kb size threshold (jQuery-File-Upload does not supply a "Content-Range" header in the request),

4. The following occurs:
   1. the file's mime type is validated, unless the current user is an active agency user
   2. the upload's status is stored in redis as "processing"
   3. the file is saved to quarantine
   4. the file is scanned and then moved to the upload directory, a process described in greater detail in step 7 below

otherwise,

4. With the first, and only the first, 512 chunk:
   1. the file's mime type is validated (only the first few bytes of a file are required to determine its type)
   2. the quarantine directory is checked for the existence of a file by the same name
   3. if an identically-named file is discovered, it is deleted and the upload is considered to have been re-initiated
5. The upload's status is stored in redis as "processing"
6. The chunk data is saved to quarantine; it is appended to an existing file after the first chunk is saved.
7. After the last chunk has been saved, the following is executed as a celery task:
   1. the now completed upload's status is set to "scanning" and the file is scanned for viruses
   2. the file's metadata is stored in redis
   3. the file is moved from quarantine to the uploads directory
   4. the upload's status is set to "ready"

If an exception is caught during or after step 3, the upload's status is set to "error".

After a file has finished uploading but before it has been moved to the uploads directory (step 4.iv or 7), the client will continually poll the upload/status endpoint until an upload status of "ready" is returned.

Cancelling/Deleting an Upload

When a user cancels or deletes an upload, or a chunk fails to upload, the client sends a DELETE request to the upload endpoint.

The location of the uploaded file or partial file is determined through the FOIL Request ID and filecode provided in the request and the upload's current status as it exists in the redis store.

Once the file has been found, it is removed.

While Editing a File Response

The upload handling process when making changes to a File Response is identical to that of adding files with the following exceptions:

• jQuery-File-Upload is bound to the "#fileupload-update-*" form for each File in the Responses section.
• The POST upload/<request_id> request body includes update: true and response_id: <id>
• After virus-scanning, the upload is moved to an "updated" directory within the uploads directory.
  • e.g. uploads/FOIL-2017-860-00001/updated/the_uploaded_file.ext
  • The file is eventually moved out of updated and into the FOIL Request ID directory, where it replaces the original file (but that is beyond the scope of this wiki page's topic!)