Design flaws of StorageManager

[jennydaman]

The StorageManager's methods are mostly blocking, eager, and buffered, which are problematic when data gets big.

ChRIS_ultron_backEnd/chris_backend/core/storage/storagemanager.py

Lines 24 to 28 in 3f6858b

	def ls(self, path_prefix: str) -> List[str]:
	"""
	:returns: a list of all files under a given path prefix.
	"""
	...

• StorageManager.ls should return Iterable[str] instead of List[str]

ChRIS_ultron_backEnd/chris_backend/core/storage/storagemanager.py

Lines 42 to 56 in 3f6858b

	def upload_obj(self, file_path: str, contents: AnyStr, content_type: Optional[str] = None):
	"""
	Upload file data to the storage service.
	:param file_path: file path to upload to
	:param contents: file data
	:param content_type: optional media type, e.g. "text/plain"
	"""
	...
	def download_obj(self, file_path: str) -> AnyStr:
	"""
	Download file data from the storage service.
	"""
	...

• StorageManager.upload_obj should accept a BytesIO instead of AnyStr

• StorageManager.download_obj should return a ContextManager[BytesIO] instead of AnyStr

• StorageManager.delete_path could take a while, so there should be a mechanism for monitoring its progress

• StorageManager.move_path could take a while, so there should be a mechanism for monitoring its progress

• StorageManager.copy_path could take a while, so there should be a mechanism for monitoring its progress

[jennydaman]

Related: #517