When using a download plugin, the size doesn't get updated/checked

[reidsunderland]

I'm not sure that this behavior is wrong. It does seem to cause v2 compatibility issues though.

When a download plugin is used, the part of the code that compares the downloaded file size with the expected size in the message is not executed.

This means that there's no check to see if the downloaded size is wrong, and if there's no size already set in the message from the upstream source, the sarra/subscriber will not set it. This causes issues for downstream v2 subscribers due to #1391.

There's a lot of ways we could change this, if we want to.

Some options that come to mind are:

• Change the return value of FlowCB.download to return a status code and downloaded size (could be optional), then modify the main flow code to look at the returned size.
• Add a separate after_work plugin that re-implements most of the size checking code from flow. (easy, but redundant, repeated code)
  • More involved: This could be a built in plugin that is added by default when download is enabled, and then we could remove the size check code from Flow.download. (fixes the redundancy issue)
  • This might make download retry handling more difficult, I haven't looked at the code in enough detail to check.

sarracenia/sarracenia/flow/__init__.py

Lines 2236 to 2307 in 372e77b

	if not self.o.dry_run:
	try:
	if accelerated:
	len_written = self.proto[self.scheme].getAccelerated(
	msg, remote_file, new_inflight_path, block_length, remote_offset, exactLength)
	#FIXME: no onfly_checksum calculation during download.
	else:
	self.proto[self.scheme].set_path(new_inflight_path)
	len_written = self.proto[self.scheme].get(
	msg, remote_file, new_inflight_path, remote_offset,
	msg['local_offset'], block_length, exactLength)
	except Exception as ex:
	logger.error( f"could not get {remote_file}: {ex}" )
	return 0
	else:
	len_written = block_length
	if ('blocks' in msg) and (msg['blocks']['method'] == 'inplace'):
	msg['blocks']['method'] = 'separate'
	if (len_written == block_length):
	if not self.o.dry_run:
	if accelerated:
	self.proto[self.scheme].update_file(new_inflight_path)
	elif len_written < 0:
	logger.error("failed to download %s" % new_file)
	if (self.o.inflight != None) and os.path.isfile(new_inflight_path):
	os.remove(new_inflight_path)
	return 0
	else:
	if block_length == 0:
	if self.o.acceptSizeWrong:
	logger.debug(
	'AcceptSizeWrong %d of with no length given for %s assuming ok'
	% (len_written, new_inflight_path))
	else:
	logger.warning(
	'downloaded %d of with no length given for %s assuming ok'
	% (len_written, new_inflight_path))
	else:
	if self.o.acceptSizeWrong:
	logger.info(
	'AcceptSizeWrong download size mismatch, received %d of expected %d bytes for %s'
	% (len_written, block_length, new_inflight_path))
	else:
	retval=0
	if hasattr( self.proto[self.scheme],'stat'):
	current_stat = self.proto[self.scheme].stat( remote_file, msg )
	if 'mtime' in msg:
	mtime = sarracenia.timestr2flt(msg['mtime'])
	else:
	mtime = sarracenia.timestr2flt(msg['pubTime'])
	if current_stat and current_stat.st_mtime and current_stat.st_mtime > mtime:
	logger.info( f'upstream resource is newer, so message {msg.getIDStr()} is obsolete. Discarding.' )
	retval=-1
	elif current_stat and current_stat.st_size and current_stat.st_size == len_written:
	logger.info( f'matches upstream source, {msg.getIDStr()} but not announcement. Discarding.' )
	retval=-1
	else:
	logger.error( f"unexplained size discrepancy in {msg.getIDStr()}, will retry later" )
	elif len_written > block_length:
	logger.error( f'download more {len_written} than expected {block_length} bytes for {new_inflight_path}' )
	else:
	logger.error( f'incomplete download only {len_written} of expected {block_length} bytes for {new_inflight_path}' )
	if (self.o.inflight != None) and os.path.isfile(new_inflight_path):
	os.remove(new_inflight_path)
	return retval
	# when len_written is different than block_length
	msg['size'] = len_written

Edit: I think this might be the same for identity, it doesn't appear to get set when the download plugin is used, but I need to adjust the options in the config and look at the code to be 100% sure