Fix #709 - properly handle downloaded files with size mismatches

[reidsunderland]

Details in #709.

Before this change, files would only get renamed from their inflight name (e.g. filename.ext.tmp) when the downloaded size matched the expected size in the message.

When there is no size in the message, or the downloaded size is different, the files would not be renamed from their inflight name and later parts of the code would fail.

This ensures that the file gets renamed properly (and contentType gets set) when downloads are successful but the size is different than expected.

[petersilva]

while you are here... can you look at "acceptSizeWrong" ... there are statements to change the "warning: size wrong" to just a debug message and ignore it. What I am unsure about is... if the size is wrong and acceptSizeWrong is False... should we mark the download as unsuccessful and retry?

I think currently it only affects the priority of the message.... should it reject/fail/discard the download? when that option is set... not sure...

[petersilva]

I decided to accept as is and we should further discuss acceptSizeWrong.

[petersilva]

good.

[reidsunderland]

Right now when the downloaded file size is different than the size in the message, and acceptSizeWrong is false the download does get rejected and an error is logged:

sarracenia/sarracenia/flow/__init__.py

Lines 1856 to 1860 in 86ccad1

	else:
	logger.error(
	'incomplete download only %d of expected %d bytes for %s'
	% (len_written, block_length, new_inflight_path))
	return False

I think that's good. In cases where we know the size might be present in the message but incorrect (DMS, some polls), we can just set acceptSizeWrong True. That's an improvement from v2.

The other case is when the message doesn't have a filesize. That's what I ran into when I tried an sr3 sarra with my v2 NASA poll. That polls posts messages with no file size.

sarracenia/sarracenia/flow/__init__.py

Lines 1842 to 1850 in 86ccad1

	if block_length == 0:
	if self.o.acceptSizeWrong:
	logger.debug(
	'AcceptSizeWrong %d of with no length given for %s assuming ok'
	% (len_written, new_inflight_path))
	else:
	logger.warning(
	'downloaded %d of with no length given for %s assuming ok'
	% (len_written, new_inflight_path))

I'm pretty sure the way it works is the best compromise, we don't know if there's anything wrong with the download so we warn the user and proceed assuming that it did download okay. acceptSizeWrong True would stop the warning messages and might indicate to someone looking at the config that the person who set it up expects to receive messages without a file size.

The logic makes sense to me the way it is.