when v2 receives a message from sr3 with no size set, it will likely truncate the file

[reidsunderland]

When v2 receives a message from sr3 where the size is not set, it seems to download a much smaller file than expected. I've observed this with HTTP at least, I'm not sure if other protocols are affected.

We already knew about this, but I don't think it's documented (I took a quick look through the open issues).

I don't think it's worth fixing. You can either use sr3 to download the file, or post the size in the message. If the size is unknown, just post (from sr3) a size much larger than the expected size and use v2 plugin on_file_repair_size to set the correct size after downloading.

For example, this sr3 plugin can be used to set a fixed, large size:

import logging
from sarracenia.flowcb import FlowCB

logger = logging.getLogger(__name__)

class Force_size(FlowCB):

    def __init__(self, options) :
        super().__init__(options, logger)
        self.o.add_option('forceSize', kind='size', default_value="10GB")

    def after_accept(self, worklist):
        for msg in worklist.incoming:
            msg['size'] = self.o.forceSize

[petersilva]

What does sr3 do when receiving a message with no size set?

[reidsunderland]

It works correctly. It downloads the full file and (assuming a download plugin is not used), it sets the size in the message to the size of the downloaded file.

If acceptSizeWrong is not set to true, it will also log a warning:

[WARNING] sarracenia.flow download downloaded 358532 of with no length given for FILENAME assuming ok

[INFO] 2354666 sarracenia.moth.amqp putNewMessage raw message body: version: v02 type: <class 'str'> 20250305172402.342253685 POST_BASEURL tmp/avalanche/FILENAME

[INFO] 2354666 sarracenia.moth.amqp putNewMessage raw message headers: type: <class 'sarracenia.Message'> value: {'atime': '20250305172353.744154453',  'mode': '666', 'mtime': '20250305172353.744154453', 'source': 'bunnymaster', 'to_clusters': 'ALL', 'contentType': 'application/octet-stream', 'parts': '1,358532,1,0,0', 'topic': ['v02', 'post', 'tmp', 'avalanche'], 'sum': 'n,17b1fbe043f48f4b22ee96cab4d90d0c'}

[INFO] 2354666 sarracenia.flowcb.log after_post posted to exchange: xs_bunnymaster topic: v02.post.tmp.avalanche a file with baseUrl: POST_BASEURL relPath: tmp/avalanche/FILENAME size: 358532

[petersilva]

OK so a perfect work-around is: migrate the subscriber to sr3. cool. definitely not worth fixing.