Update size parameter to accept callable for callback.set_size()

[OneSizeFitsQuorum]

Allow 'size' parameter to be a callable that returns an int. So that we can set size properly here

[martindurant]

This is maybe OK, but I didn't understand the original issue.

Certainly, this PR should add documentation about how to use this change, and a test showing it in practice.

[OneSizeFitsQuorum]

Thanks a lot for reviewing this! Maybe we can change this line from callback.set_size(getattr(f1, "size", None)) to callback.set_size(getattr(f1, "size", None)()) in here.
or add a wrapper function to ensure the parameters passed to set_size is a int, then we do not need update set_size(). What's your opinion?

[OneSizeFitsQuorum]

@martindurant hi, could you please review this again~

[martindurant]

I guess this is fine, but I still don't know why size would be a function: can you make a test case that hits this extra line?

[OneSizeFitsQuorum]

@martindurant HadoopFileSystem inherits from ArrowFSWrapper, and ArrowFSWrapper inherits from AbstractFileSystem. However, AbstractFileSystem does not have a size field, only a size function. Therefore, when you use getattr(f, "size", None) to retrieve it, you get a callable function rather than a field. You need to invoke the function to obtain the corresponding size value.

[martindurant]

Yes, agreed that size is a method on filesystems, .size(path)->int; I think the .size you are after is an attribute on a file object, though - no? e.g., https://github.com/fsspec/filesystem_spec/blob/master/fsspec/spec.py#L1920 .