Encryption and chunk file read

[salvatorenitopi]

Hi, i think that this is a very smart idea, I have some possible enhancements for you:

1. reading and computing file in chunks
2. adding AES encryption (in CBC mode)

To be more specific I would read the file in blocks long as the size of a page, then I would encrypt the block and upload it on Google documents (as a page)

A very simple example:

with open(path, 'rb') as f:
    for chunk in iter(lambda: f.read(4096), b''):
        enc = encrypt_fx (chunk)
        upload_page_fx (enc)

Let me know what you think.

[salvatorenitopi]

Sorry for bad indentation in my example code. And by the way I meant "encryption" in the title.

[stewartmcgown]

@salvatorenitopi Good idea! I reckon it will allow us to process larger files without the enormous memory footprint needed currently :-)

[stewartmcgown]

Multithreading added with 13079d5. Encryption will be next!

[salvatorenitopi]

Nice, I would use AES - CBC as encryption method (example: https://gist.github.com/forkd/168c9d74b988391e702aac5f4aa69e41 to edit according below)

I would encrypt a page in this way (CFB example):

Block0 -> random IV
Block1 -> AES ( xor (x1, Block0) , key)
Block2 -> AES ( xor (x2, Block1) , key)
BlockN -> AES ( xor (xN, BlockN-1) , key)

(Where x is the plaintext)

Multithreading may not be possible using this method.

[stewartmcgown]

Multithreading should be possible with this :-)

[jarmo]

Why not create a possibility for piping data into uds so that you could encrypt with any tool, which reads file in and prints encrypted output to stdout? In that way you would eliminate the need to have encryption built into the uds itself and everyone could use whatever encryption they like.

[Fuco1]

Using uds as a backend for git-annex would be also a pretty easy way to achieve this. But it comes with all the complexity of that tool attached.