bugs.org

Short description: On host A, ~/.emacs.d/conf/cloud/individual.passes is saved in BJW.gpg, and when I switch to host B and run (cloud-start), the system thinks that ~/.emacs.d/conf/cloud/individual.passes is not yet clouded and assigns it another (CAP) cipher-name.

here is the log: ===============

Loading /home/shalaev/Private/work/done/cloud/tests/ert.el (source)… :debug 17:53:03.000 debug-set:old values –> :debug 17:53:03.000 root-test-dir= nil :debug 17:53:03.000 updated values –> :debug 17:53:03.000 root-test-dir = tmp/cloud.test.V75fnh Running 1 tests (2021-05-25 17:53:03-0400) Loading /home/shalaev/Private/work/done/cloud/tests/cloud.el (source)… :debug 17:53:03.000 debug-set:old values –> :debug 17:53:03.000 ~= home/shalaev :debug 17:53:03.000 localhost= kalinin :debug 17:53:03.000 *config-directory*= home/shalaev.emacs.d/conf/ :debug 17:53:03.000 updated values –> :debug 17:53:03.000 ~ = tmp/cloud.test.V75fnh/hostA :debug 17:53:03.000 localhost = hostA :debug 17:53:03.000 config-directory = ~/.emacs.d/conf/ :debug 17:53:03.000 1 local/host/conf= tmp/cloud.test.V75fnh/hostA.emacs.d/conf/cloud/hostA.conf :debug 17:53:03.000 1 configured= yes :debug 17:53:03.000 2 configured= yes entering cloud-start :debug 17:53:03.000 4 configured= yes

entering add-file entering cloud-locate-FN leaving cloud-locate-FN entering cloud-get-file-properties leaving cloud-get-file-properties entering new-file-in leaving new-file-in :info 17:53:03.000 file ~/.emacs.d/conf/cloud/individual.passes is now clouded leaving add-file entering reset-Makefile leaving reset-Makefile entering cloud-sync entering cloud-connected-p leaving cloud-connected-p entering TS leaving TS :info 17:53:03.000 first run: creating tmp/cloud.test.V75fnh/hostA.emacs.d/conf/cloud/hostA/all and AbC entering reset-Makefile leaving reset-Makefile entering write-all entering print-hosts entering backspace leaving backspace leaving print-hosts entering print-actions leaving print-actions entering format-file leaving format-file leaving write-all entering gpg-encrypt (Shell command succeeded with no output) leaving gpg-encrypt entering save-Makefile entering upload entering enc-make-stanza leaving enc-make-stanza entering make-cloud-older entering cip-ext leaving cip-ext entering cloud-get-file-properties leaving cloud-get-file-properties leaving make-cloud-older entering cip-ext leaving cip-ext leaving upload entering reset-Makefile leaving reset-Makefile leaving save-Makefile :debug 17:53:04.000 Makefile tmp/cloud.test.V75fnh/hostA.emacs.d/conf/cloud/cloud.mk :debug 17:53:04.000 containing HD=/tmp/cloud.test.V75fnh/hostA/

cloud=/tmp/cloud.test.V75fnh/mnt/remote/

password=”12345” gpg=gpg –pinentry-mode loopback –batch –yes enc=$(gpg) –symmetric –passphrase $(password)-odec=$(gpg) –decrypt –passphrase $(password)-olocalLog=$(HD).emacs.d/conf/cloud/hostA/log

MK=$(HD).emacs.d/conf/cloud/cloud.mk date=`date ‘+%m/%d %T’`

$(HD).emacs.d/conf/cloud/pass.d/updated:$(HD).emacs.d/conf/cloud/individual.passes awk ‘{print $$2 > “$(HD).emacs.d/conf/cloud/pass.d/”$$1}’ $<echo$(date) > $@-chgrp-Rtmp$(HD).emacs.d/conf/cloud/pass.d/*

$(cloud)BJW.gpg:$(HD).emacs.d/conf/cloud/individual.passes @$(enc) $@$&lt; -@echo “$$(date): uploaded $<”>>$(localLog)

all: /tmp/cloud.test.V75fnh/mnt/remote/BJW.gpg echo “background (en/de)cryption on hostA finished $(date)”>>/tmp/cloud.test.V75fnh/mnt/remote/history@sed‘s/12345/\ast \ast \ast \ast \ast \ast /g^{\prime }$(HD).emacs.d/conf/cloud/cloud.mk > $(HD).emacs.d/conf/cloud/cloud.mk.bak

:info 17:53:04.000 make started on 17:53:04.008 (Shell command succeeded with no output) :info 17:53:04.000 make finished on 17:53:04.716 :info 17:53:04.000 done syncing entering TS leaving TS leaving cloud-sync leaving cloud-start :debug 17:53:04.000 leaving hostA with these files in tmp/cloud.test.V75fnh/mnt/remote: . .. AbC.gpg BJW.gpg history Loading /home/shalaev/Private/work/done/cloud/tests/cloud.el (source)… :debug 17:53:04.000 debug-set:old values –> :debug 17:53:04.000 ~= tmp/cloud.test.V75fnh/hostA :debug 17:53:04.000 localhost= kalinin :debug 17:53:04.000 *config-directory*= ~/.emacs.d/conf/ :debug 17:53:04.000 updated values –> :debug 17:53:04.000 ~ = tmp/cloud.test.V75fnh/hostB :debug 17:53:04.000 localhost = hostB :debug 17:53:04.000 config-directory = ~/.emacs.d/conf/ :debug 17:53:04.000 1 local/host/conf= tmp/cloud.test.V75fnh/hostB.emacs.d/conf/cloud/hostB.conf :debug 17:53:04.000 1 configured= yes :debug 17:53:04.000 2 configured= yes entering cloud-start :debug 17:53:04.000 4 configured= yes

entering add-file entering cloud-locate-FN leaving cloud-locate-FN entering cloud-get-file-properties leaving cloud-get-file-properties entering new-file-in leaving new-file-in :info 17:53:04.000 file ~/.emacs.d/conf/cloud/individual.passes is now clouded leaving add-file entering reset-Makefile leaving reset-Makefile entering cloud-sync entering cloud-connected-p leaving cloud-connected-p entering TS leaving TS :debug 17:53:04.000 updating tmp/cloud.test.V75fnh/hostB.emacs.d/conf/cloud/hostB/all obsoleted by /tmp/cloud.test.V75fnh/mnt/remote/AbC.gpg entering gpg-decrypt leaving gpg-decrypt entering read-all entering cloud-host-add entering new-action :debug 17:53:05.000 new-action> clouded-hosts= hostB hostA entering format-action entering full-TS leaving full-TS leaving format-action :debug 17:53:05.000 new-action> “2021-05-25 17:53:05 EDT” 3 1 “hostB” hostA leaving new-action entering add-to-actions leaving add-to-actions leaving cloud-host-add entering str-to-DBrec entering parse-time leaving parse-time leaving str-to-DBrec entering plain-name leaving plain-name entering cloud-locate-FN entering plain-name leaving plain-name leaving cloud-locate-FN leaving read-all entering write-all entering print-hosts entering backspace leaving backspace leaving print-hosts entering print-actions entering format-action entering full-TS leaving full-TS leaving format-action leaving print-actions entering format-file leaving format-file leaving write-all entering gpg-encrypt (Shell command succeeded with no output) leaving gpg-encrypt entering save-Makefile entering upload entering enc-make-stanza leaving enc-make-stanza entering make-cloud-older entering cip-ext leaving cip-ext entering cloud-get-file-properties leaving cloud-get-file-properties leaving make-cloud-older entering cip-ext leaving cip-ext leaving upload entering reset-Makefile leaving reset-Makefile leaving save-Makefile :debug 17:53:06.000 Makefile tmp/cloud.test.V75fnh/hostB.emacs.d/conf/cloud/cloud.mk :debug 17:53:06.000 containing HD=/tmp/cloud.test.V75fnh/hostB/

cloud=/tmp/cloud.test.V75fnh/mnt/remote/

password=”12345” gpg=gpg –pinentry-mode loopback –batch –yes enc=$(gpg) –symmetric –passphrase $(password)-odec=$(gpg) –decrypt –passphrase $(password)-olocalLog=$(HD).emacs.d/conf/cloud/hostB/log

MK=$(HD).emacs.d/conf/cloud/cloud.mk date=`date ‘+%m/%d %T’`

$(HD).emacs.d/conf/cloud/pass.d/updated:$(HD).emacs.d/conf/cloud/individual.passes awk ‘{print $$2 > “$(HD).emacs.d/conf/cloud/pass.d/”$$1}’ $<echo$(date) > $@-chgrp-Rtmp$(HD).emacs.d/conf/cloud/pass.d/*

$(cloud)CAP.gpg:$(HD).emacs.d/conf/cloud/individual.passes @$(enc) $@$&lt; -@echo “$$(date): uploaded $<”>>$(localLog)

all: /tmp/cloud.test.V75fnh/mnt/remote/CAP.gpg echo “background (en/de)cryption on hostB finished $(date)”>>/tmp/cloud.test.V75fnh/mnt/remote/history@sed‘s/12345/\ast \ast \ast \ast \ast \ast /g^{\prime }$(HD).emacs.d/conf/cloud/cloud.mk > $(HD).emacs.d/conf/cloud/cloud.mk.bak

:info 17:53:06.000 make started on 17:53:06.146 (Shell command succeeded with no output) :info 17:53:06.000 make finished on 17:53:06.857 :info 17:53:06.000 done syncing entering TS leaving TS leaving cloud-sync leaving cloud-start :debug 17:53:06.000 leaving hostB with these files in tmp/cloud.test.V75fnh/mnt/remote: . .. AbC.gpg BJW.gpg CAP.gpg history

Errors

05/24 5th macro test fails

(easily fixable?) gpg-files are not auto-clouded when opened

05/10 create-save-delete: file remains

Create a file in emacs, save, then erase it. It will be created on other hosts because action (erase) comes before the file is downloaded from the cloud.

04/29 (occurs when adding/clouding multiple files) member: Wrong type argument: arrayp, nil

The problem is that it is not reproducible…

M-x cloud-add ~/learn/html

02/19 check if directories can sometimes be clouded

02/01 cloud-add allows unexisting files

(might be fixed on 02/01 by adding -k option to make) cloud-add may add a file that will not exist at the time of subsequent cloud-sync

01/21 Makefile is growing

(might be fixed on 01/27)

1. it is not cleaned after cloud-sync
2. the problem disappears after emacs is reloaded

01/21 cloud-forget does not cancel previously scheduled upload

might be fixed on 01/27

01/20 same files downloaded multiple times

• again and again, until they are not locally updated (and thus become younger than the remote ones).

Note that there is no similar problem with upload.

01/20 remote delete

does not work for clouded files. Deleted (on host A) file creates an action to be executed on host B, but on host B the file is uploaded instead of deletion.

.gz files

are uploaded, then clouded on another host, but are not downloaded there write test!

ghost actions

On one host I cloud-forget a file which was not clouded on another host (kalinin) So there was nothing to do on kalinin, but the action did not disappear, it multiplicated itself instead, so

(mapcar #'format-action remote-actions)

To do

1. Write tests for renaming or deleting files.
2. Probably I should use start-process or call-process instead of shell-command because start-process is claimed to be more versatile for elisp than (interactive) shell-command or async-shell-command. On the other hand, I had problems when using call-process for gpg encoding, see learn.org.
3. Files should be unclouded after, say, 30 days of being not updated. For such files, regular daily backup is enough.
4. Tired project
5. Garbage cleaning: some of the files in the remote directory are not referenced in ~/.emacs.d/cloud/`hostname`/all; Such “abandoned” files were created during debugging; they should be identified and erased. Similarly, some of the lines in ~/.emacs.d/cloud/individual.passes are unused and should be removed.

Ideas on future development

JPEGs and PNGs are encrypted with AES-algorithm that may become vulnerable if the same password is used for multiple images; this is why every image gets an individual password.

I did not expect this project to grow that much; some of the desired functions are still not implemented or half-implemented.

Replace clouds with torrents?

lftp supports torrents.

I will think of using torrents instead of clouds or using them together.

(At least when mounted using WebDav) clouds are probably even slowlier than torrents for large files: for example, it took me 13+ hours (!) to copy 403-megabyte archive to Yandex Disk. (This is not an exception – I am used to the fact that WebDav-clouds are very slow.)

I did not use torrents for looong time but I think they we faster even 10 years ago.

Advantages of torrents:

1. With torrents we can forget about space limitation we always have for clouds, and
2. The more people use torrents, the harder it is to enforce DMCA; probably 50 million of torrent users in the US would be enough to make it meaningless and powerless piece of paper.

Recently there was also a video about freenet which is probably even better than torrents.

Permutate image blocks

For every clouded image file, we create a sub-directory in the remote directory. This sub-directory will contain NxM small pieces (icons) of the image, where N,M≥12. Most of these pieces will be equally sized rectangulars, combined together into the inner part of the image. Others will be parts of the thin frame. On the one hand, N and M can not be to small; on the other hand each inner rectangular image should not be too small – at least 50x40 pixels – so that JPEG (or another) image compression algorithm remains efficient. This kind of protection is probably ok to protect innocent photos from face recognition robots. And in case someone writes an algorithm trying to guess the correct icons’ position, creating obstacles for it (so that decryption of a single photo would take at least an hour) seems easy.

Other ideas

1. convert runs without parameters (e.g., controlling jpeg quality) for now. Thus, for example, a low-quality JPEG file may be grow about 3 times larger after it was encrypted and then decrypted back.