|
| 1 | +--- |
| 2 | +layout: post |
| 3 | +title: "dircmp.py - a plan to improve and extend" |
| 4 | +categories: unix python |
| 5 | +--- |
| 6 | + |
| 7 | +## dircmp.py |
| 8 | + |
| 9 | +### A plan to improve and extend |
| 10 | + |
| 11 | +This note pertains to [dircmp.py](https://github.com/decuser/decuser_python_playground/blob/master/dircmp/dircmp.py) a program that I wrote to give me information about two directories for the purpose of deciding what to keep and what to remove and to learn about python. The note is a draft note and as such it's not very refined and it may lack in many ways, but I thought it might be interesting to put it out there and let anyone see it. Email me if you have comments or suggestions. |
| 12 | + |
| 13 | +<!--more--> |
| 14 | + |
| 15 | +#### Caveats |
| 16 | + |
| 17 | +The comparisons here are between simple folders and files (including hidden files). By simple, I don't mean small - I use the program to compare very large directories. However, the tool was developed for comparing directories containing user files and not as a system maintenance tool. I haven't done much investigating of hard / soft symlinks or exotic setups. |
| 18 | + |
| 19 | +#### Random Observation |
| 20 | + |
| 21 | +When originally creating the program, I had the thought that git's organization would provide an ideal filesystem for keeping changes and being able to see those changes easily. Just keep file contents in blobs, their digests, names, and locations elsewhere. Files with identical contents would share the blob and digests, but the names and locations could differ. When saving a new file, do a digest, check the registry, link to the blob, etc. Sure, it'd be slow, but integral. A little above my paygrade, but percolating in the back of my mind. |
| 22 | + |
| 23 | +#### Features under consideration |
| 24 | + |
| 25 | +* Synchronization Planning |
| 26 | +* Synchronization Execution with Undo and potentially segregation |
| 27 | + |
| 28 | +#### Other potential enhancements |
| 29 | + |
| 30 | +* Historical comparisons (save results and use for future compares) |
| 31 | + |
| 32 | +#### Future research |
| 33 | + |
| 34 | +* symlinks and exotic setups :) |
| 35 | + |
| 36 | +#### Background |
| 37 | +Currently, the program does a great job of identifying differences and matches between two directory trees. However, it does not do a good job of providing the user with plans to synchronize those trees or the ability to synchronize directories. |
| 38 | + |
| 39 | +When I started this project, my goal was to identify, not address, so the program met its objectives. Now, I want to have the program generate plans to synchronize and perform the synchroniztion. |
| 40 | + |
| 41 | +#### Current functionality |
| 42 | + |
| 43 | +Looking at the program as a whole and as a black box, it takes a directory or pair of directories and compiles a set of results... |
| 44 | + |
| 45 | +##### Results |
| 46 | + |
| 47 | +Pair Results Report |
| 48 | + |
| 49 | +* duplicate files found in either directory |
| 50 | +* exact matches found in both directories |
| 51 | +* files that only exist in one of the directories |
| 52 | +* files that have the same names but different digests |
| 53 | +* files that have different names but same digest |
| 54 | + |
| 55 | +Single Results Report |
| 56 | + |
| 57 | +* duplicate files found in directory |
| 58 | + |
| 59 | +##### Method of Operation |
| 60 | + |
| 61 | +The comparisons are done using a calculated digest for every file that exists within the scope of the comparison, either a single level, or recursive. |
| 62 | + |
| 63 | +##### Options |
| 64 | + |
| 65 | +The program supports the following options for controlling its behavior: |
| 66 | + |
| 67 | +* -h, --help - show a help message and exit |
| 68 | +* -b, --brief - Brief mode - suppress file lists |
| 69 | +* -a, --all - Include hidden files in comparisons |
| 70 | +* -r, --recurse - Recurse subdirectories |
| 71 | +* -f, --fast - Perform shallow digests (super fast, but necessarily less accurate) |
| 72 | +* -d, --debug - Debug mode |
| 73 | +* -c, --compact - Compact mode |
| 74 | +* -s, --single - Single directory mode |
| 75 | +* -v, --version - show program's version number and exit |
| 76 | + |
| 77 | +#### Discussion |
| 78 | + |
| 79 | +All in all, it works great. I've used it a lot. It's fast and accurate. However, in using it, it has become apparent that what I really want it to do is tell me how to get two directories to synchronize. |
| 80 | + |
| 81 | +I have used rsync (prolly best of breed) and tried many, many other programs to quickly and easily sync directories, and I haven't liked any of them, in the end. Usually, I wind up losing files that I don't want to lose, sometimes through unintentional misuse of the tool, especially with rsync's arcane syntax, but usually through an inability to figure out what the tool actually does (not how it does what it does, but what the results are) and thinking two directories are synced after running the tool only to find out later that they weren't ... not exactly. |
| 82 | + |
| 83 | +At least with dirsync.py as it exists today, I know exactly what it is doing. The results are detailed and precise. It lives to tell me, in detail, what differences exist between two directories. With this information in hand, it is possible to determine a finite plan to synchronize them. |
| 84 | + |
| 85 | +Interestingly, synchronization can be accomplished with several distinct outcomes, as follows. |
| 86 | + |
| 87 | +#### Synchronize how? |
| 88 | + |
| 89 | +In the following discussions, I will use left and right to differentiate the two directories being discussed and will only discuss synchronizing two directories. |
| 90 | + |
| 91 | +##### One Way Synchronizations |
| 92 | + |
| 93 | +* Left to Right - *right directory is made to exactly match the left directory* |
| 94 | +* Right to Left - *left directory is made to exactly match the right directory* |
| 95 | + |
| 96 | +Conflicts arising in one way synchronizations are resolved by order definition - files and directories from one side are chosen whenever there is a mismatch. |
| 97 | + |
| 98 | +##### Two Way Sychronizations |
| 99 | + |
| 100 | +Whenever two way synchronizations are performed, there is a likelihood of conflicts and it is important to consider strategies to resolve those conflicts. This is where synchronization gets tricky. |
| 101 | + |
| 102 | +Here are the possible strategies |
| 103 | + |
| 104 | +* Preserve None - *remove conflicts from left and right (neither win)* |
| 105 | +* Preserve Left - *merge left into right (left wins)* |
| 106 | +* Preserve Right - *merge right into left (right wins)* |
| 107 | +* Preserve Both (versioning) *merge both ways (both win) and create versions when there is a conflict* |
| 108 | + |
| 109 | + |
| 110 | +##### Thoughts before diving into the details |
| 111 | + |
| 112 | +I think that inline with providing undo, it may be useful to preserve conflicts for the user... as in, when there's a conflict, move the loser (one side, or both) into a separate area (preserving prior location information) for the user to decide what to do with. Given a robust enough functionality, this may be moot, but I remember doing this sorta thing before and it being useful. |
| 113 | + |
| 114 | +#### Rough sketch strategy |
| 115 | + |
| 116 | +* Analyze directories to determine what needs to change |
| 117 | +* Report status |
| 118 | +* Stage changes |
| 119 | +* Make changes (as economically and safely as possible) |
| 120 | + * Stage a modification |
| 121 | + * Save recovery information |
| 122 | + * Make the modification |
| 123 | +* Report changes |
| 124 | + |
| 125 | +#### Thoughts related to the economics |
| 126 | + |
| 127 | +The expenses in this program are the costs of computing digests, comparing those digests, and copying files. Deletions are cheap, as are moves. So, the program should only compute digests as requested. When fast mode is active, the algorithm only reads a portion of the file, rather than the entirety, so this must be taken into account. Comparisons of the digest are mandatory. Copying files is expensive and should be minimized. |
| 128 | + |
| 129 | +Interestingly, when I started looking at this part of the code, I figured out that my fast digest approach probably needs to be improved. My premise, when I wrote it was that big files tend not to change in small ways over time - movies, images, and such. So, files over 10MB were considered candidates for an optimization of the digest process... This has proved to be a good intuition, but there are certainly some exceptions that could cause problems with the simple approach currently in the code (read the file size in bytes, read the first 1MB and the last 1MB and use these for the digest). I remember coming up with a strategy more along the lines of 100MB being the threshold and taking 100MB of random samples from the file. I don't remember why I changed it, but it was prolly just a matter of it taking too long and/or being somewhat more complicated to implement for the time I set aside to do the work... either way, the current code is quite basic, but fast... and it's worked fine because the only large files that I've used it on have been normal user files that simply don't change much in the middle. Still, this is definitely an area to optimize. The easiest example of a file that would be problematic that I can think of is a VM drive file... When in doubt, don't use fast mode :). |
| 130 | + |
| 131 | +#### Stuff to think about |
| 132 | + |
| 133 | +* Fast digest - what's a better approach that's still fast (sampling is slow, but is it necessary)? |
| 134 | +* I seem to remember counting being challenging - does the program count correctly or does it need a fix? |
| 135 | +* Symlinks are weird, but are they a problem? |
| 136 | +* How to develop a changeset - linear, in a single pass, what? |
| 137 | +* How to handle the undo functionality |
| 138 | +* What to do about destructive changes - save the to be destroyed file somewhere |
| 139 | +* How to handle duplicate versioning - naming, save somewhere |
| 140 | +* Given past experience, what to do about voluminous reporting - definitely need better delineation of sections (very hard to differentiate in terminal) |
| 141 | +* Tkinter? I dunno, I prefer something like avalonia, but that's .net, still should this have a ui? |
| 142 | +* Order to do the coding - left to right and right to left first, then merges, which preservation strategies in what order? |
| 143 | + |
| 144 | +The playground has the latest code and branches [https://github.com/decuser/decuser_python_playground](https://github.com/decuser/decuser_python_playground) |
| 145 | + |
| 146 | +*post last updated 2022-12-15 17:53:00 -0600* |
0 commit comments