A sneak peak at a possible Compactor future

[Freaky]

So with the latest release of egui finally supporting native OS accessibility features like screen readers, I've been messing around mocking up a new UI for Compactor so we can get away from the use of WebView.

It's been going pretty well, I think:

[Freaky]

Some of this GUI actually works now. The Compact/Settings/About buttons select the appropriate pages, the Select Folders button lets you select folders without blocking the rest of the UI, the folder list is properly populated and lets you delete items, and as demonstrated, the little sun icon does in fact flip you over to light mode (it defaults to whatever you have the system set to).

Nothing actually useful yet, but we'll get to the rest of the owl in due course.

[Freaky]

Select Folders is now Add Folders and Add Files which add to the existing selection, with a Clear button to reset the list. You can also use drag and drop to add stuff to compact.

Bikeshed question: Should the file list be sorted, or just based off the order you added stuff?

The centred buttons aren't natively supported by egui, but done by rendering one frame of them left-aligned, and using their rendered width to apply an indent the next frame. Such is life in immediate mode - I can't say I've noticed any flicker.

[Freaky]

New icon I quickly hacked together. Resized rather poorly - sadly winit's icon support seems rather rudimentary and I can't pass multiple custom-resized versions in. Looks nice on my taskbar at least.

I got rid of the second Compactor - I had considered getting rid of the window frame, but reimplementing the chrome isn't well supported yet. The header looks unbalanced now, but I'm not sure if it's just because I'm not used to it.

Wrote some supporting unsafe Rust to get a feel for the more modern windows crate from Microsoft - this is what's detecting if a drive is an SSD and if the filesystem is supported. SSD/HDD detection will be used to decide how many threads to use, since HDDs are unlikely to benefit from more than one.

I also redid the output display so it'll scale to an arbitrary amount of data.

[Freaky]

And we have first light on compression/decompression - all the buttons now do stuff. I added a file, analysed, decompressed, analysed, recompressed, changed compression, decompressed, recompressed.

Again, all the unsafe code is new, using windows instead of the venerable winapi. It's a lot easier to be confident in this stuff when the company that makes the OS provides you with struct and function definitions instead of having to write them yourself by hand!

Still a fair way to go before it's anything more than a proof-of-concept - the buttons run stuff directly on the main thread, settings aren't saved, there's no file walking, etc. But there's a good chance you'll be able to use it to break your computer in time for Christmas.

[Freaky]

Hey, look - a settings page.

[Freaky]

Starting to look like a real app.

There's now filtering for minimum file sizes and ages, so you can skip all those tiny files that don't compress meaningfully or only compress files that haven't changed in the past hour/day/week/month.

The big list of default globs has been replaced with a sorted list of file extensions in a drop-down list and a text box for adding more. Globs remain for advanced filtering.

[Dr-Emann]

Just to plant a seed, probably way far out: I'm working on a rust version of https://github.com/RJVB/afsctool (basically, the equivalent of transparent compression for macos) and it'd be pretty cool to be able to be able to have compactor work on Mac too.

[Freaky]

That would be awesome. I like the look of what you've got so far!

[Freaky]

Making slow but steady progress, following some delays due to holiday debauchery, other projects, yak shaving, and general life stuff - including a short inpatient hospital stay (I'm OK, probably, but getting a scan to confirm).

I just redid the threading control stuff again - stop, pause, resume, etc. Third time lucky?

First was a simple wrapper around a crossbeam channel Receiver - kind of nice, but didn't offer any feedback for whether threads have actually noticed a state change - a similar issue to @Dr-Emann's earlier work on the old control loop, where Pause would not have an intermediate "Pausing" state while the worker was still busy.

Second was a generic ControlHandle/Token that could be fully shared and offered feedback on how many threads were active while having only minimal cost (a single atomic read to check state, and a mutex and condition variable for state changes). I kind of liked this and might extract it to another crate at some point.

It's in writing this that I realised that @Dr-Emann's optimization for the existing pause function has a short race condition:

The background worker thread runs this every iteration.

        while self.is_paused() && !self.is_cancelled() {
            paused = true;
            thread::park();
        }

And the foreground controller thread does this to unpause:

        self.control.resume();
        self.thread.unpark();

It's possible for the background thread to perform its atomic is_paused() read, realise it needs to pause, and for the foreground to then call resume() and unpark(), followed by the background running its thread::park(). This would cause activity to never resume (or at least wait until the thread spuriously unparks, or until the user hits Pause/Resume a second time). You must be THIS tall to write multi-threaded code.

Anyway, third time around yesterday was to rewrite the trivial ThreadPool I'd made to support dynamically changing the number of threads at runtime, because obviously it's really smart to pack in as much functionality as possible into the hairy multithreaded code I'm clearly not qualified to write. Not only does this mean you can save your config changes during operation and have them all actually be applied immediately(ish), but it enables Pause/Stop/Resume through just setting the target concurrency to zero and back.

I was also happy to find my code stopped around 300 lines in - about the same length as the now-unneeded second attempt.

And hey, if you very reasonably don't trust me to write 300 lines of multithreaded code, our friendly alternative CompactGUI implemented their multithreading in two words (Parallel.ForEachAsync). It's a bit less flexible, though. It can't be configured, it can't be paused. It doesn't feel pity, or remorse, or fear. And it absolutely will not stop.

[Freaky]

Ah, thanks. I've obviously forgotten all about how it actually works and defaulted to assuming it was more primitive than it actually is, and of course never bothered to check before mouthing off.

[Dr-Emann]

No problem, I had to do some checking to make sure, but I thought I remembered thinking about that exact scenario and justifying it somehow 😄.

[Dr-Emann]

I am glad I defaulted to SeqCst for is_paused/resume, relaxed wouldn't be enough. I think I could justify Aquire/Release being enough (it is what's used in the thread::park example, which is very similar to what we're using it for), but I wouldn't bet money on it.

[Freaky]

There's something almost comforting in the knowledge that even thread::park didn't get it right first try!

I think relaxed ordering is fine for this sort of use - it's a simple flag that isn't protecting any sort of shared state, and the park/unpark are Acquire/Release points anyway.

I think it's also fine for my current use, where the atomic represents a packed pair of u16's describing a target and current thread count. This allows workers to cheaply determine if they should exit (load, while target_threads < current_threads, compare/exchange with current_threads - 1 and exit on successs). Anything else is protected by a Mutex/CV that's taken every thread spin-up/down.

Both points are kind of academic on x86, which being strongly-ordered only offers SeqCst or Acquire/Release. But of course it's good not to have code that only works due to an accident of implementation.

[Freaky]

As further evidence of Relaxed being fine in this scenario, here's the recently-published book Rust Atomics and Locks using it in an example with a flag and park_timeout.

[TommyC81]

Hi there, just wanted to say good work on this! The changes you're looking into look very interesting and will add even more value. Thanks for your work on this!