Crappy 2.0.0 is out !

[WeisLeDocto]

Hi there !

I'm glad to announce that the version 2.0.0 of Crappy was just released earlier this week !
It represents almost one year's worth of work, and brings major backwards-incompatible changes to the module at many levels.

This post describes the rationale of the update from version 1.5 to 2.0, and lists the most notable changes to Crappy in the new version. Guidelines for updating your code from version 1.5 to 2.0 are also given, and in the last section I give a few words about the short- and long-term perspectives of the project.

But first, I'd like to thank @PIERROOOTT who's been actively contributing to the project for a few months already !
Don't hesitate to provide any feedback on version 2.0.0, either by replying to this Discussion or creating a new Issue if you spot anything wrong.

Why a version 2.0.0 now ?

When @Dad0u stopped maintaining Crappy a bit more than a year ago, I became de facto the new maintainer. In my opinion, the project at that time suffered several major flaws:

• It was not so well documented, although targeting an audience with beginner to intermediate code level
• It was not fully meeting the recent Python coding standards (PEP8, typing, etc.)
• It was -still IMO- lacking some features a library driving hardware should possess (stability at runtime, logging)
• The project as a whole was not following the best practices of a stable and well-maintained Python module (structure of the repository, test suite, layout of the git branches)
• There were a few critical limitations in the Python module itself, especially for code driving cameras

All my efforts during the past year were directed towards improving the module on these points. On each of these aspects there is currently still room for improvement, but much has already been done. I will certainly keep working on Crappy until I'm fully satisfied, probably somewhere around version 2.2-2.3. Hopefully, the next maintainer -if there's any- will inherit a clean, stable, well-structured, and well-documented project.

What are the main changes ?

In line with the avenues of improvement listed above, here is a condensed list of the main changes brought to Crappy as a Python module and as a project in version 2.0.0. For a more detailed change list you can refer to the release post, and you'll get the full picture by looking directly at the changelogs. The main changes, in no specific order, are:

• The replacement of the old Camera Block and of the Displayer Block by a single new Camera Block that handles the acquisition, processing, display, and recording of the images. All these operations are now parallelized, when they were previously performed sequentially, therefore leading to a significant increase in performance.
• The complete refactoring of the base Block, making full use of the possibilities offered by the multiprocessing module, that drastically improved the stability at runtime. The most notable enhancement is visible when a script finishes, either due to an exception or because of a CTRL+C interruption. The termination of scripts used to be particularly unstable, it is now most of the time very smooth.
• The various calls to the print function were replaced by a proper logging mechanism. It provides a much richer feedback, both in the console and in a log file, and users are free to tune the logging level for each Block independently.
• A number of classes, methods, attributes, argument, functions and files were renamed to match the PEP8 specification. This change alone is the source of 90% of the backwards incompatibility of version 2.0.0.
• The structure of the Python module was refined, with the creation of numerous sub-modules for a clearer separation between files serving different purposes. Many long files containing several classes were also split into smaller files, again for more clarity. These changes almost didn't affect the API.
• The structure of the project was changed from flat to src, which is now an increasingly popular project layout (See this post for a discussion about the advantages of the src layout.).
• The root of the project was almost entirely changed: some folders were moved, other renamed, other deleted, and other added.
• Users can now create custom Generator Path objects. That was previously not possible.
• The packaging-related files (setup.py, requirements.txt, etc.) and the configuration files for the documentation (.readthedocs.yml, docs/source/conf.py, etc.) were updated to the recent standards for their respective formats.
• Terminating a script in Crappy with CTRL+C is now deprecated. Users should instead rely on the new StopBlock and StopButton Block. Otherwise, Crappy will (by default) raise an exception when returning, even if the termination went smoothly.
• The examples/ folder of the project was totally refactored, and now features a much richer collection of example scripts. Each script is heavily commented, and most scripts can now run locally without requiring hardware (using fake hardware classes instead).
• The documentation of the project was brought to a whole other level. Each object exposed in the API is now properly documented on ReadTheDocs, and most of the code in the project in now extensively commented. In addition, the tutorials in the documentation were fully re-written to provide more information, and cover more features of Crappy.

How to upgrade your code from version 1.5 to 2.0 ?

As stated above, many changes from version 1.5.11 to 2.0.0 are backwards-incompatible ! In the general case, code running fine with version 1.5.11 will not run on version 2.0.0. Changes of that magnitude should definitely have been better advertised in the previous releases, and I apologize if some users have a hard time upgrading their code. The reasons why there were so few announcements about the big changes of version 2.0.0 are: a lack of perspective on my work, the scale of the changes, and my little experience as a maintainer. The future improvements will be handled in a much smoother way !

In practice, the first thing you should do before upgrading to version 2.0.0 is to first upgrade to 1.5.11 (python -m pip install --upgrade crappy==1.5.11). When running your code, you should see FutureWarnings signaling the classes and methods that were renamed in version 2.0.0. If you run your old code in version 2.0.0, it will also raise ad hoc exceptions signaling deprecated names and the replacement ones to use. Such exceptions were not implemented for the names of the arguments to pass, so you'll need to check the API section of the documentation to see if the old argument names are still valid. This should be clearly indicated (e.g. "Changed in version 1.5.10: renamed P argument to gain"). In most cases, upgrading your code will simply be a matter of correcting a few names.

If your code uses the Camera Block, or if you defined your own Blocks, then the upgrade will be a bit more painful. I cannot give clear guidelines that would fit every situation, so my best advice in that case would be to get familiar with the new API for the Block and Camera Block objects. The tutorials about creating custom Blocks, using the Camera Block, and creating custom Camera Blocks can surely be of great help to you. These steps are a bit of a pain, but the upgrade is totally worth the trouble !

What's next ?

As I mentioned earlier, I consider the few goals that I set for Crappy to only be partially met with version 2.0.0. Therefore, I still have quite some work up ahead for the coming months. You can now get an overview of the planned changes in the Projects section on GitHub. In broad outline, these changes will include:

• The addition of a test suite, running as a GitHub workflow, to better detect errors in the code. The tests will only run on the base objects (Block, InOut, Modifier, etc.), because implementing such test for code driving hardware is not realistically feasible. Note that the tests/ folder of the project already contains a draft version of it.
• The addition of a more flexible Camera Block, able to direct streams arbitrarily between the different processes (acquisition, recording, display, processing) and to perform several processing tasks at once (see Create a more flexible Camera Block #61).
• The complete removal of legacy objects from Crappy, and the transfer of many of the supported hardware classes to a separate module partially independent from crappy (probably something like crappy.collection, on the same model as matplotlib.pyplot). The goal is to keep in crappy only the core of the module, and a few hardware classes that will be actively maintained and promoted. On the other hand, crappy.collection will be a collection of hardware classes, with less support from Crappy's maintainer(s). It is IMO a nice solution to keep extending the hardware library of Crappy, while keeping the core of the project reasonably small.

[Dad0u]

Amazing work @WeisLeDocto and @PIERROOOTT ! I am happy to see that not only Crappy was actively maintained after i stopped working on it, but it kept evolving to become a much more accomplished project. I hope that the LaMcube and many other users will benefit from this powerful toolbox !