Status/Plan for future Lantz?

[bilderbuchi]

It's been mentioned several times around 2015 that the lantz repo has been split into several repositories within this organisation. Has this split been completed, and is the lantz repo deprecated? Has the split stalled while in-progress? What's the future plan for those repositories (if any)?
Also, there is the https://github.com/lantzproject organisation now, is that going to be the future home of Lantz?

I am trying to revive the labpy organisation as a central place for discussing python lab automation, to transition away from the mega-issue at pymeasure/pymeasure#53

Pinging @hgrecco @alexforencich @MatthieuDartiailh

[MatthieuDartiailh]

As far as I am concerned this is dead. The split was never fully complete and the development effort stopped. @hgrecco created lantzproject about a month ago but I have no idea what he has in mind for it. However I see no issue in using this for discussions, and I wish you good luck in motivating people.

[bilderbuchi]

I wish you good luck in motivating people.

Thanks! I might be in over my head if I have to do that alone, I fear, the problem space is large and rereading/restarting/structuring the discussion yesterday was already quite taxing.. :-/

[vascotenner]

@hgrecco: What are your plans with the labpy organization? Can you add more developers to this group such that things will start running again?

[bilderbuchi]

@vascotenner @awsch I have noticed that you have been further developing lantz in your forks for quite some time. What are your plans concerning this? Are you planning/willing to upstream your changes? Is this something that can be upstreamed (into the labpy organisation)? If so, have you coordinated with @hgrecco already?

[hgrecco]

Hi there. Sorry for the long silence but it has been a very busy time. As you may have noticed, I faded way from this repo as I think that decisions should be made by people who contribute code. In my experience, design-by-committee does not work and I believe that is one of the reasons that the project stalled.

While I have never stopped worked with Lantz, in the last months I started to work again in Lantz

First:

• Migrated to Python 3.6 due to __init_subclass__ and __set_name__
• Moved a large part of the Feat/DictFeat/Action functionality to a new package pimpmyclass. This allows for easier testing.
• Renamed lantz-ui to lantz-qt and migrated to PyQt5 (as it is the default in anaconda)
• Splitted the package into multiple integrated subpackages using namespace packages. This allows for faster iteration.

After this initial work,

• Added mesage based feats to core (mfeats)
• Created a package to simplify the development of Python - Arduino instruments (lantz-ino)
• Added a lantz command line utility that each subpackage can add subcommands
• Added a way to create automatic simulators for drivers using mfeats in lantz-sims
• Improved the Frontend/Backend experience
• and lots of other small things

I have published everything under https://github.com/lantzproject and I am considering changing the name of the project to avoid confusion. If most of Lantz users are interested in this development, I will be happy to keep the name.

(regarding the LabPy organization, it is not really mine to decide what to do. It belongs to all of us).

[vascotenner]

@bilderbuchi @awsch : We are using Lantz in our lab every day. I have a contract for the next 2 years, and plan to keep using it. Until now I have added some additional drivers. I try to keep up with the branch from @awsch . I have written several PR's but they there is not responded on them.

@hgrecco: I understand from a developers perspective that it is better to split the packages over several repositories. However, to which extend will this obfuscate how to use, install and develop lantz for new users? Recently I have been following a "python in the lab" course from @aquilesC and there we discussed that if users new to python wants to use and develop software, it might be better to use "simple" language constructs (and hence use not all the power that the python language offers), to make sure that there is not much magic under the hood.

@hgrecco: I see that you already did some work porting the drivers to the new structure. Could you first do a pull from my or awsch most recent driver_develop(v) branch? Do you have a manual how to migrate to the new syntax?

[hgrecco]

@vascotenner Splitting into subpackages is not only good for the developer but also for the end user as they can see some subpackges updated more frequently. And they can opt-in to update them in their own installation. For example, the core will be updated less frequently than the drivers. And the user can update the drivers to get support for new instruments without fearing a breaking change (because the core will not change).

**But most importantly, there is no real difference for the end user to do a full installation: I have metapackage that just install the latest version of all packages. **

Regarding the use of advanced python constructs, my aim is to let the user do the right thing without repeating themselves too much. Some things are hard to get right (initialization, GUI, async). Using some accepted "magic" is the best way to this. Lantz is not doing anything weird (no monkey-patching, no eval, etc).

There is no new syntax in lantzproject and it is mostly backwards compatible. The only reason I have not release on PyPi is because I think it will confuse existing users.

[bilderbuchi]

It's good that there is some new activity!

@hgrecco I think it would be better to keep the name the same (if only for mindshare). If it should reside in labpy or lantz-project, I don't know and don't have strong opinions about (although again, I think staying with the known org has advantages).

I am not so sure about splitting into 4+ subpackages. While it has advantages in independent release cadence, I think this could turn into a disadvantage because it's much more complicated to keep the compatibilities between the respective subpackages in sync. Also it can be seen from the instructions for installing which become more complicated - 6 separate pip commands, which will probably also be the case when doing -e installations for developing).
Anyway, we will see how it turns out, hopefully I am wrong! :-)

Getting an update (after hopefully pulling in the new changes by @vascotenner / @awsch) on pypi would be great!
Seeing as you use anaconda, have you considered publishing on conda-forge, too? At my lab, I have managed to have most people on conda-only workflow, because most packages are available on conda-forge now (even pymeasure and dependencies!). It makes installing/upgrading so much easier if you don't mix conda and pip!

Is there a chance of compatibility/replaceability of the driver package with i3py? I think it would be great if one "definitive" driver package arose out of the confusion of different packages (see #23), that could be used by many different wrappers/layers/...
Where will the docs and the issue tracker live? How do you imagine the CI setup? Every subpackage on their own?

@vascotenner great to see that you are using it regularly and there is some future security for you so you can continue developing! Some semi-regular development effort would be great to grow such a project.

@hgrecco who else is in the labpy org beside you and @alexforencich to decide to add new members/admins and where to take the org?

[MatthieuDartiailh]

I left the org when I got kind of frustrated with the slow path and lack of response to my review requests. I believe @alexforencich is the only one left.

I plan to keep working on https://github.com/Exopy/i3py but I have been busy with other projects recently. Based on some comments, I guess it would seem too magical for some people even though I believe it is the right way to treat subsystems and channels.

[hgrecco]

@bilderbuchi

For the end user, installing something developed with subpackages can be as simple as:

pip install <exposed package name>

and this can trigger the installation of everything else. After a few months, I can tell you that developing with subpackages is much better for something like this.

Before pushing lantz project to pypi as it is now, I want to make sure that everybody agrees. I would rather change the name than disturbing people with a change like this. When I put lantz here I lost the prerogative to do it unilaterally.

[vascotenner]

@hgrecco: I just had a look to the new lantz feat class. This looks way cleaner, with much less repetition.

I fully agree with @bilderbuchi that keeping the seperate sub packages into sync. Consider the following scenario: I added some new functionality to lantz-core, and write a driver in lantz-drivers. Now one has to update both core and drivers. I do hardly forsee updates in core, withouth affecting the drivers, e.g. for every change in core one has to update drivers anyway. Hence for the end user it is much more convinient to update just one package, instead of 2 (or 5).

With the splitup of the packages, is it now possible to run lantz withouth qt installed?

[bilderbuchi]

Now one has to update both core and drivers

I don't know how much impact that will have - in my practical life, I do conda update --all anyway. :-D In addition, if one update necessitates an update in a companion package, that should be resolved with version constraints in setup.py and automatically updated in addition. Ideally the end user won't notice, if there is a good packaging/releasing process in place, they would just update lantz.
My point aimed more at the challenge of keeping a good understanding of the mutual compatibility constraints.
On the upside, the separated structure forces you to be more clean and careful in your dependencies between the respective parts of lantz, and so should enable better decoupling (like you noted concerning qt), which is a good thing!

[hgrecco]

@vascotenner The classes are much simpler now thanks to the new features of Python 3.6. Everything is much easier to reason about and reusable (take a look at pimpmyclass). Also Qt is no longer needed for lantz it is very easy to use to a lantz-qt even as an afterthough.

Regarding updating, It think it will not be cumbersome.

To basic end users that want everything just tell them to pip install -U lantz[full]. That's it. Notice that is just one package.

But advanced end users can have granularity. Let's define minimal lantz as core + drivers + sim

• Use pip install -U lantz to install/update minimal lantz
• Use pip install -U lantz[qt] to install/update minimal lantz and lantz-qt
• Use pip install -U lantz[ino] to install/update minima lantz and lantz-ino
• Use pip install -U lantz[full] to install/update minima lantz and lantz-qt and lantz-ino (and everything that might appear in the future.)

Notice again that is just one command.

And power end users can even install/upgrade the individual packages (e.g. they want to update to the current drivers but do not want to touch anything else, just tell them to pip install -U lantz-drivers)

Regarding assuring version compatibility, and constrains that is for us developers. We have to make sure that setup.py is right (as @bilderbuchi mentioned) and that is all. We can make some simple rules such as:

• lantz-qt/ino/sims/other can only depend on lantz-core
• lantz-qt/ino/sims/other will have the same major.minor version as the required lantz-core

Splitting such a large package into smaller ones that work together is the way to go. It creates a little bit more of work to the developer but it is worth it.

[bilderbuchi]

@hgrecco: I see that you already did some work porting the drivers to the new structure. Could you first do a pull from my or awsch most recent driver_develop(v) branch? Do you have a manual how to migrate to the new syntax?

@vascotenner this pull happened in lantzproject/lantz-drivers@5506938. I don't know the status/how much this has been tested already.

[vascotenner]

Perfect that this pull happened. Is it a real pull, the log says copy of new files. What about bug fixes of "old" files?

In that branch there are also some edits in other files, including feats.py. Changes sets:
feats.py:
vascotenner/lantz@6f4c068
vascotenner/lantz@d246108

Lantz monitor:
vascotenner/lantz@e466886

action.py:
vascotenner/lantz@4be7216

processors.py & test_feat.py
vascotenner/lantz@2a42718

Is there any conversion needed for that drivers?

Currently I am writing drivers for smaract motion stages. I will make a PR to the new lantz-drivers distro.

P.S.
All altered files are:

$ git diff --name-only --diff-filter=M 6fbff4dd293d4110d002d477d2642a7e75cc9ace HEAD
AUTHORS
lantz/action.py
lantz/drivers/newport/__init__.py
lantz/drivers/ni/__init__.py
lantz/drivers/ni/daqmx/__init__.py
lantz/drivers/ni/daqmx/base.py
lantz/drivers/ni/daqmx/channels.py
lantz/drivers/ni/daqmx/constants.py
lantz/drivers/ni/daqmx/tasks.py
lantz/drivers/stanford/__init__.py
lantz/drivers/tektronix/__init__.py
lantz/feat.py
lantz/foreign.py
lantz/processors.py
lantz/testsuite/test_feat.py
lantz/ui/widgets.py
lantz/utils/qt.py
lantz/utils/qt_loaders.py
requirements-full.txt
scripts/lantz-monitor
setup.py

[hgrecco]

@bilderbuchi Indeed I have merged the drivers (only the drivers) I noticed that there a few broken imports due to missing (re-)exports in the new lantz. I fixed that already if you use the version on github. Still, real testing (i.e. with the device) would be nice.

@vascotenner Thanks for pointing out this differences. Some are not needed any longer but I will look at them with more detail.

Regarding driver migration. Lantz 0.5 is almost fully backwards compatible regarding driver so migration is not needed. But Lantz 0.5 exposes mfeats (MessageBased Feats) which allow writing for message bases instrument much easier. The API is almost stable.

[bilderbuchi]

@hgrecco regarding tests, it would be great to build up a test suite to have some checks even without devices present. I opened a separate issue for this, see lantzproject/lantz-drivers#2.