Interfacing Crappy

[Elbub]

(follow-up of these issues and questions)

There's still a few questions/things to discuss, so let's do it here.

I completely forgot to answer your last question :

I'm not sure to completely get it, why couldn't you just append or not the pre-stretch and release phases to the list of Generator path, depending for example on the values of checkboxes ?

People here are used to the old software that has one button for the pre-stretch, one for the release phase and one for the main test. It allows people to manually drive the bench (with the remote) to where they want without necessarily using the pre-stretch or release functionalities. Since these two functions aren't always used, or are sometimes used and complemented with manual driving between them and the test, I can't just append them to the main list of paths.

Aaah ok, I hadn't got it that your goal was to code an interface displaying the shape of all the successive paths. Could be a nice contribution to Crappy by the way. Just saying.

Displaying the shape, not exactly, but it's something I'll try to implement if I've got the time (which I doubt, at least during my internship, but maybe later). For now, we have a window showing subframes with one path each. Each subframe contains a label explaining it, such as "3 cycles of ramps : 1.2T/s for 6s, -4T/s down to 2T", a button to modify it and one to suppress it. I'll try to prepend a basic image of the shape. I can e-mail you the code now, but it's still a bit rough, or two, three weeks, if you don't need it right now.

PS : I'm pretty new to using Github actively, so didn't really know about this section.

[WeisLeDocto]

There's a solution that we use in some of our setups for driving motors from Python before the real test starts, e.g. for adjusting the distance between the jaws of tensile test machines.
The trick is to simply instantiate the desired Actuator object, but not through a Machine block.
And then to call the standard methods open(), then set_position() or set_speed(), etc.
There's an example here, at lines 31 to 35. It might be a good solution for your pre-stretching and release phases.

I can e-mail you the code now

Well if you're allowed to share the code, I strongly recommend you to upload it on GitHub ! I might be wrong, but it's also probably a good way to share it internally in your laboratory on the long term and in a reliable and accessible way. IMO the few extra hours it would cost you are totally worth it.

Weis

[Elbub]

That could be a solution if we had access to the actuators, but we don't. We have access to the card and it manages the actuators, but we don't control them directly.

Creating a github for that could be great to have feedback and help, but the thing is that it would have to be managed by an employee of the company, not an intern like me, but nobody knows much about coding here except me. I'll discuss it with my colleagues and see what's the best option.

Another day, another problem : yesterday, since we got a working NI-DAQmx back, we tried to run some tests of my program, but can't update the drivers of the card. It's caused by mirrors problems, and no solution other than "retry in a few hours/days and it'll work" worked out. Thus, I tried coming back to windows, since it shouldn't encounter the problems faced before anymore, since Crappy is launched from another process. However, now, we get a pickling error when hitting the crappy.start() instruction. Moving the crappy_launcher function (target of a thread to parallelize it) to a module other than the main didn't change anything. I've got no idea where this lock comes from, and apparently it's the only thing to remove to make the code work. Here's the trace :

Exception in thread Thread-7 (crappy_launcher):
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "C:\Program Files\Python310\lib\threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "c:\Users\R&D\Desktop\Stage Inorope\Pour les tests\SoftC10TL27_avec_modif.py", line 2531, in crappy_launcher
    customblocks.demarrage_de_crappy_fake_machine(consignes_generateur = consignes_du_generateur,   
  File "c:\Users\R&D\Desktop\Stage Inorope\Pour les tests\customblocks.py", line 672, in demarrage_de_crappy_fake_machine
    crappy.start()
  File "C:\Users\R&D\AppData\Roaming\Python\Python310\site-packages\crappy\blocks\block.py", line 217, in start_all
    cls.prepare_all(verbose)
  File "C:\Users\R&D\AppData\Roaming\Python\Python310\site-packages\crappy\blocks\block.py", line 151, in prepare_all
    instance.start()
  File "C:\Program Files\Python310\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Python310\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python310\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Python310\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__  
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\Python310\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object

[WeisLeDocto]

Sorry I didn't specify it, but the trick of instantiating Crappy objects will work with any type of object, not just Actuators. So it will also work fine with InOuts, Cameras, Tools, and even Blocks under certain conditions. With Blocks it would work only if the block doesn't need to receive or to send data. It would also work a bit differently, as you would need to call :

import crappy

block = your_block(*args, **kwargs)
block.prepare()
while your_condition:
    block.loop()
block.finish()

crappy.reset()
...

Regarding the bug you're encountering, I somehow managed to reproduce it although I'm not sure if it's representative of your specific situation. Here's the code raising a similar error, based on the code provided in this issue. In this example the problem is that a Process/Block to be started has a Threading object as an attribute. Only pickable objects can be passed to processes, and most Threading objects are not pickable, hence the error.

So, make sure your Blocks do not have any Threading objects as attributes. If after double-checking it the error is still there, please try to provide the minimal code that raises this error. Also, check if the structure of your project is similar to the one of my example or if there are differences.

Weis

[Elbub]

After hours of searching and testing on different Linux distros, a colleague finally found something deeply hidden on NI's website stating that the specific DAQmx we have has no available driver on anything else than windows. So, definitively back to windows and its bugs.

Nice trick to know. I'm gonna try that later, but I'm wondering : if Crappy's already running (we need a recorder for the whole thing, not just the auto-driven part of the test), wouldn't crappy.reset() mess with the rest ?

Edited : About that thread bug, the dashboard is causing it. Adding a Dashboard block makes that bug appear, even when it only has an empty list as labels. There's no thread attribute, so I don't understand Why it doesn't work. It raises the same error whether I put it in the fake_machine test or the card-driven one, and nothing changes whether it's linked or not. The only other tests I ran with a Dashboard block were on Linux, and it worked perfectly, so it must be Windows related. I'll keep digging to see if I can solve it.

I've finally put my code on Github, though it's still messy. The relevant part is here :

def demarrage_de_crappy_charge(consignes_generateur = None, fichier_d_enregistrement = None,
                        parametres_du_test = [], labels_a_enregistrer = None):
   """TODO : doc"""
   gen = crappy.blocks.Generator(path = consignes_generateur,
                                 cmd_label = 'consigne',
                                 spam = True,
                                 freq = 50)

   carte_NI = crappy.blocks.IOBlock(name = "Nidaqmx",
                                    labels = ["t(s)", "sortie_charge", 
                                             "sortie_deplacement"],
                                    cmd_labels = ["entree_decharge", "entree_charge"],
                                    initial_cmd = [0.0, 0.0],
                                    exit_values = [0.0, 0.0],
                                    channels=[{'name': 'Dev1/ao0'},
                                    {'name': 'Dev1/ao1'},
                                    {'name': 'Dev1/ai6'},
                                    {'name': 'Dev1/ai7'}],
                                    spam=True,
                                    freq = 50)

   pid_charge = crappy.blocks.PID(kp=1,
                                 ki=0.01,
                                 kd=0.01,
                                 out_max=5,
                                 out_min=-5,
                                 i_limit=0.5,
                                 target_label='consigne',
                                 labels=["t(s)", 'entree_charge'],
                                 input_label='sortie_charge_transformee',
                                 freq = 50)

   pid_decharge = crappy.blocks.PID(kp=0.5,
                                    ki=0.0,
                                    kd=0.0,
                                    out_max=5,
                                    out_min=-5,
                                    i_limit=0.5,
                                    target_label='consigne',
                                    labels=["t(s)", 'entree_decharge'],
                                    input_label='sortie_charge_transformee',
                                    freq = 50)

   y_charge = customblocks.YBlock(out_labels = ["t(s)", "consigne", 
                                                "sortie_charge_transformee"],
                                 freq = 50)

   y_decharge = customblocks.YBlock(out_labels = ["t(s)", "consigne", 
                                                "sortie_charge_transformee"],
                                    freq = 50)
   
   graphe = customblocks.EmbeddedGrapher(("t(s)", "consigne"), 
                                          ("t(s)", "sortie_charge_transformee"),
                                          freq = 3)

   y_charge = customblocks.YBlock(out_labels = ["t(s)", "consigne", "sortie_charge_transformee"], freq = 50)

   y_record = crappy.blocks.Multiplex(freq = 50)

   demonstration_sign = crappy.blocks.Dashboard(labels = [])

   if fichier_d_enregistrement is not None :
      record = customblocks.CustomRecorder(filename = fichier_d_enregistrement,
                              labels = labels_a_enregistrer,
                              parametres_a_inscrire = parametres_du_test)
  
   crappy.link(gen, y_charge)
   crappy.link(gen, y_decharge)
   crappy.link(carte_NI, y_charge, modifier=_card_to_pid)
   crappy.link(carte_NI, y_decharge, modifier=_card_to_pid)
   crappy.link(y_charge, pid_charge)
   crappy.link(y_decharge, pid_decharge)
   crappy.link(pid_charge, carte_NI, modifier=_pid_to_card_charge)
   crappy.link(pid_decharge, carte_NI, modifier=_pid_to_card_decharge)
   crappy.link(carte_NI, gen, modifier=_card_to_pid)

   crappy.link(pid_charge, y_record, modifier=_pid_to_card_charge)
   crappy.link(pid_decharge, y_record, modifier=_pid_to_card_decharge)
   crappy.link(carte_NI, y_record, modifier=
                           [_card_to_recorder_and_graph, 
                           crappy.modifier.Diff(label="sortie_charge_transformee", 
                                                out_label="derivee_voltage")])
   if fichier_d_enregistrement is not None :
      crappy.link(y_record, record)
   crappy.link(carte_NI, graphe, modifier=_card_to_recorder_and_graph)
   crappy.link(gen, graphe, modifier=_gen_to_graph_in_tons)

   crappy.start()
   crappy.reset()


def main():

   def crappy_launcher():
      parametres = []
      parametres.append("Parameters, you want to")
      parametres.append("print in the, csv file")
      labels_voulus = ["t(s)", "consigne", "sortie_charge", "sortie_charge_transformée", "sortie_deplacement"]

      launch_crappy_event.wait()
      launch_crappy_event.clear()
      if type_d_asservissement == ASSERVISSEMENT_EN_CHARGE :
         demarrage_de_crappy_charge(consignes_generateur = consignes_du_generateur, 
                        fichier_d_enregistrement = str(datetime.datetime.now())[:11] + test_title + ".csv",
                        parametres_du_test = parametres, 
                        labels_a_enregistrer = labels_voulus)
      else :
       #TODO : change to demarrage_de_crappy_deplacement
         demarrage_de_crappy_fake_machine(consignes_generateur = consignes_du_generateur, 
                        fichier_d_enregistrement = str(datetime.datetime.now())[:11] + test_title + ".csv",
                        parametres_du_test = parametres)


   Thread(target = crappy_launcher, daemon = True).start()

Edited : On a side note, since we're back on windows, crappy.stop() doesn't work properly once again, but I've managed to circumvent it.

[WeisLeDocto]

So, definitively back to windows and its bugs

Well, wouldn't be fun if it wasn't challenging !

if Crappy's already running, wouldn't crappy.reset() mess with the rest

We normally use it before calling crappy.start(), so we never face this problem. Now since crappy.start() is blocking, you won't anyway be able to use the trick while Crappy is running. I'm afraid if you want to record the data during the pre-stretch phase you will indeed need to come up with a complex solution using custom blocks and / or modifiers.

I've finally put it on Github

Nice, it will make things way easier for both of us ! Now I've taken a quick look at your code and it's extremely hard to read and understand. I strongly advise you to subclass the Tkinter objects rather than calling them in functions, as explained in this tutorial. It would for example allow you to call fonction_principale() in a non-recursive way. Also, you should as much as possible store the different classes/functions in separate files.

crappy.stop() doesn't work properly once again

I'm not going to be working through all the termination issues anytime soon, especially on Windows. You'll need to wait a few months for that.

apparently, it only does that with the fake machine

I had a very hard time spotting the exact cause of the problem, but I managed. First, when I'm talking about a minimal code that reproduces the bug, I was expecting something like that. The bug is specific to Windows, or more precisely to the spawn start method of Multiprocessing. It doesn't appear when using the fork start method, the default one on Linux.

It actually comes from the Dashboard block, which is the reason why you only had it with your demarrage_de_crappy_fake_machine() function since the other functions do not feature a Dashboard. The problematic line is this one, because it gives the Dashboard a non-pickable attribute before its process starts. Moving this line to the beginning of the prepare() method, that is only called after the process starts, solves the problem for me.

You could also use the newest version of the Dashboard block that hasn't been released on PyPI yet. You could integrate it in your project as shown here. The same solution works if you choose to keep your current implementation of the Dashboard and to just move the problematic line. You would just have to change the dashboard.py file accordingly.
We could have spotted the problem easily if we had tested more on Windows... I suspect a similar problem would occur when using the Client_server block, but except for this one and the Dashboard all the other blocks should be safe.

Weis

[Elbub]

I'm afraid if you want to record the data during the pre-stretch phase you will indeed need to come up with a complex solution using custom blocks and / or modifiers.

Challenging, indeed. That's not my main priority, but I'll try.

To be honest, it's my first project in Python, and I had to take over a 4000 lines long code with cryptic function and variable names. I've learned OOP on structured languages (mainly JAVA) but had to learn Python's ways to handle objects pretty much from scratch. This is why my code and its organization are messy, and I've still got a lot to learn on those point.
Unfortunately, I don't have much time left for my internship (a little more than a week), so organizing isn't on top of my priority list, but I'll try to work on that afterwards anyway.

Sorry about the big chunk of code, but a colleague was waiting on me to show him how the new program works, so I didn't really clean before sending it. I figured out it was the Dashboard this morning, but you got that by yourself. I'm gonna try switching that line for now, and I'll check the new Dashboard later. When I'll have time, I might check on that other block too.

[WeisLeDocto]

I have to say we're positively surprised that a company is using our module !
Please tell the people working there that if they're having trouble with Crappy they can always add an issue on GitHub and we'll try to help.
Specifically, switching to Crappy v2.0.0 when we release it in a few months should improve the overall behavior but might require the code to be slightly adapted.

Weis

[Elbub]

Oh, it's not a company that big, but still, you can take pride in your researchs being used in industry.
They already know. My supervisor followed our discussion, even if some points may be a little cryptic to him. He'll stay tuned for updates, and I'll check too from time to time.
About a user-friendly way to select paths, I'll try to get you the parts used for that in my code, and even translate it by the end of the week if I got time. You've been so helpful all along, so it's the least I can do.

[Elbub]

Hello again.

I've found ways to do (or circumvent) most of what I wanted to, but I'm still stuck with a few problems :

• To record both manual- and auto-driven parts, I thought about using trig_link (old version) or safe_start (new version) and putting a global boolean that's checked in the modifier that feeds the generator to give it all that's received or an empty dict. The problem is that, even when setting that boolean to True, the modifier doesn't seem to detect any change. I guess it's probably because it's called from a different thread. Do you think there is a way to get it to work as we want to (starting recording, then starting the generator later) ? The only solution I see would be to add a physical button/actuator, but I'm not sure we'll be able to do that.

• Crappy's block definitions and start are in a function called from a thread at the start of the main function. Once Crappy stops (a button from the main thread can stop all blocks and Crappy), it should return from that function and terminate. When starting another test (calling the main function with the same values), the previous Crappy-launching thread should already have terminated and the new one should create blocks and start normally. However, the first one never terminates (join() freezes the program, and the debugger shows that, without join(), that thread keeps existing) and pressing the start button again just terminates the newly created thread without starting crappy (well, I suppose that it tries to start Crappy again and crashes because it can't be started twice). What's even weirder is that the minimalist version that follows works (well, threads don't terminate, but Crappy still gets started). As far as I can tell, there's no difference that should matter between that and my program.
  Is there a way to force the first thread to terminate and would it properly stop Crappy ?

from tkinter import *
from threading import Event, Thread
import crappy


list_of_used_blocks = []
launch_crappy_event = Event()

def start_crappy(path_wanted = None):
  global list_of_used_blocks

  launch_crappy_event.wait()
  launch_crappy_event.clear()

  gen = crappy.blocks.Generator(path = path_wanted ,
                                spam = True)
  list_of_used_blocks.append(gen)

  graph = crappy.blocks.Grapher(("t(s)", "cmd"))
  list_of_used_blocks.append(graph)

  crappy.link(gen, graph)
  crappy.start()

def crappy_launcher():
  start_crappy(path_wanted = [{"type" : "constant", "value" : 3.0, "condition" : None}])

def stop_crappy():
  while len(list_of_used_blocks ) > 0 :
    block = list_of_used_blocks .pop()
    block.stop()
    try :
      crappy.blocks.Block.instances.remove(block)   # Pretty sure it does nothing because it tries to remove 
                                                    # from another thread, thus considering Block as a new class.
    except KeyError:
       pass
  crappy.stop()   # Same thing here, I guess, hence why it doesn't really stop Crappy.
  crappy.reset()

def main(init_parameter = None):

  def start_crappy():
    launch_crappy_event.set()
    start_button["state"] = "disabled"

  def restart_test():
    nonlocal restart_requested
    restart_requested =True
    root.destroy()
  
  parameter = init_parameter    # Other stuff not shown here can change it.
  Thread(target = crappy_launcher, daemon = True).start()
  restart_requested = False

  root = Tk()
  start_button = Button(root, text = "Start", command = start_crappy)
  start_button.grid(row = 0, column = 0)
  Button(root, text = "Stop", command = stop_crappy).grid(row = 0, column = 1)
  Button(root, text = "Restart", command = restart_test).grid(row = 0, column = 2)

  root.mainloop()

  if restart_requested :
    main(init_parameter  = parameter)

if __name__ == "__main__" :
  main()

[WeisLeDocto]

Hi Elbub,

• First, the modifiers and links live in the main process, so at least multiprocessing is not a concern here. If you want to share a True/False information between threads, the recommended way to do it is by using threading.Event. Using the global keyword is most of the time a bad practice. Also, the trig_link or safe_start arguments will wait for any data to be received on the incoming links, including an empty dict. If you want to use them to prevent the Generator from starting, then don't send anything at all on the upstream links. Except for that, I think raising a flag in a Modifier could work in your case.
• What I find a bit weird about your code is that you start multiple threads managing Crappy, meaning that you could potentially have several threads running Crappy simultaneously. Instead, what was proposed in another issue was this code, where a single thread manages Crappy. It ensures that the crappy.start() method returns and that crappy.reset() is always called. It is still possible that some Blocks would still be running in the background (bug), but they cannot interfere with the main execution thread. By the way, crappy.stop() and crappy.reset() should be enough to stop and clean up the Blocks, stopping the Blocks one by one should do more harm than good. Is there a specific reason why you chose to start multiple threads ?
  I cannot help further as the minimal example indeed works OK. To answer your question, there's no clean way to stop threads in Python, and even if you use a workaround it wouldn't stop the Processes of Crappy. It would unblock the code though, and the zombie Processes would just stay alive somewhere in the background.

Weis

[Elbub]

• Since the function creating the blocks and links is itself started from a thread, I'm still unsure about the links and modifiers living in the main thread. I'll see to modify it, but if I recall correctly, I did that to avoid being unable to do anything with the main window while Crappy runs. I'll try to refactor it anyway, probably tomorrow.
  I tried using threading.Event in differents ways, but it seems to be unpicklable (using Windows is fun, yay...) when it's not just doing nothing.
• That's the bit of code that gave me the impression that blocks' definition ant crappy.start() should be in the same function. So, should I just call crappy_launcher() from main() and have it use a thread to call start_crappy() or is there a simpler way ?

Oh, and, side question : can a dictionnary used as path contain other pairs too ? I'd like to merge the position and load driving into a single driving that manages both with mixed conditons. To be able to do so, I'll need to tell if a path's condition is a position or a load, and the easiest way would be to add something like "condition_type" : "load".

[WeisLeDocto]

I'm still unsure about the links and modifiers living in the main thread

I said they live in the main process, that's completely different ! But it sure is confusing.

• Normally as long as the Event was created in the parent thread before starting the child threads, it should be able to be set and waited for in any child thread.
• The layout that makes the more sense IMO is to have 1 thread running, with a while loop or equivalent if you want to be able to restart the test, and at each loop the thread waits for an Event to be set before it can start Crappy. The start button just sets the Event, just like in the example code. For crappy.stop(), I would also tend to place it in a Thread so that it is separate from the GUI, but that might not be needed as it doesn't mess with Processes as much as crappy.start().

It's currently not possible to have extra keys in a Generator path, because they're used as kwargs by the Paths objects and an exact number of them is expected. Wouldn't be hard to implement though, but not available right now.

Weis