Cello sometimes needs to be restarted

[rkrishnasanka]

The case and the Full Demo examples are not working on the online instance. It seems like its timing out before it completes the entire computation.

[tim-tx]

After a restart it's working, but this shouldn't be necessary.

[rkrishnasanka]

What was the issue ?

[tim-tx]

I have no idea why this happens, I think maybe others (Prashant or Shane) might have a few more details, but the stalling you noticed goes away when you just manually kill and restart the instance of Cello.

[rkrishnasanka]

Hmmm. That is weird, I'm trying to think of a reason why its stalling/timing out. Regardless one of the major problems with Cello right now is its execution engine. One can't submit multiple jobs, see the status of a running job, etc. It makes the API almost useless.

[PrashantVaidyanathan]

This is partly because of JVM issues due to which Cello essentially freezes and doesn't process any jobs. Unfortunately since the site is still up, the automated shell scripts do not pick this up and do not restart cello.
This could also be due to a potential memory leak in Cello and/or other dependencies used by Cello.