Continued maintenance and development

[roaldarbol]

Hi @mooch443.
I am just wondering what the future looks like for TRex? I know you wrote it during your PhD, and it's a massive, and super useful piece of software - which is also why I'd hate for it to dwindle away. Is there a plan going forwards, either with you in the lead or is Ian planning on getting someone to take over? It's unfortunately such a common thing to see amazing new developments of open source scientific software be "a project", and then never to be worked on again, and of course understandably so.
I'm also asking as I am considering which basket to put my eggs in going forward, and over the years maintenance and continued development has become the most important thing to me, ranking higher than performance most of the time.
So yeah, out of both curiosity and affection for this awesome tool you've made!
Cheers,
Mikkel

[mooch443]

Hey @roaldarbol ,
I know I haven't released a full new version yet -- but I have many exciting things that I am adding to the software at the moment, and that I will release this year. I am currently still employed in the same place and doing the same thing basically, continuing development. It's just that I am involved in many other projects at the moment, the technical parts, so it's overall a bit slower.
However, if you have a look at the interface branch here in this repository, you'll see that I am working on things :-) There is also a trex-beta channel, but it will be fairly broken oftentimes because that is what I use for trying out stuff.

I can expand more on what I am doing if you like,
Thanks for your interest!
-Tristan

I am also creating a GUI installer (or I am basically done with that), btw. and since you are seemingly interested, here is a first sneak peek:

[roaldarbol]

Oh wow, that's amazing news!!! Yes please, would love to hear about what you are working on! And great news about the installer (and the super cool logo!) 😃

[roaldarbol]

Also, if/when you need anyone to test things out in the wild, please do give me a shout, I'd be happy to help. 😊

[mooch443]

Thanks for offering!

Currently the following things are planned:

1. a more unified interface: this means that converting and tracking is done from within the same program in the future. currently this is still pretty much an open issue, although i have begun to put all kinds of stuff into a single one and laid the ground-work to make this possible. so i would say this is about 50%.
2. using various machine learning approaches to segment videos, instead of the more traditional background subtraction. this is pretty much done, but lacking some user-experience issues. i would say this is about 90% in terms of time.
3. support for videos in 8k or slightly more. this was just a change in terms of IDs for blobs and i think its done (limited testing so far).
4. support for color (limited). previously there was only grayscale, which is fine in most cases. however, imagine you have individuals that are different essentially only in color. then even a limited color palette (256 colors) is potentially helpful. this can be done without any grand changes in the architecture, so... done.
5. specialized support for color for visual identification. a separate point because i am still not sure this actually helps - whether the color is encoded as a single grayscale value vs. splitting that same number into three numbers might not make a huge difference to the network at all. maybe it does. so I'll have to test that (about 10%).
6. some GUI issues that were proposed, of which i have implemented a couple so far.
7. other bug fixes and speed improvements, which i have already tackled or am working on right now. difficult to say what the progress is there, since that the list of things always changes.
8. i have created an installer that is itself in beta, but i would reckon about 80% there.

Yeah, think that's about it. You can check some of the things out already, but since its not done there might be changes to ID formats and so one, making the results potentially less future-proof (like, if I change the video format, then your videos generated with the beta version would not be compatible with newer versions anymore). You can try the trex-beta channel. So for example:

conda create -n beta -c trex-beta -c pytorch -c nvidia trex

If you're on a linux/windows machine. On mac you may have to remove nvidia, and/or exchange pytorch for pytorch-nightly.

[roaldarbol]

Thanks for the update - they sound like some very useful features! I'm sure the unified interface will be helpful for getting people started! And yes, colour tracking could prove very useful indeed. Often in working with ants, we paint them with dots of colours to be able to recognise the individuals within the colony - taking advantage of that sounds useful. The segmentation improvements will also be super welcome! It's an issue across all the tracking software I've tested, especially when recording super long videos that then have to be partitioned (I chop them into 6 hour chunks - and sometimes individuals stay more or less still for the entire duration).

On a tangent: Could it be possible to recognise individuals across trials? So if a network has been trained to recognise a particular number of individuals, and can distinguish them very well, then that network could identify the same individuals in future videos?

I'll try and download the beta sometime in the coming weeks for sure!

[mooch443]

Regarding recognizing the same individuals in different videos. Sure, this is possible to a certain extent - if you have the same group, or a subgroup of that group again. Identification always runs against the entire group, so it learns to maximize differences between individuals within the group. Can't recognise them outside of that context, only in comparison with others basically. Not sure there's an easy way around that, though. Could train a specialist network for that individual, but i think the easiest would be to train a network, let it save the training data - train a different network on the other group and then try to make both networks agree on identities. Other ideas are welcome. In general, if group composition and other stuff is similar, there should be no problem though.

Quick question regarding the chunks: so you had long videos and the software you tried wasn't able to analyse the full video in one piece? Too much RAM?

Good application for color support right there, thanks for the example! I will probably keep updating the beta channel frequently, so don't despair if it doesn't work properly one time ;)

[roaldarbol]

Keeping IDs within a group between videos is already great, so if that could be exposed that would be useful I think. I wouldn't necessarily prioritise making changes to the way the networks work, but maybe something to keep in mind should you end up making changes in the future.

Actually that was with TRex about 1.5 year ago, and yes, it was too much RAM. It was on my Intel Mac, so couldn't make use of GPUs.

No worries! I'd imagine it would be helpful for any animals with sexual dimorphisms too (though I don't know how widely they are used in these kinds of experiments).

[roaldarbol]

Hi @mooch443, hope you're well! Hope development is going well! Do you have a rough ETA for 2.0 (no pressure!)? I'm approaching a batch of tracking where the machine learning segmentation could likely prove quite really useful. Do you know how it performs on drastic changes in lighting?

[mooch443]

This year, is my ETA for finishing a usable version. So then you'll also have access. Currently the main part of the app (for you guys) is kind of broken, so it does not make a huge amount of sense to publish it - although you can try it out in the beta channel.

Regarding performance: machine learning is pretty good at dealing with noise. It all depends on your training data. But we can already achieve some pretty cool tracking with those kind of models - like tracking sharks in the wild from above. What you can already do is have a training dataset for your kind of application (unless such a thing already exists). By dataset I mean stuff like this: https://universe.roboflow.com/ for training a YOLO8 network on pose/detect/outline.

Making your own datasets is reasonably easy (e.g. using Roboflow, no affiliation). If you're not familiar with that, you could have a look to familiarise yourself, since working with ML models will be the future anyway :-) Although stuff like Meta's SegmentAnything may help with that.