New component : database storage/database input

[Elbub]

Hi there !
It's been a while since I checked on this project, and I see that it has grown well.
We currently have a big annual project for school (~100h/person, 3 persons) for which we are free to find a subject relative to the theme "effort". It can be a stand-alone app or an API or module for an existing tech, we're quite free on that. Thus, I thought we could contribute to Crappy by creating a database block that could serve as an output (store your test's data) or an input (generate graphs or dashboard from previously stored tests), and maybe add (to the side) a user-friendly simplified interface to create such a database (local postgreSQL since it's free, or cloud ones with free trial such as BigQuery).
Do you think it could prove useful to some Crappy users ? And if we do implement these, would you rather recommend that we fork the project, create a branch or even something else ?

[WeisLeDocto]

Hi !

I'm glad to see you consider contributing to Crappy for your assignment ! However, it might not be the best fit if you're looking into building experience with data bases.

creating a database block that could serve as an output (store your test's data)

There are already a Recorder and HDFRecorder Blocks for saving data. A new one would have to bring extra functionality compared to the existing ones.

Also, in most use cases, data is acquired from sensors and recorded as a time series. Traditional databases are, AFAIU, optimized for highly structured and transactional data, which time series are not. I don't think they would be of any use to the general audience of Crappy.

Nevertheless, there are a few time-series-specific database programs out there (like TinyFluxDB), these would probably integrate more easily with Crappy. Might be an interesting approach in Crappy as part of a more modern data visualization solution for example.

or an input (generate graphs or dashboard from previously stored tests)

This would likely fall out of the scope of Crappy, which is focused on data acquisition and test management, not data post-processing. This would of course still be of interest to Crappy's users, but would certainly not be part of the Crappy project.

would you rather recommend that we fork the project, create a branch or even something else ?

Currently, you don't have any write access to Crappy, so creating a branch on the base repository is not an option. But anyone is of course free to fork the project on their own account and contribute to the base repository with pull-requests ! If anyone would become an active contributor, I'll grant them the write access after a few merged pull requests.

Do you think it could prove useful to some Crappy users ?

My overall feeling is that time-series databases in Crappy could be an interesting way of sharing data between the Blocks or with external processes or devices (third-party visualization tool, remote microcontroller, etc.). But I don't have enough experience on this topic to evaluate what kind of new functionality such Blocks would bring compared to the existing ones. And I really cannot think of a use case for classic data bases in Crappy.

[Elbub]

Hi !

It's been a while and we've discussed a lot of things about a lot of different aspects of our project. As you stated, what we first presented may not be really useful to Crappy users, so we tried to challenge the points you raised and finally decided to let go of it and switch to another subject. We're not really comfortable with TinyFluxDB and it seems Crappy doesn't need much of a DB.

The initial idea might have been useful for lengthy experiments with lots of sensors (and so, huge data volumes), where local recorders wouldn't suffice, but it's an industry problem rather than a research one, and seems to fall out of the scope of Crappy. And about using such a block as an input, we thought replayability could provide some use, but saving the data directly in another format (tables, charts) from already existing Crappy blocks is certainly more effective.

Still, don't hesitate to ask if you ever feel like such a block could be added in the future, as we may at least have notes that could fasten the conceptualization.