Feedback on experience using the "annex" package

[asitkm76]

Hi everyone,
Apologies, I am 24 hours late on the feedback timeline.
Thanks to Reto for the very helpful and comprehensive videos. Thanks to Gabriel for organizing and putting things together.

I actually had fun trying the package out. I first used the demo data and then used some T and RH data we have forma previous study. This is a reasonably large dataset, 300,000 rows. I did not face any issues with the code.

I have one suggestion.
If I understand correctly, the idea is to run the data from individual homes through the stats extraction process and then if we have multiple homes from a single study, we append the data to the final write-up Excel file. Do correct me if I am wrong here.
I was wondering, could it be an easy modification for the package to be able to deal with a file which has data from multiple homes together? The scenario would be studies that have already been carried out and for which we have the data of all homes together, as one csv file (or on one sheet of an Excel file). Could the package deal with this situation where each home is denoted using a Column that may be called "HomeID"?

This is not really a necessity. The existing functionalities are great. Just suggesting if this might be worth considering for existing/old data sets.

Thanks,
asit

[retostauffer]

Hy @asitkm76

first of all I do have to apologize for responding late! Thank you very much for your feedback, always welcome and very valuable.

I hope I did understand your question correctly: You do have one fairly large XLSX file with data for multiple homes for a specific study and would like to process all of them in one go.

The package allows to do so. We might update the vignettes/articles and make this feature more obvious.

Hints

If you have a look at the data set used in Prepare data from XLSX (check sheet annex_configuration in demo_UIBK.xlsx) you will see that this configuration contains multiple homes (e.g., W1, W2, ...) for the demo study. This is the same demo dataset used in Write and validate output which creates the file "final_UIBK.xlsx" as output. If you have a look at the sheet STUDY you should see that there are statistics for multiple homes in there. In addition, the sheet META-Home will list all the different homes (the ID is <userID>-<study>-<home>).

How it works

When calling the function annex() you need to provide a formula (first input argument) which in our demo looks as follows:

annex_df    <- annex(RH + T + CO2 ~ datetime | study + home + room,
                     data = prepared_df, tz = "Europe/Berlin")

The first argument (the formula) here is:

RH + T + CO2 ~ datetime | study + home + room

... which read as "calculate the statistics for RH, T, and CO2 given date and time (datetime) grouped by study, home and room.

The last defines how to group the data. Thus, if you have multiple studies and/or homes and/or rooms, the function will group your data accordingly and calculate the statistics based on it.

Short answer: The package can even process multiple studies with multiple homes with, in itself, multiple homes all at once.

Implications - Massive data sets

Yo mentioned 300k rows. At some the limit of what your hardware can handle might be reached. The package (the output format) currently allows you to include multiple studies, homes, and rooms all at the same time, however, in case that sets your computer on fire it is also fine to write several output files (XLSX files) containing e.g, only one study, or even one study and one home.

The output format (should) allows to always assign every measurements statistics to a specific user, study, home, room, and variable (e.g., T, CO2, ...) and should thus be unique.

All best and thanks once more for your response (and sorry for the inconvenience of my late response).

Reto

[asitkm76]

Hi @retostauffer thanks for the comprehensive response and no issues about the delay.
That should work. In my case, what I would need to do is change my dataframe which is in long format to wide format and then import it so that different homes and different rooms come in as different columns. The rest should be fine.
Thanks a lot!

[retostauffer]

Hy @asitkm76

I am (infamously) persistent with my delays these days. If you could provide an example of your data set (first few hundred of lines) I am happy to have a look on how to easily reformat it. My suggestion is that tidyr::pivot_wider() will do the trick.

Feel free to send me a sample via e-Mail to reto.stauffer at uibk.ac.at; the easiest would be as a binary file. Simply:

# Let us assume 'data' is your data.frame
data_sample <- head(data, n = 200) # assuming 200 is enough
saveRDS(data_sample, file = "data_sample.rds")

... and send me the resulting file. I'll not give it to a third party, however, in the best case make sure the data set does not contain sensitive (personal) information.