Reproduction of a comparison between operant and classical conditioning of identical stimuli in tethered Drosophila

[brembs]

Original article:
Brembs B and Heisenberg M (2000): The Operant and the Classical in Conditioned Orientation of Drosophila melanogaster at the Flight Simulator. Learn Mem. 7(2): 104–115. doi: 10.1101/lm.7.2.104
PDF URL original article:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC311324/pdf/x2.pdf
PDF URL submitted article:
https://github.com/brembs/DFS_reproduction/blob/master/article.pdf
Metadata URL:
https://github.com/brembs/DFS_reproduction/blob/master/DFSreproduction.yaml
Code URL:
https://github.com/brembs/DFS_reproduction/tree/master/evaluation_code

Scientific domain:
Neurobiology
Programming language:
Turbo Pascal, R, C++
Suggested editor:

[brembs]

I'm still in the process of getting DOIs for the repository, I hope they can be provided later?

[khinsen]

@brembs For the review, we don't need a code DOI, that can wait for later. And it's actually easier today to get a Software Heritage ID for a GitHub repository. However, the Metadata and Code URLs you cite don't work!

Update: I discovered that my GitHub superpowers allowed me to fix the two links myself. I hope you don't mind!

[brembs]

Ok, not sure where the issue was. I changed one of your links and all links work for me now. Thanks a lot!
Will go and get a Software heritage ID!

[rougier]

@brembs Thanks for the submission.
@gdetor @eroesch @oliviaguest @benoit-girard Can one of you edit this submission in neuroscience for the the Ten Years Reproducibility Challenge (only one reviewer needed)?

[gdetor]

@rougier I can handle this.

[rougier]

@gdetor Great, thank you.

[gdetor]

Hi @eroesch, could you review this work?

[rougier]

Gentle reminder

[gdetor]

@apdavison @benoit-girard Could one of you review this submission?
Thank you.

[benoit-girard]

What would be the delay to do the review?
I am really sorry, with the lockdown & the kids at home, I have very little time to allocate to additional work right now, so I can do it, but if you give me enough time...

[gdetor]

It' works for me. @rougier Is a potential delay in the review process acceptable?

[rougier]

Given the situation, I think we can have a delay yes (like my late answer). Can we target end of June? @brembs Would that be acceptable ?

[brembs]

I'm in no hurry and I have had own kid at home myself and have been late on reviews. So no pressure at all from my end.

[gdetor]

Ok, so @benoit-girard is being assigned as reviewer for this submission.
Thank you.

[benoit-girard]

I'm a bit lost, I'll need some help/clarification: the PDF URL refers to the original work, not to the pdf of the replication paper; this pdf will of course be useful, but where is the pdf of the replication paper, the ReScience one?

[brembs]

Sorry, this must be my misunderstanding of the instructions! The submission is in the repository, named "article.pdf": https://github.com/brembs/DFS_reproduction
That was my understanding of the instructions, sorry if I misunderstood.

[brembs]

I have updated my submission above with the direct link.

[rougier]

You updated header is actually better than the template. Maybe we should adopt it as the default.

[benoit-girard]

The author documents the reproduction of data analyses using C++ code developed in the 90s.

A few point need to be clarified:

• Could the author add a reference section to his manuscript, and provide there the exact reference to the paper that is reproduced. Otherwise, consulting only the pdf of the paper, one cannot identify the original paper with certainty.

• It seems that Fig. 1 is reproduced from the original publication. If so, please state it explicitly in the legend "(reprinted from XXX)", get the authorization from he original publisher to reproduce it here, and add the mention that the publisher will probably require.

• It is mentioned page 2 that Reinhard Wolf is a co-author of the original publication, which does not appear to be the case (the github submission refers to: Brembs B and Heisenberg M (2000)). Can the author clarify that point?

• The original paper contains many results figures (from fig. 2 to fig. 8 of the original paper). The reproduction of fig. 2 only is presented. What prevented the reproduction of the other figures? The conclusion mentions that: "However, for evaluations other than performance indices (and some other evaluations provided by the C++ application), there is currently no solution." Could the author be more precise: which evaluations are completely lost, because they were not computed by the C++ application? What prevents the author to present those that are provided by the C++ application, as done in the Fig. 2 of the present paper?

• From a formal point of view, the paper is not following the ReScience template. @rougier is it mandatory for this special issue? If yes, some work has to be done to convert the document.

[gdetor]

@brembs You could use AbiWord and convert the GDoc to a .tex file. Then you can add the ReScience template packages at the preamble. Once you run a make (following the instructions on how to compile the article) the final pdf should be ready.

[brembs]

I'll try that,when it is time, thanks!

[rougier]

@brembs Any progress on the conversion?

[brembs]

I haven't done anything, yet, as we are still waiting for @benoit-girard to let us know if the manuscript should contain additional data, or if this one figure is sufficient. Once I know that the manuscript is (near) final, I'll try the suggestions here.

[gdetor]

@benoit-girard Gentle reminder

[benoit-girard]

Oh sorry! The notifications of this thread got lost in the middle of all ReScience notifications... I am trying to improve my workflow so that it does not happen anymore.

there are three additional figures (2 panels and one full figure) that I could immediately reproduce with the identical workflow as the one I used for figure 2 (I chose this figure because of the scientific content, which is important for my research and I still use this figure in teaching).

I feel these figures (those that can be reproduced without effort) should then be added to the manuscript.

There is one figure, Fig. 7, which I should be able to reproduce by writing another R-script that collects the relevant PIs from the 12 CSV files and plots them. Would be tedious, but the same principle as the other figures.
For the remaining three figures (two panels and one full figure), one could try to run the old TP code on FreePascal. It may require some tweaking of the code, but my experience with FP (many years ago) was quite decent.

For the remaining figures, I would be nice to try to reproduce them, but I do not consider it as mandatory.

[gdetor]

@benoit-girard Thank you for the comments.
@brembs Could you address reviewer's comments?

[brembs]

It is easy to just run the same procedure on more figures - what worked for one figure will work for the others. I'll generate new figures for them. I'll also have a look and see if I can get one other figure reproduced that isn't part of the same workflow, but I can't promise I will be able to get it done. All of this will take some time, but I don't know how much. I'm pretty much at capacity right now, so it will have to wait until I got some items off of my to-do list.

[gdetor]

Hi @brembs any progress on this matter?

[brembs]

It was looking good for a while late October, then manuscripts and theses needed revising and reviewing. Apparently, everybody also submits their manuscripts before the holidays, so I'm inundated with reviews and such. No chance I can get it done this year, sorry.

[rougier]

@gdetor Any progress?

[gdetor]

Hi @brembs Do you think we could finalize the process?

[brembs]

I've been trying, but still swamped. Hope to get it done before the summer holidays, which start in August here. No more teaching until then, so the chances look good.

[gdetor]

Hi @brembs Thank you for the update.

[gdetor]

@brembs Gentle reminder

[brembs]

I have been thinking that there was something last summer that I forgot to do :-)
Yes, I've put it on my to-do list and will see that I get to it. Thanks for the reminder!

[rougier]

@gdetor @brembs Gentle reminder

[brembs]

Yes, yes :-)

[gdetor]

@brembs Happy new year. Gentle reminder

[brembs]

Thank you and happy new year to you, too! It is on my list and I am frustrated I wasn't able to get it done before the end of the break.

[gdetor]

@brembs Gentle reminder

[brembs]

Please believe me, I'm still trying. Difficulty lies in prioritizing this work, in part since the remaining work would only constitute a repetition of what I already showed for the first dataset.

[gdetor]

Hi @brembs any updates?

[brembs]

It's still on my list, but new things keep coming up. Once there is an opening, I'll finish it. Just don't know when this will be.

[rougier]

@brembs Can we consider closing this issue or do you intend to work on it? We'll soon hit the 1000 days target since submision :)

[brembs]

I've always intended to work on it, but given what other projects I have, it seems like too much work for a very small improvement on the manuscript. Working on the text is not an issue, but if the example I showed is not sufficient to show that I could in principle do the same thing with the rest of the data, then I'm afraid I must pass, sorry.

[rougier]

What is blocking actually ? If this is he conversion to ReScience template, I can take care of that if you want.

[brembs]

No, IIRC, the reviewers felt that taking one dataset as an example was not sufficient, I needed to take the other, analogous datasets that made up the other figures in the original publication and generate the other figures as well. It's essentially doing the exact same pipeline I did for the first version and repeat that pipeline for the other figures. It's not completely unreasonable, but it adds nothing other than volume to the paper, IMHO. It makes the paper more 'complete' in that I haven't reproduced just a single figure of the original article,but the majority of figures.

[rougier]

Ok. We can leave the submission open just in case you find some spare time in the coming weeks.