Review of "Total Recall: flmake and the Quest for Reproducibility"

[mandli]

Review of "Total Recall: flmake and the Quest for Reproducibility"

Reviewer: Kyle Mandli
Department: Institute for Computational Engineering and Science
Institution: University of Texas at Austin
Field: Applied and Computational Mathematics
Country: USA
Article Reviewed: Total Recall: flmake and the Quest for Reproducibility

General Evaluation

below doesn't meet standards for academic publication
meets meets or exceeds the standards for academic publication
n/a not applicable

• Quality of the approach:

  Meets

• Quality of the writing:

  Meets

• Quality of the figures/tables:

  Meets

Specific Evaluation

• Is the code made publicly available and does the article sufficiently describe how to access it?

  Yes

• Does the article present the problem in an appropriate context? Specifically, does it:

  • explain why the problem is important,

    Yes

  • describe in which situations it arises,

    Yes

  • outline relevant previous work,

    Yes

  • provide background information for non-experts

    Some, there's a bit of jargon thrown in sporadically but I do not think it significantly detracts from the topic.

• Is the content of the paper accessible to a computational scientist
  with no specific knowledge in the given field?

  Yes

• Does the paper describe a well-formulated scientific or technical
  achievement?

  Yes

• Are the technical and scientific decisions well-motivated and
  clearly explained?

  Yes

• Are the code examples (if any) sound, clear, and well-written?

  Yes, although I think a slight modification to make the CLI examples a bit more readable would be helpful (just use $> or something).

• Is the paper factual correct?

  To my knowledge yes.

• Is the language and grammar of sufficient quality?

  A few corrections have been suggested.

• Are the conclusions justified?

  Yes

• Is prior work properly and fully cited?

  Yes

• Should any part of the article be shortened or expanded? Please explain.

  Yes - I think my major suggestion is for the article to either concentrate on flmake and mention that one of its features is that it addresses the reproducibility problem and shorten the section that addresses reproducibility or make the article address reproducibility and show how flmake in particular solves this. As it is the article seems to have a bit of a split personality with a long section that vaguely seems related.

• In your view, is the paper fit for publication in the conference proceedings?
  Please suggest specific improvements and indicate whether you think the
  article needs a significant rewrite (rather than a minor revision).

  Yes, I would strongly encourage the author to think about reorganizing the paper along the lines suggested above but as a whole the article is worthy of publication in the SciPy 2012 proceedings.

[scopatz]

Thanks for reviewing @mandli - I'll see if I can't sneak in an oppurtunity to fix these (though the likelihood is low.)

[mandli]

If you have any questions let me know.

[scopatz]

Yes, although I think a slight modification to make the CLI examples a bit more readable would be helpful (just use $> or something).

So the reason that I did not do this was to express clearly that flmake functions outside of the flash code directories explicitly. This is an important point not to be overlooked by users of flmake.