-
Lack of Incentives for Producing High-Quality Software. Software's role in enabling and accelerating research cannot be understated. But the incentives for producing software in academia are still very thin. One reason is that people do not cite the software they use; the academic currency is still citations.
-
Lack of Good Ways to Track the Consequences of Software Bugs (Errors). Research outputs are a function of the code researchers write themselves and the third-party developer code they use. Let's assume that the peer review process vets the code written by the researcher. This leaves code written by third-party developers. What precludes errors in third-party code? Not much. The code is generally not peer-reviewed though there are efforts underway. There is also no easy way to track the impact of bugs in third-party code on research outputs.
Unlike ideas, which need to be explicitly cited, software dependencies are naturally explicit in the code. Thus, conventional reference lists (~ a bad database) are a bad way to address the macro concerns. Reference lists are, however, likely a good short-term solution for #1.
If all the research code is committed to a system like Github (Dataverse lacks the tools for #2) with enough meta information about (the precise version of the) third-party software being used, e.g., import statements in R, etc., we can create a system like the Github dependency graph to calculate the number of times software has been used (and these metrics can be shown on Google Scholar, etc.) and also create systems that trigger warnings to authors when consequential updates to underlying software are made.
One reason researchers do not cite software is that they don't see others doing it. One way to cue that software should be cited is to show a message when the software is loaded --- please cite the software. Such a message can also serve as a reminder for people who merely forget to cite the software. For instance, my hunch is that one of the stargazer has been cited more than 1,000 times (June 2023) is because the package produces a message .onAttach to remind the user to cite the package.
- Script to email a random sample of R maintainers here
- Script for producing similar messages in Python here
- Example of the
.onAttachmessage using the citation function here
- Write software to PR a list of R and Python packages on GitHub with a similar message.