An example of open software and innovative academic publication in econometrics
OSCR Member Andreas Alfons is Associate Professor of Econometrics at the Erasmus School of Economics. His work focuses on the development of robust statistical methods, machine learning techniques, and their applications in the social and behavioral sciences.
He recently published on a journal that values open code as primary output and uses innovative peer-review workflows. I asked him to share his experience with the community:
My research centers around developing methodology for statistics and machine learning. I find writing code to be the most fun part of my work, and it helps me keep my sanity during a challenging research project. My background is a mix of math and computer science, and thanks to the latter I have always put a lot of emphasis on producing well-documented, well-structured, and easy-to-use code for the methods that I develop. It’s in the nature of the peer-review process that, as researchers, we have to pick up projects again after certain periods of inactivity, which I find much easier if my code is well-structured and well-documented. Moreover, I believe that providing an easy-to use implementation is of equal importance to the methodological work. Methodology is getting more and more complex, and if no code is available, the broader research community simply cannot use a method, no matter how promising or powerful it might be. The easier it is to use your code, the greater the potential for impact.
As a result of this way of working, I have authored various R packages, and I recently published one of them in the Journal of Open Source Software (JOSS). Note that I write that I published the software in the journal, and not that I published an article about the software, but more on that later. Publishing in JOSS is quite different from publishing in other journals, which is why I want to share my experience with the community. I will focus on a comparison with two other open-access software journals: the Journal of Statistical Software (JSS) and the R Journal (RJ). To elaborate on my experience with those journals, I have published in JSS and JOSS but not RJ, and I have peer-reviewed submissions for all three.
All three journals review the manuscript and the software package together. But my impression is that in JSS and RJ, the articles are viewed as a substantial additional resource for the software that somewhat stands alone from other documentation. For instance, JSS requires a usage example to be “an enlighting non-trivial case study” and RJ requires articles on add-on packages to go “beyond package vignettes in aiming to provide broader context”. The scope of JOSS is different in that the software package seems to be the main contribution, and that the paper is a short note that discusses the academic need for the software and highlights how it differs from related software. In fact, JOSS papers can be as short as one page! For example, there is no need to include usage examples in the submitted manuscript. Authors are instead required to provide enough examples in the regular documentation of the software, so that they are easily findable to all users of the software rather than only to readers of the paper. For my paper, I still found it useful to include some of the examples from the documentation also in the manuscript. Even though my initial manuscript was only 6 pages long (including references), I was still asked to reduce the length.
Another main difference between the journals is the review process. Whereas JSS and RJ still use traditional, blinded peer-reviews, I find the review process in JOSS very refreshing:
- The review is extremely fast. For my article, it took just over a month from submission to publication.
- Both the pre-review suitability checks and the review are carried out as issues on GitHub and are public. The reviewers are listed on the published paper, and the website of the published article contains a prominently-placed link to the review. From the author’s perspective, reviewers cannot hide behind anonymity and have to agree to a code of conduct, which creates a very constructive discussion. From the reviewer’s perspective, there is a clear record of reviewer contributions.
- Reviewers get a checklist of criteria that the submission should satisfy (but of course they can raise any point they deem relevant in the discussion). Those criteria are also clearly communicated to authors before submission. In other words, the criteria for publication are open and transparent.
On the other hand, one cannot ignore that fact that JOSS is not indexed in the Journal Citation Reports, while RJ and in particular JSS have high article influence scores. This brings us to the question: “To which journal should I submit my software package?” As with any other research article, I’d say it’s horses for courses. For some of my current and future software packages, I’ll aim for a high-impact journal such as JSS due to its standing in the research community, and I’ll happily go through the extra effort to prepare an extensive manuscript (and accept the much longer review times). For other packages, JOSS is a very welcome outlet due to the fast reviews and the small amount of additional work once the software is already well-structured and well-documented.
No matter the journal, I can only recommend to make the code for your developed methodology available, and to aim for a publication of your code in a software journal. Publication in a peer-reviewed journal puts a stamp of quality control on your software, which should encourage more users to try it. Furthermore, a published article may alert some researchers to the existence of your software in the first place.
In short, software publications have great potential to increase the impact of your methodological work, and as a bonus you can get an additional publication out of a research project. In my book, that’s a win-win!
We hope that Andreas’ experience will inspire more researchers to openly share their code. If you are affiliated with any school at Erasmus University, contact Antonio for personalized assistance. Also, consider joining OSCR to crowdsource knowledge from our members!
Andreas Alfons and Antonio Schettino