If the question arises as to why it is necessary to engage with programming and software development concepts, why not simply start programming straight away, rather than having to set up, learn and use additional infrastructure as well? The short answer is that this is the only way we can meet the requirements for software used in scientific data analysis, as derived from the criteria for scientific rigour: reusability, reliability and verifiability.
Every routine used to analyse real data should therefore, at the very least, be under version control (VCS) and bear a unique version number. Documentation and a licence are also helpful and strongly recommended for users. Only in this way can the minimum criteria of reusability, reliability and verifiability – which should reasonably be applied to software for scientific data analysis – be met to some extent. However, text editors and a bug tracking system are also essential aspects of a minimal infrastructure for scientific software development.
This chapter provides an initial overview of the aspects of a minimal infrastructure discussed in more detail below, presents a cost-benefit analysis, and discusses further tools and aspects of “soft infrastructure”.
These questions are intended to encourage personal reflection on the topic, but no answers are provided.
In the book, six aspects of infrastructure are discussed: version control, version numbers, external documentation, bugtracker, editors/IDEs, and licenses.
If I were to answer for myself which solutions I would use or do use for the various aspects, the following list would probably be the result. As the infrastructure used depends in part on the chosen programming language, this is also specified here.
| Aspect | Solution |
|---|---|
| Programming language | Python 3.x |
| Version control system | git with Gitea as web frontend |
| Version numbers | SemVer |
| External documentation | DokuWiki; alternative: Sphinx |
| Bugtracking | external: Bugzilla; alternative: Gitea |
| Editors/IDEs | PyCharm |
| Licenses | GPL, BSD |
Important: This is sometimes a very personal choice, and I won’t go into the reasons for it here. Everyone should think carefully about which choice they make for themselves. If necessary, you can always change your mind ‘along the way’, even if that can be a bit of a hassle.
Editors and IDEs are not just a matter of personal preference, but also depend on the programming language used. If you use MATLAB, you’ll (almost) inevitably have to use the editor that comes with it. For Python, PyCharm is highly recommended, and the free version is perfectly adequate for scientific programming. Otherwise, Eclipse is always an interesting option.
An annotated and carefully selected list of further reading on the subject. The selection is, by its very nature, subjective.
Although project management extends beyond the infrastructure discussed in the book, many aspects summarised here under the heading of ‘infrastructure’ can be embedded within this context.
A “classic” work on project management is the report by Frederick Brooks from his time at IBM, where he was responsible for developing the operating system for the first IBM mainframes (OS/360). The book is well written and, in its 1995 edition, also includes, amongst other things, his equally famous (and well worth reading) essay “No Silver Bullet—Essence and Accident in Software Engineering”.
The realisation that programming and software development also have a very significant social component, due to the necessary close interaction between developers, contributed—along with dissatisfaction with the rigid rules of formal project management—to the emergence of the “Agile Manifesto” [Martin, 2008Martin, Robert C. (2008): Clean Code. A Handbook of Agile Software Craftmanship, Prentice Hall, Upper Saddle River, New Jersey]. However, these aspects had already been described in detail in the literature prior to this. [Brooks, 1995Brooks, Frederick P. (1995): The Mythical Man Month, Addison Wesley Longman, Boston, DeMarco, 2013DeMarco, Tom; Lister, Timothy (2013): Peopleware. Productive Projects and Teams, Addison-Wesley, Upper Saddle River, NJ, Weinberg, 1998Weinberg, Gerald M. (1998): The Psychology of Computer Programming. Silver Anniversary Edition, Dorset House Publishing, New York, NY, Constantine, 2001Constantine, Larry L. (2001): The Peopleware Papers. Notes on the Human Side of Software, Yourdon Press, Upper Saddle River, NJ].
In their book “Effective Computation in Physics” [Scopatz, 2015Scopatz, Anthony; Huff, Kathryn D. (2015): Effective Computation in Physics, O'Reilly, Sebastopol], in particular in its second half, Scopatz and Huff discuss in greater detail a large part of the infrastructure mentioned in this chapter. In addition, there are a number of articles in specialist journals which, in accessible language, explore the importance of infrastructure for software development in the natural sciences [Osborne, 2014Osborne, James M.; Bernabeu, Miguel O.; Bruna, Maria; Calderhead, Ben; Cooper, Jonathan; Dalchau, Neil; Dunn, Sara-Jane; Fletcher, Alexander G.; Freeman, Robin; Groen, Derek; Knapp, Bernhard; McInerny, Greg J.; Mirams, Gary R.; Pitt-Francis, Joe; Sengupta, Biswa; Wright, David W.; Yates, Christian A.; Gavaghan, David J.; Emmott, Stephen; Deane, Charlotte (2014): Ten simple rules for effective computational research, PLoS Comput. Biol. 10:e1003506, Sandve, 2013Sandve, Geir Kjetil; Nekrutenko, Anton; Taylor, James; Hovig, Eivind (2013): Ten simple rules for reproducible computational research, PLoS Comput. Biol. 9:e1003285, Wilson, 2006Wilson, Greg (2006): Software carpentry. Getting scientists to write better code by making them more productive, Comput. Sci. Eng. 8:66-69].