Topics

Starting point: The software crisis

Approach: Software engineering

Problem facing academia: Writing code is only a peripheral issue

Key to success: Concepts and personal discipline

Chapter at Springer (German)

https://link.springer.com/chapter/10.1007/978-3-662-71607-6_2

• Abstraction is the key to
  mastering complex problems.
• Intellectual manageability is an essential criterion.
  It requires discipline when programming.
• Reusability, reliability and verifiability
  require modular, tested and readable source code.
• Code quality (particularly readability, with all that this entails)
  is not optional for scientific analysis software.
• Despite its importance, programming is given too little priority
  in the natural sciences.

In addition to the requirements of science itself (see previous chapter), there is an intrinsic motivation arising from software development itself to engage with this subject: all relevant software is of such complexity that it simply exceeds the human capacity to comprehend and control it. Scientific software is no exception here, but rather a confirmation of this: the intrinsic complexity of scientific problems compels us to develop software of corresponding complexity. Without (technical) tools and knowledge of strategies developed over decades, however, this is doomed to failure.

From a software development perspective, the book is not about writing code that works (in the sense of delivering a correct result), but about code that works in a broader sense – that is, code that is, in particular, reusable, reliable and verifiable.

The motivation behind the book can be summarised in a single sentence as a provocative thesis: Code written by scientists is generally of far too poor quality to be meaningfully reused. However, this touches directly on the very foundations of science, for without traceability there can be no science and no gain in knowledge.

The good news is that it is possible to develop software that meets the requirements of science. The less good news is that this requires discipline, a structured approach, the ability to think abstractly, knowledge and understanding of the relevant tools and techniques, and the professionalism to apply and implement this knowledge effectively.

These questions are intended to encourage personal reflection on the topic, but no answers are provided.

• Which key concept in software development enables us to tackle complex problems successfully?
• Which fundamental aspect of programming is perhaps most crucial to the quality of the resulting code and, consequently, its reusability, reliability and verifiability?
• What is perhaps the biggest problem with programming in a scientific context?
• Which two aspects hold the key to success for high-quality software and the corresponding productivity (and enjoyment of the work)?

An annotated and carefully selected list of further reading on the subject. The selection is, by its very nature, subjective.

A good book for those who wish to explore programming in greater depth and, above all, in a practical way within an academic context and in the more physics-oriented disciplines (including physical chemistry) — and who find the book lacks practical relevance:

• Scopatz, Anthony; Huff, Kathryn D. (2015): Effective Computation in Physics, O'Reilly, Sebastopol

Other books (also) tailored to Python and focusing on its use in a scientific context include (this list is by no means exhaustive):

• Hill, Christian (2015): Learning Scientific Programming with Python, Cambridge University Press, Cambridge, UK
• Downey, Allen B. (2015): Think Python, O'Reilly, Sebastopol
• VanderPlas, Jake (2016): Python Data Science Handbook, O'Reilly, Sebastopol
• McKinney, Wes (2017): Python for Data Analysis, O'Reilly, Sebastopol

For anyone wishing to explore Python in greater depth, “The Hitchhiker’s Guide to Python” by Reitz and Schlusser is also highly recommended:

• Reitz, Kenneth; Schlusser, Tanya (2016): The Hitchhiker's Guide to Python, O'Reilly, Sebastopol

A highly readable, personal account from the early days of computer science and programming, written by one of its key figures, is Edsger Dijkstra’s speech delivered in 1972 on the occasion of his receiving the Turing Award. In it, he uses the term ‘intellectual manageability’ and emphasises why, together with abstraction, it is central to programming.

The term ‘software engineering’ was first brought to wider attention at the NATO conference on software development. Naur and Randell authored the proceedings of this conference.

Another great classic, focusing on project management and, moreover, very well written, is Frederick Brooks’ *The Mythical Man-Month*. In it, the author describes his experiences developing the operating system for the first IBM mainframes in the 1960s and reflects candidly on what went well and what went wrong. The 1995 anniversary edition also includes, amongst other things, his equally famous essay “No Silver Bullet—Essence and Accident in Software Engineering”. The central thesis of this essay is:

There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity. Frederick P. Brooks: No Silver Bullet

• Naur, P.; Randell, B. (1969): Software Engineering: Report of a conference sponsored by the NATO Science Committee, Garmisch, Germany, 7-11 Oct. 1968
• Brooks, Frederick P. (1995): The Mythical Man Month, Addison Wesley Longman, Boston
• Dijkstra, Edsger W. (1972): The humble programmer, Commun. ACM 15:859-865

Ian Sommerville provides a fairly comprehensive and well-structured introduction to software engineering in his book “Software Engineering” [Sommerville, 2018Sommerville, Ian (2018): Software Engineering, Pearson, Hallbergmoos].

• Sommerville, Ian (2018): Software Engineering, Pearson, Hallbergmoos

The problem of programming’s lack of prominence in the sciences is now well known. Perhaps one of the first people to have engaged intensively with this problem and possible solutions is Greg Wilson. He founded “Software Carpentry”, a non-profit organisation dedicated to teaching programming concepts in an academic setting. One of the reasons articles are published in biology-focused journals is that bioinformatics would be inconceivable without computers. However, this does not mean that the average biologist is a better programmer than a chemist or physicist… (quite the opposite, in fact).

• Baxter, Susan M.; Day, Steven W.; Fetrow, Jacquelyn S.; Reisinger, Stephanie J. (2006): Scientific software development is not an oxymoron, PLoS Comput. Biol. 2:e87
• Hastings, Janna; Haug, Kenneth; Steinbeck, Christoph (2014): Ten recommendations for software engineering in research, GigaScience 3:31
• Merali, Zeeya (2010): ...why scientific programming does not compute, Nature 467:775-777
• Wilson, Greg (2006): Software carpentry. Getting scientists to write better code by making them more productive, Comput. Sci. Eng. 8:66-69
• Wilson, Greg; Aruliah, D. A.; Brown, C. Titus; Hong, Neil P. Chue; Davis, Matt; Guy, Richard T.; Haddock, Steven H. D.; Huff, Kathryn D.; Mitchell, Ian M.; Plumbley, Mark D.; Waugh, Ben; White, Ethan P.; Wilson, Paul (2014): Best practices for scientific computing, PLoS Biol. 12:e1001745

Ultimately, the quality of programmes has a great deal to do with the attitude of the programmers involved. See the discussion in Chapter 33 of [McConnell, 2004McConnell, Steve (2004): Code Complete. A practical handbook of software construction, Microsoft Press, Redmond, Washington] and the book *The Pragmatic Programmer* by Thomas and Hunt [Thomas, 2020Thomas, David; Hunt, Andrew (2020): The Pragmatic Programmer, Addison-Wesley, Boston].

• McConnell, Steve (2004): Code Complete. A practical handbook of software construction, Microsoft Press, Redmond, Washington
• Thomas, David; Hunt, Andrew (2020): The Pragmatic Programmer, Addison-Wesley, Boston