Does Your Ten-Year-Old Code Still run?

How many ten-year-old codes are still running today? Young people around the world nowadays does not know what a 3.5-inch floppy disk is? Just like handphone batteries are used to be taken out and changed.

In the 1983 cold-war film War Games, high-school hacker David Lightman uses one to break into the school’s computer and give his girlfriend top marks; he later hacks into a military network, narrowly averting a global thermonuclear war.

Okay, how many of those ancient programs are still running? Or once-important programs have lost their supporting software/hardware and gone in the wind.

Nicolas Rougier wants to find out the answer to that question.

Rougier is a computational neuroscientist and programmer at INRIA, the French National Institute for Research in Digital Science and Technology in Bordeaux. The Ten Years Reproducibility Challenge is conceived in 2019 together with Konrad Hinsen, a theoretical biophysicist at the French National Centre for Scientific Research (CNRS) in Orléans, the challenge dares scientists to find and re-execute old code, to reproduce computationally driven papers they had published ten or more years earlier. Participants were supposed to discuss what they learnt at a workshop in Bordeaux in June, but COVID-19 scuppered those plans. (The event has been tentatively rescheduled for June 2021.)

Although computation plays a key and ever-larger part in science, scientific articles rarely include their underlying code, Rougier says. Even when they do, it can be difficult for others to execute it, and even the original authors might encounter problems some time later.

The Ten Years Reproducibility Challenge aims “to find out which of the ten-year-old techniques for writing and publishing code are good enough to make it work a decade later”, Hinsen says.

“Ten years is a very, very, very, very long time in the software world,” says Victoria Stodden, who studies computational reproducibility at the University of Illinois. In establishing that benchmark, she says, the challenge effectively encourages researchers to probe the limitations of code reproducibility over a period that “is roughly equivalent in the software world to infinity”.

The challenge had 35 entrants. Of the 43 articles they proposed reproducing, 28 resulted in reproducibility reports. ReScience C began publishing their work earlier this year. The programming languages used ranged from C and R to Mathematica and Pascal.

“Visual Basic had its glamorous days and was replaced completely by new languages. It is rather difficult to re-create an operating environment to just run it. There have been generations of changes in function names and parameter types. Therefore, it is quite surprising that an ancient language, if there is an active community now, is more likely to run successfully today. Because their functions and interfaces have already stopped updating, the form is the same as 10 years ago. On the contrary, if the language is widely used and frequently updated, compared with 10 years ago, the various APIs have long been completely different.