back to article The underbelly of simulation science: replicating the results

Replicating computer-simulated science is harder than you think, as a group of aerospace researchers from George Washington University have found. In fact, without decent versioning, documentation, publication of data and rigorous evidence standards, simulations that attract headlines both in academic and general media should …

  1. SeanC4S

    https://youtu.be/jN9L7TpMxeA

  2. Anonymous Coward
    Anonymous Coward

    Computer modelling is more of an art than a science

    You basically tweak your models until they give you the answer you were expecting in the first place.

    1. Deltics

      Re: Computer modelling is more of an art than a science

      A fact which seems to be overlooked in a cavalier, one might say downright criminal, fashion when governments choose to hit the average man in his pocket in order to line the pockets of their climate change buddies when presented with the latest "model data" to "prove" the "need" for governments to spend "their" money "fixing" that "problem".

      1. itzman
        Facepalm

        Re: Computer modelling is more of an art than a science

        well yes, except climate models ignore fluid dynamics turbulence and convection and most non linear effects anyway cos they are too hard to model

        1. Lars Silver badge
          Happy

          Re: Computer modelling is more of an art than a science

          "except climate models ignore", and global wobbling.

          https://www.youtube.com/watch?v=lPgZfhnCAdI

      2. Paul Crawford Silver badge

        Re: "hit the average man in his pocket "

        Would these be the same men who want a future for their kids with both energy & food, but have an irrational fear of anything with "nuclear" in the title?

        1. Deltics

          Re: "hit the average man in his pocket "

          Some of them.

          But the most vocal faction against nuclear anything are those who are also the loudest protestors decrying man made climate change (and yet curiously silent on the largest contributing factors to that change).

          It's almost as if they are more interested in maintaining a popular position than in actual scientific arguments. They wouldn't want to risk upsetting their donors by having to do a volte face and point out the actual, uncomfortable truths. i.e. that people will have to stop drinking milk and eating burgers and accept that their smartphones will need to be recharged using power from fission.

    2. Anonymous Coward
      Anonymous Coward

      Re: Computer modelling is more of an art than a science

      You basically tweak your models until they give you the answer you were expecting in the first place.

      As someone who spends a good fraction of my time on performing computer modelling and interpreting the results (and the rest of the time designing the software to do the same) , I can asure you that you don't. In fact, we have a simple rule: whenever practical, the person doing simulation and the analysis does not get to see the experimental result until the simulation and the analysis is complete. Furthermore, 90% of the time we spend in computer modelling is devoted to establishing sensitivity of the results to the parameters of the simulation (both physical and technical). As often as not, we end up abandoning a simulation after concluding that we can't draw reliable conclusions from it. At least in the field of science I work in, this is all standard practice, and has been since the time immemorial.

    3. jzl

      Re: Computer modelling is more of an art than a science

      The key finding from climate studies - that the Earth is warming rapidly - is reproducible. Highly so.

      All of these diverse models reproduce it, even though they often differ in the details. At this stage in the game, there's no doubt in the scientific community that climate change is happening. It is treated, rightly, as a fact. A data point.

      People who think otherwise spend far too much time listening to scientifically illiterate hacks with a political axe to grind, and far too little time reading New Scientist or Scientific American.

      1. Anonymous Coward
        Anonymous Coward

        Re: Computer modelling is more of an art than a science

        The key finding from climate studies - that the Earth is warming rapidly - is reproducible. Highly so.

        Yes, same garbage in to same inadequate models produces the same results completely at odds with reality, time and again.

        What else would you expect? They don't model real climate, with convection and turbulence and micro and macros scale feedback effects. They model an idealised and cartoon scale climate where all the stuff that is 'too difficult to model ' is 'parametrized' and given arbitrary values that 'get te result we expect' or if you are cynical 'get the results the bandwagoneers want us to get'.

        Superficially climate modelling looks like good science. Until you start asking awkward questions. Then they give you a deniers cap, shove you in a corner and commit grantdectomy on you.

        It is of course no longer science, but naked self interest, political expediency and corruption. Fraud carried out using computer models instead of smoke and mirrors.

        1. Anonymous Coward
          Anonymous Coward

          Re: Computer modelling is more of an art than a science

          As was shown after "Climategate" released the UEA's climate software and data to the rest of the world. The analysis of this "software" was very revealing about how useless the models were, how poor the data was, how bad the coding standards and overall code controls were. It was not even version controlled ffs!

          Reproducablity is hard, but REAL SCIENCE demands your results can be reproduced, otherwise they are nothing but propaganda, marketing or PR.

        2. Tom 7

          Re: Computer modelling is more of an art than a science

          Tosh!

          I spent many years knocking up microchips which were simulated using spice and digital simulators. We NEVER simulated at the quantum level which is how the devices really worked but the simulations using simplified models were accurate enough to get working devices with tens of thousands of components. I'm guessing they do similar with devices of billions of devices today.

          You dont need to model convection and turbulence on micro of macro scales to model climate. Stupidly accurate weather perhaps but if you have a model that can give you reasonably accurate correlation with past events over the time data is available and it predicts temperature rises in the future only a bloody idiot would ignore it. It might be wrong but its got a far higher probability of being right than casual disinterest.

          1. BebopWeBop

            Re: Computer modelling is more of an art than a science

            Bravo! If we are not careful some idiot will be decrying all models that don't begin with QED in order to predict the effects of two cars crashing into one another.

        3. Naughtyhorse

          Re: Computer modelling is more of an art than a science

          Yet curiously the predictions of the warmists come to pass time and time again.

          It must be so frustrating knowing you are right and having the data dispute your.... deeper understanding/prejudice (delete inaplicable) time after time

        4. jzl

          Re: Computer modelling is more of an art than a science

          "Yes, same garbage in to same inadequate models produces the same results completely at odds with reality, time and again."

          The point is the models are often wildly different, in their methodologies, in their area of focus, in the datasets they use. No model can model everything. That's why it's called a model. But they're not guesses. They're based on years of work in physics, hydrodyamics, oceanography.

          And yet they all agree. The Earth is warming. Time and again, no matter how we apprach the problem, no matter how we slice and dice the data, we get the same answer. It's overwhelming. It really is a mountain of evidence.

          1. Deltics

            Re: Computer modelling is more of an art than a science

            All these wildly different approaches, methodologies, focus and datasets and yet... and yet... they all reach the same conclusion. Any right thinking person would expect that at least ONE of those vastly different models would come up with a different answer.

            Unless of course, all of these different models, methodologies, focus' and datasets start from these different points but all with the same goal - i.e. to prove the same thing. In which case of course it makes perfect sense that they all reach the same conclusion that they all set out with the express intent of proving in the first place.

            It's a mountain of prejudice. Not evidence.

      2. Naughtyhorse

        Re: Computer modelling is more of an art than a science

        You must be new here!

        Lewis this is jzl, jzl this is Lewis (whatever you do dont mention the tinfoil hat)

      3. Deltics

        Re: Computer modelling is more of an art than a science

        The fact that climate change is real does not require complex modelling to "prove". Nobody with even half a brain cell would try to claim that climate change is not happening. Climate Change happens all the time. It happened long before man arrived and will continue to happen long after we have gone.

        Nobody with half a brain cell would realistically claim that anyone else would sensibly claim it is not happening.

        So when someone says "Climate Change is real" and then conflates that with "and is man made" and then further conflates the first FACT with the second SPECULATION, they can only have an agenda, because rational thinking is clearly not in play.

        The FACT of Climate Change is not the same thing as establishing WITHOUT DOUBT the CAUSE for any particular change in the climate, which is what these models all set out to "prove". And they do so with intention of proving that the cause of the change and the nature of that change is man-made and we just need to prove it, ergo we just need a model which demonstrates it. So guess what the models prove ? Astonishing coincidence eh ?

        Of course, all these models that "prove" the link ultimately prove to be flawed as our understanding of underlying influences on climate change improves so the models get thrown out and new models created which - again astonishingly - also prove what the model builders WANT to prove.

        As for grinding political axes, I'm not sure that there is a "No Climate Change" lobby movement. All the political axe grinding comes from the Climate Change Faith Movement (it's where they get their funding after all, so who can blame them really ?).

  3. glen waverley

    and this is called chaos theory...

    From article "And in unsteady fluid dynamics, small floating-point differences can add up over thousands of time steps to eventually trigger a flow instability"

    I think this was discovered by Edward Lorenz in 1963 with his butterfly effect.

    But good to see this work showing sensitive dependence on initial choice of software and hardware packages.

    1. JeffyPoooh
      Pint

      Re: and this is called chaos theory...

      It's amazing how many scientists haven't got the first clue about Chaos Theory.

      I base this conclusion on the things that I read them saying. It's sometimes obvious that they'd benefit from somebody throwing a copy of James Gleick's book 'Chaos...' at their head as hard as possible.

      1. itzman

        Re: and this is called chaos theory...

        What is more concerning, is that no one at all has a clue about anything outside their narrow speciality.

        Almost no scientists of the common or garden cooking variety have a clue about philosophy of science, which might tell them when their speculations are exceeding limits of applicability, and those schooled in manipulating linear equations haven't a clue about the relevance of non linear equations to almost every problem that remains unsolved, and as for the people who run politics and the media, and probably the banks too, they dont even understand what science is!

        If I've had one online conversation, I've had a thousand all of which can be boiled down to a simple class

        "You can't build a perpetual motion machine"

        "How do you know until you have (spent a billion or ten public money) trying?"

        "Science says you can't"

        "But science could be wrong. It's always changing"

        "Odd. I thought you lot were saying that the science was settled, and it was right"...

        If only we could send all these people to a country where they could run it according to their naive idealistic principles, and simply remove themselves from the gene pool thereby..

        I re-read a Jane Austen Novel last week. Jane Austen is heralded as a prototypical modern woman and feminist. However much as I enjoy her work, it has a hole in it big enough to drive ten nuclear reactors through. In Jane's society (dominated by the sensibilities of those of the female persuasion) the function of the cardboard cutout men is to provide income support and amusement for the main purpose of life, which is social climbing and a little bit of baby rearing. Only on three occasions is Work mentioned., Mrs Smith (persuasion) has to do needlework to stay alive Ugh!. And in Mansfield park the Master of the house is absent for a protracted period doing something industrious to do with slaves and sugar in the west indies or something, but this is mere plot device...and of course there is the Navy, which is one supposes more about robbing the enemy for prize money than actual work, and indeed, is eventually dismissed as utterly dull in the context of Social Life: 'Of Vices and Rears I have had my fill!' daringly exclaims one risqué protagonist.

        The point? Oh the point!. Austen's Women have become the norm in the middle classes of today, endlessly discussing this and that, and being widely read and in possession of fine opinions, but all part of some game that is completely and utterly divorced from the actual realities of ensuring there is bread on the table water in the tap, the lights still work and sewage isn't piling up in the street.

        It's even reached science. So called. In the pinnacle of insanity, science has ceased to become the development of models that inform us about behaviour of the reality we have already observed and is now the development of metaphysical models that seek to determine what the reality we observe shall be.

        If we dont actually observe any global warming, that doesn’t disprove AGW, oh no., In fact it just shows that we need more funding to determine why stuff is stopping global warming appear to be happening, because we know it must be still happening, because the models say so.

        The models no longer model pre existent reality, as a derivative product of it. Oh no, models now determine reality.

        Well in a sense they always have, and that is what metaphysics is all about, but we didn't use to call it 'science' . Mostly we called it 'religion'. One might argue endlessly about why God's Will expressed itself in the way that it dim, but we never questioned that there was a God, with a Will, and that's what was really happening.

        in 1600 97% of early scientists agreed that there was a God, and that was that.

        1. Naughtyhorse

          Re: If we dont actually observe any global warming,

          Oh but we do...

          The problem is fucktards upon being presented with a living breathing elephant insisting there aint no such animal.

          pic an unusually high energy weather event, any unusually high energy weather event......

        2. Thomas Steven 1

          Re: and this is called chaos theory...

          I believe the country you mentioned is called something like the Caliphate and is currently being set up in the odds and ends of other countries such as Syria. If you want a place run on naive idealistic principles, it looks pretty good.

        3. TDog

          Re: and this is called chaos theory...

          I used to work for the NHS. And as a clinician one of my biggest hatreds was the Health Education Council. There was the poster of the Polar Bear with the caption "And try telling him vegetables are for wimps"; when challenged as to how to find vegetables growing from seals the response was - "It's alright; no one knows that!".

          And then the £1M that was spent in Wales (in the 80's) on a look after your heart campaign. Net result - not a jot, tittle, or iota of change. Response - "We need more money".

          This was prior to Chaos Theory but was most certainly a local bias in arbitrary random behaviour.

    2. Paul Crawford Silver badge

      Re: and this is called chaos theory...

      Thing is, there are lots of science/engineering systems that show such sensitivity to initial state so I am not in the least surprised that they found it really hard to reproduce things. That simple fact alone ought to be part of any publication!

      Sadly not many researchers take the time & effort to determine how deterministic the process is or how much differing tools can vary. For an insight in to just one of MANY possible issues you might care to read this blog and some of the commentards that follow:

      https://randomascii.wordpress.com/2014/10/09/intel-underestimates-error-bounds-by-1-3-quintillion/

      1. Paul Crawford Silver badge

        Re: and this is called chaos theory...

        Actually this is more appropriate for this thread:

        https://randomascii.wordpress.com/2013/07/16/floating-point-determinism/

    3. Mage Silver badge

      Re: and this is called chaos theory...

      The problem isn't just Chaos theory.

      You have to ask how reliable some simulations are at all. Reproducible results does not equal correct results. If an equally good library or different hardware gives a different answer, which one is right? Maybe neither.

      1. Bronek Kozicki

        Re: and this is called chaos theory...

        It's amazing how many scientists haven't got the first clue about Floating-Point Arithmetic. FTFY.

        1. Roo
          Windows

          Re: and this is called chaos theory...

          "It's amazing how many scientists haven't got the first clue about Floating-Point Arithmetic. FTFY."

          Amen to that.

          I suspect that covers 99.99% of folks writing code (not excluding myself). :)

        2. JeffyPoooh
          Pint

          Re: and this is called chaos theory...

          "...first clue about Floating-Point Arithmetic. FTFY."

          Chaos Theory got its very start with similar rounding errors, as discovered and investigated by Edward Lorenz. Your 'Fixed That For You' (FTFY) seems to betray that perhaps you're unaware of this history.

          To be clear, the field has branched out from there.

        3. swm

          Re: and this is called chaos theory...

          I recall a scientist calculating a matrix inverse on an IBM 704 and getting a set of answers (27 bits). Being careful he repeated the calculations on a Burroughs 5000 (about 40 bits). The answers were totally different. So he redid the calculation on the IBM 704 with double precision (54 bits) and got about the same answers as he did previously on the IBM 704. Further investigation revealed that the problem was not round off error but exponent overflow (the Burroughs machine had a greater exponent range) and that the Burroughs machine had gotten the correct answer.

          But shouldn't the occurrence of an exponent overflow generate a trap and a message of some kind? Further investigation found out that the manager of the IBM data center was tired of hearing complaints of exponent overflows that he had the overflow traps disabled.

          It is very hard to do a long complicated floating-point calculation accurately without controlling every step of the process.

  4. Anonymous Coward
    Anonymous Coward

    Science is in need of corrective action

    There are so many things broken in science.

    Raw data and negative results must be published (just dump it on the 'net; indexed somewhere). The obviously-evil Paid For journals with high paywalls need to be killed off. Peer review needs a major tweak, improved ethics and more speed. Hype must be eliminated. Reproduction Studies must be funded and rewarded. Dietary-Health papers, the problem child of science, later proven false (endless examples) should result in severe consequences. Journalists misreporting results should be fined. Less quasi-religious 'faith' in Science As A Religion, skepticism always. P-hacking, searching for correlations. Lack of rigour in stats. On and on...

    Step 1 is acknowledging that there are endless major problems.

    None of them are too difficult to fix. The biggest problem seems to be with Step 1.

    1. Anonymous Coward
      Anonymous Coward

      Re: Science is in need of corrective action

      I'd up vote you, but that would be recognizing the problem ;-)

    2. Anonymous Coward
      Anonymous Coward

      Re: Science is in need of corrective action

      Raw data and negative results must be published (just dump it on the 'net; indexed somewhere).

      While I am generally in favour of publishing raw data, there may be reasons making what you propose impractical (e. g. legal protections for handling of personally identifiable information; copyright and contractual restrictions; the sheer size of data sets; and so on). Furthermore, many (most?) funding agencies these days require that the raw data is safely archived and available for inspection (as an example, see the code of conduct imposed by DFG - one of the main funding agencies in Germany: "good scientific practice"). In practice, this typically means that the raw data gets deposited with the institution's library.

      1. theModge

        Re: Science is in need of corrective action

        On a website which must be maintained for at least 5 years after the project completes is a funding condition for a lot of work we do. It tends, realistically to stay up for longer than that because a) people get attached to there projects or the inverse happens and b) people move to entirely different fields and forget there projects exist. This means that you get to learn exactly how long a website can go without maintenance (years. many many years. IT will notice when someone finds a hole in and starts exploiting it, then the search the for guilty will commence, only to find they left years ago...)

  5. Neil Barnes Silver badge
    Boffin

    Why do you need reproducability?

    Surely "Scientists say X" is all that's necessary? Are these scientists not the outstandingly trustworthy and intelligent people on the planet, with nothing more than the good of humanity in mind?

    Or have I been reading the wrong sort of Golden Age Scientification?

    1. Anonymous Coward
      Anonymous Coward

      Re: Why do you need reproducability?

      Prezactly. In fact you dont even need to DO the science. All you have to do is miscount occurrences of words in scientific papers and if you can then show that '97% of scientists say that red is blue' (omitting the magic conditional, 'when redshifted by an expanding universe'), then you can guarantee that the New Scientist and the Guardian will both headline it, (followed by the BBC) to prove that after all, up is down and socialism and science working together will make a Utopia by next Wednesday, except they need a couple of billion more in public money, each, to do it.

  6. Doctor Syntax Silver badge

    A very long time ago I was at a lecture in a British Ecological Society meeting (that's ecology as a branch of biological science, not a political movement). Someone had spent a lot of time measuring the growth of heather in the lab in response to light, temperature, water, etc. Based on that they then used a model to predict growth of heather in field conditions based on monitoring all these factors. They reported that the model hadn't performed well, being out by a factor of 3. A comment from the floor was reassuring: in modelling terms a factor of 3 was a good result. I've approached modelling with scepticism ever since.

  7. Anonymous Coward
    Anonymous Coward

    Systematic

    In the UK (cannot comment on situation elsewhere) a major issue is "funding by results" which way too often means a race to achieve lots of publication credits & citations (Note that lots of publications / citations does not always mean good "paper" - citation counts a joke as essentially compulsory to cite anything even tenuously related to area of your publication, so if you work in a popular field, citations mount up rapidly irrespective of the papers quality)

    Too much focus on doing stuff at breakneck speed in "hot" fields where lots of teams active in same area and being first with a new finding is "important" (be it kudos, financially) e.g. medical areas with potential profit if a better drug / potential cure for something found.

    Lots of scientists would like the time & opportunity to try & replicate work published elsewhere, but do not have chance.

    These issues apply across the board, whether it be in traditional academia (e.g. university) or in overtly commercial research (e.g. pharma company)

    Just like in many jobs, those slogging at the research coalface are often not happy with the constraints they have to work under but get little choice in the matter.

    AC as although I no longer work in science, lots of my friends still do

  8. Graham Marsden
    Alert

    Can I just say...

    ... Flying snakes?

    Fucking FLYING SNAKES...?!

    1. Richard 12 Silver badge

      Re: Can I just say...

      They're just falling with style.

    2. Ugotta B. Kiddingme
      Joke

      Re: Flying snakes?

      Just another of those countless "Australian Things That Will Kill You" ?

  9. Alan Brown Silver badge

    Mystifying?

    "Even more mystifying, the boffins write, their own software delivered different results, depending on whether it was running on GPUs (using the Cusp library) or parallel CPUs (using the PETSc library)."

    We saw this a few years back when a researcher turned up substantial differences between 32bit systems, 64bit systems and gpus on 64-bit. When running the _same_ software.

    The answer was "cumulative rounding errors" and researchers who need a good hard kick in the pants because they make assumptions which aren't valid.

    Floating point units all give some degree of difference in the final few decimal places and no matter how accurate they are, the final answer is rounded. You cannot simply take the printed result from one set of equations and feed it as input to the next. That results in things like solar system simulations which fly apart after a few tens of millions of years.

    The answers for the 32bit systems had been assumed accurate and the others wrong. In fact all systems had given wrong answers. The only difference was the degree of wrongness. Once that was realised (and why) , going back to first principles resulted in more accurate answers that tracked closely over the different architectures.

    Floating point math is _hard_ and validating the results is essential. Most importantly of all, cascading calculations are a fundamental error which needs to be eliminated where ever encountered.

    1. Anonymous Coward
      Anonymous Coward

      Re: Mystifying?

      Think this is a gentle reminder that computers are bad at floating point math.

      You have got to love rounding errors.

  10. computinghomer

    Wouldn't it be of scientific interest to develop specialized CPU's with much larger floating point units for simulation.

    1. Roo
      Windows

      "Wouldn't it be of scientific interest to develop specialized CPU's with much larger floating point units for simulation."

      That's been done repeatedly, the hardware often lies idle. There are various libraries that offer arbitrary precision, interval arithmetic etc, etc etc, but these tools mostly lie idle mostly through ignorance of their existence and applicability.

      If folks are made aware of these tools there are additional barriers:

      1) Performance will take a hit. Folks tend to want a credible but wrong answer faster than the correct answer slower (see the background to Seymour Cray's "Parity is for farmers" quote).

      2) It will be "too hard" to validate the results (which they know will be different so they can't simply run diff).

      3) It will be "too hard" to get downstream systems/regulators/phbs to accept the new results because they are different.

  11. Michael H.F. Wilkinson Silver badge

    Often modelling simply adds rigour to thought experiments

    Results from computer models should always be scrutinized carefully (just like any other result in science, really). It is essential that all source code (including version information), parameters, and hardware information should be made public.

    In many cases, I find that failure of simulations is often the most interesting part. People postulate a mechanism to explain some phenomenon, and might leave it at that: a just-so story. By drawing up a computer simulation you can test ideas more rigorously, and you sometimes find that the proposed mechanism cannot produce the observed effect. Once you have ruled out numerical instability or just programming errors, you can then safely state that the proposed mechanism does not explain observed behaviour, and that therefore the theory requires some adaptation.

    Likewise, leaving out parts of a system because they are too hard to model, or you first want to start with a simple model, and add complexity later (always a good approach) can also give important information. I have done some modelling of bacterial communities in the intestine (insert "your model is shit" joke here), and found that quite a few properties can be modelled well enough fairly simply. For example, even without modelling an immune system, ratios of aerobic vs anaerobic bacteria could be reproduced very well simply from decent estimates of the amounts of nutrients and oxygen entering the system. Practically no other parameter had ANY impact on that ratio (ODE/PDE solvers used, hardware used, time steps used etc.). Not that surprising to biologists perhaps, but medical researchers had assumed the immune system controlled the intestinal microflora. It might do so, but apparently it is not necessary to control the aerobe/anaerobe ratio.

    By contrast, if your simulation explains things nicely, you have not learnt that much, only that your mechanism is plausible. Proof requires a lot more than mere simulation

    1. Naughtyhorse

      Re: Often modelling simply adds rigour to thought experiments

      Stay there while I go find my copy of 'chaos' and load it into the pocket trebuchet i just aimed at you :-)

      the reason 'cfd' is hard is that it is fundamentally chaotic, and the combination of floating point and positive feedback is what caused all the problems. high sensitivity to small variations in initial conditions, coupled with noisy data. you can't really model something like that incrementally - you can theorise, and apply limited tests under highly controlled conditions, but as soon as you try to do anything real world all bets are off.

      There is still much work to be done in understanding nonlinear systems. and the current political shenanigans between the scientific and oil/coal producing industrial complex is not helping.

      1. Michael H.F. Wilkinson Silver badge
        Boffin

        Re: Often modelling simply adds rigour to thought experiments

        Trust me, I have read more on chaos theory than most (though there are certainly deeper thinkers on the topic), I even gave a lecture or two, and have written a paper or two about the subject. Gleick's book is nicely accessible, but there are more rigorous works (Ott, Sauer, and Yorke, Coping with Chaos (1994) contains a good collection of papers, I found, but that is a bit outdated now, perhaps).

        One of the first things to do in any system of non-linear equations is to see if positive Lyapunov exponents are expected (sometimes in a simplified version of the full set), and the system might therefore be expected to be chaotic. In the case of my intestinal micro-flora simulations the (simplified) system showed no chaotic behaviour whatsoever. That makes life a lot easier.

        The problems with floating point inaccuracies coupled to chaotic behaviour is well-known to any scientist worthy of using HPC systems for numerical simulation (and, yes, there are many that aren't worthy). The Shadowing Lemma has a thing or two to say about this, in particular, that you may not be tracking the path form the initial state, but there exists a path close enough to the simulation run for an arbitrary length of time, starting from some (unknown) perturbed version of the initial state. The simulation trajectory "shadows" a trajectory of the true system. Having said that, a true chaotic system with one or more positive Lyapunov exponents cannot safely be simulated with just one run. Typically you have to run a huge ensemble of runs with different starting points for each set of parameter settings before you can say anything about the model system's behaviour. Only then can you start making sensible comparisons with experimental results, or real life. Should real life observations differ from the model significantly, your understanding of the problem as captured in the model is inadequate, or the simulation method is flawed.

        1. Hargrove

          Re: Often modelling simply adds rigour to thought experiments

          @ MHF WIlkenson:

          Re: The problems with floating point inaccuracies coupled to chaotic behaviour is well-known to any scientist worthy of using HPC systems for numerical simulation (and, yes, there are many that aren't worthy).

          Amen, amen, and again, I say amen. Nearly 20 years ago now I read an article in which the Dassault folks, who knew a thing or two about advanced combat aircraft design, had some cogent comments of the amount of testing needed to verify that the granularity of discrete simulations was adequate. This last paragraph is one of the clearest and most concise summaries I've seen on the subject.

  12. Justicesays

    Here's a simple experiment...

    Compile the same (somewhat complex) program from source code, twice.

    Almost certainly the two binaries produced will be different, and not just in the time stamps.

    Due to parallelism, optimization code paths etc. can be affected by all kinds of things , like the memory layout affecting the response time for certain calculations differently, what background tasks are otherwise running, left over state from previous runs affecting disk/memory layout. Even if you ran the same massively parallel simulation twice in a row with all the same parameters, you almost certainly would not get exactly the same results , never mind changing the hardware or libraries!

    1. GrumpenKraut

      Re: Here's a simple experiment...

      > Almost certainly the two binaries produced will be different,...

      If you really see this happening, file a bug report.

      A compiler should produce exactly the same machine code every time it is run with identical parameters (and code).

      Your arguments like

      > ...background tasks are otherwise running, ...

      frankly sounds unsound to me.

      1. Richard 12 Silver badge

        Re: Here's a simple experiment...

        Indeed, compilers must be 100% deterministic, because otherwise they fundamentally could never comply with the standard for their language.

        However, two different compilers - inc. versions of compiler or compiler settings - probably won't give the same result as each other as they will favour different optimisations when multiple are permitted, or use a different implementation of the appropriate standard libraries.

        So if your code relies on undefined behaviour, it can and will blow up in your face the moment you update or change compiler.

        For a great example of undefined behaviour: in C++, "char" is not 8 bits.

        For others - illegal operations are not allowed, and thus the code path that results in an illegal operation cannot happen, thus can be optimised out of the binary. It can't happen, so why waste space?

        1. Anonymous Coward
          Anonymous Coward

          Re: Here's a simple experiment...

          Indeed, compilers must be 100% deterministic, because otherwise they fundamentally could never comply with the standard for their language.

          That's a nice example of non sequitur.

          With a few exceptions, language standards only specify what the code does, not how it is done, or which specific optimizations are applied by the compiler, or how the compiler reaches the decisions it does. For example, a compiler may wish to bound the amount of time it spends on looking at code alternatives to make user experience less frustrating (and most good optimizing compilers do this, at least as an option). Depending on random events, the best code it finds may well be different for each compilation...

      2. Justicesays

        Re: Here's a simple experiment...

        Takes a lot of work to get parallel stuff to be deterministic:

        https://groups.csail.mit.edu/commit/papers/09/asplos073-olszewski.pdf

        This paper discusses a system that only purports to reproduce or avoid race conditions by enforcing lock acquisition and release in a repeatable fashion.

        It does not guarantee results are the same for sequential runs.

        If you compile with optimization and in parallel, your output code will (likely) be different each time.

        If you compile with very specific options, single threaded and minimal optimization, you can get to a state with byte-by-byte identical output (with some compilers).

        There is a Debian project to make all their packages build byte-for-byte the same each time.

        It's not complete and has been running for 2 years now. At the start they were mostly not.

        https://wiki.debian.org/ReproducibleBuilds

      3. Roo
        Windows

        Re: Here's a simple experiment...

        "> Almost certainly the two binaries produced will be different,...

        If you really see this happening, file a bug report."

        Before filing that bug report I would do a quick double check to see if it was using some form of profile driven optimisation. The EPIC (aka Itanic) folks used that approach to help them achieve parity and eventually beat much smaller, faster, simpler and easier to code super-scalar RISC cores (which were fabricated on fraction of the die area using a process that was at least 4 years out of date).

        Computer architecture folks had already worked out that you needed dynamic instruction scheduling to get the most out of the hardware decades earlier, Intel *and* HP had already learnt that lesson so you have to wonder how they convinced themselves that an entire ISA explicitly designed for statically scheduled instructions would help.

    2. swm

      Re: Here's a simple experiment...

      The GNU compiler is compiled three times, first with the native compiler, second by the just compiled compiler, and thirdly by the second compiled compiler. A diff is then run between the last two compiler object files to detect any differences. If there are differences then there is an error somewhere.

      Undefined behavior is tricky.

      Test: will the following loop ever terminate?

      for(int i = 1; i != 0; i++) {

      // do something

      }

      Answer: this is undefined behavior so it might and might not. Adding one should surely cause the value of i to wrap and eventually reach zero. But signed integer overflow is undefined behavior and the compiler can assume it doesn't happen and delete the "i != 0" check.

      for(unsigned int i = 1; i != 0; i++) {

      // do something

      }

      This loop is guaranteed to terminate because unsigned overflow is guaranteed to wrap.

      Writing good, solid code is hard.

  13. Bucky 2

    Small floating point differences

    In college physics, we were required to calculate not simply the results of an experiment, but also the margin of error. This had to be included in each step of the primary calculations. This was referred to as "showing your work."

    Surely small floating point differences are part of the computation of the margin of error? Don't they calculate that in real life, or do they just do that in school?

    1. Anonymous Coward
      Anonymous Coward

      Re: Small floating point differences

      In numerical analysis of non-linear dynamical systems, those "small float point differences" can diverge exponentially after relatively few time steps. The problem is that biological systems, whether they be flying snakes or bumblebee wings or the beating of the heart, often rely on walking the knife's edge between stability and chaos, with subtle, minute feedback mechanisms to keep things working, and it is often those mechanisms we are trying to study via modeling.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like