Sign in / up

back to article Software glitch led to London Ambulance Service outage – report

A software glitch in the London Ambulance Service’s 999 call handling system, which prevented the system’s recycle bin from being emptied, led to an outage at the New Year – according to an official report. Control room staff were forced to log all emergency calls by pen and paper and pass information to ambulance crews over …

COMMENTS

Post your comment
House rules Send corrections
Add to 'My topics' unstar *
  1. Alan J. Wylie

    Links to the report

    http://www.londonambulance.nhs.uk/news/news_releases_and_statements/investigation_into_999_compute.aspx

    http://www.londonambulance.nhs.uk/pdf/New%20Year's%20Day%20CAD%20outage%20report%20-%2027%20June%202017.pdf

    1 0 Reply
    1. Alan J. Wylie
      Reply Icon

      Oracle "Recycle bin"

      https://docs.oracle.com/cd/B28359_01/server.111/b28310/tables011.htm

      The recycle bin is actually a data dictionary table containing information about dropped objects. Dropped tables and any associated objects such as indexes, constraints, nested tables, and the likes are not removed and still occupy space. They continue to count against user space quotas, until specifically purged from the recycle bin or the unlikely situation where they must be purged by the database because of tablespace space constraints.

      7 0 Reply
      1. TRT Silver badge
        Reply Icon

        Re: Oracle "Recycle bin"

        "When you drop a tablespace including its contents, the objects in the tablespace are not placed in the recycle bin and the database purges any entries in the recycle bin for objects located in the tablespace. The database also purges any recycle bin entries for objects in a tablespace when you drop the tablespace, not including contents, and the tablespace is otherwise empty."

        Not exactly crystal clear.

        2 0 Reply
      2. A Non e-mouse Silver badge
        Reply Icon

        Re: Oracle "Recycle bin"

        So you ask the database to delete something, then you have to ask the database to *really* delete it. And if you forget to ask the database to *really* delete it, the system will eventually crash as it fills up.

        So what's the point of the first delete request?

        7 0 Reply
        1. Korev Silver badge
          Reply Icon
          Joke

          Re: Oracle "Recycle bin"

          So you can decide if you want to commit

          0 0 Reply
        2. Doctor Syntax Silver badge
          Reply Icon

          Re: Oracle "Recycle bin"

          "So you ask the database to delete something, then you have to ask the database to *really* delete it."

          Or maybe really, really delete it. Or even really, really really delete it.

          My brief encounter with Oracle simply left me with the feeling that it was thoroughly obfuscated. I'm glad I was able to make my living with saner alternatives.

          5 0 Reply
          1. TRT Silver badge
            Reply Icon

            Re: Oracle "Recycle bin"

            I got about 16 words into that piece of documentation and the voice in my head turned into that of Sir Humphrey Appleby, GCB, KBE, MVO, MA (Oxon).

            9 0 Reply
        3. Robert Carnegie Silver badge
          Reply Icon

          Re: Oracle "Recycle bin"

          We had that in dBase. I've seen people in the 21st century talking about "soft delete". In dBase, you could delete a row (record) from the data table (file), then change your mind and "undelete" it. You could even browse through the data with all the deleted data visible. I think the command to really delete it is "pack". And then it still probably was liable to be left on the disk, in space uncommitted from file use. But I think we also wrote a sub-routine to replace all the fields with blank space, to erase one record - or maybe it was a built-in command of "REPLACE BLANK". Then you'd delete it, maybe.

          1 0 Reply
  2. Ben1892

    Really, "glitch" was it ? or did they just run out of tablespace ?

    I love the Appendices in the report - I think "Empty the Recycle Bin" is my favorite action closely followed by "Disable the Recycle Bin functionality"

    8 0 Reply
  3. Commswonk
    Coat

    Mmm... Puts a whole new slant on "whose turn is it to empty the bins".

    8 0 Reply
  4. Anonymous Coward
    Anonymous Coward

    Wow

    I've never seen the recycle bin enabled on a production Oracle database before.

    The outage was caused by incompetence.

    13 0 Reply
    1. Julz
      Reply Icon

      Re: Wow

      Exacly...

      5 0 Reply
      1. Anonymous Coward
        Reply Icon
        Anonymous Coward

        Re: Wow

        And surely the disk space monitoring would have generated timely alerts that there was a problem so that the incompetence could be caught before there was a system outage?

        2 0 Reply
  5. iron Silver badge

    So the problem was a simple Oracle configuration issue and their solution is to employ two new top level bosses? Surely employing one Oracle expert techie would be cheaper and far, far more effective.

    17 0 Reply
    1. dinsdale54
      Reply Icon

      If you didn't have any IT expertise at the top, they may well not have realised that you need a decent Oracle DBA to run an Oracle database.

      Good Oracle DBAs aren't cheap as Oracle is pretty complex.

      3 0 Reply
      1. Anonymous Coward
        Reply Icon
        Anonymous Coward

        "Good Oracle DBAs aren't cheap as Oracle is pretty complex"

        So once you hired the new bosses no money are left to hire competent DBAs, and monitor the system proactively to catch issues before they become deadly.

        7 0 Reply
        1. 's water music
          Reply Icon

          Re: "Good Oracle DBAs aren't cheap as Oracle is pretty complex"

          So once you hired the new bosses no money are left to hire competent DBAs, and monitor the system proactively to catch issues before they become deadly.

          But at this stage you have a really good handle on the problem and can produce an accurate risk register and apply for more budget to recruit a cheap dba and another senior manager to maintain the ratios

          3 0 Reply
  6. Doctor Syntax Silver badge

    I particularly liked the way their fourth review was to review what they should have learned from previous balls-ups and didn't.

    4 0 Reply
  7. John Smith 19 Gold badge
    Unhappy

    At first I thought they were talking about the Windows recycle bin, but that made no sense,

    as I've never heard of anyone having any trouble with it (just about the only piece of Windows I haven't heard of some problem about).

    But this Oracle whatsit is a whole different ball game.

    4 1 Reply
  8. rob_leady
    WTF?

    I wonder just how old the servers and software actually are ?

    These two recommendations made me cringe a bit:

    R19 Update the CAD servers to a modern, supported platform that replaces the existing Itanium servers.

    R20 Update the Oracle version to current (or one below as a default).

    So probably still on Oracle 10g (10.x) if not earlier, but HP-UX or OpenVMS ?

    0 0 Reply
  9. Anonymous Coward
    IT Angle

    Software glitch caused London Ambulance Service outage

    Well that explains everything, it's this kind of technical reporting that I come here for :)

    1 0 Reply
    1. cosymart
      Reply Icon
      Pint

      Re: Software glitch caused London Ambulance Service outage

      I'll drink to that, is it Friday yet?

      2 0 Reply
  10. Hans 1
    Thumb Up

    R E S P E C T

    I, Hans 1, also would like to "pay tribute to [the] staff who responded incredibly well, ensuring patients continued to receive care during our busiest time of the year.”

    Above all, I think that these guys/gals deserve the greatest respect of all, even with a failing IT system, I am 100% sure they tried their best to get patients treated!

    Thank you!

    Love from hans1

    Microsoft MHP

    Adobbe MHP

    2 1 Reply

POST COMMENT House rules

Not a member of The Register? Create a new account here.

Forgotten password ?

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like