back to article Tech-for-insurers biz out of action for 10 days now. Hope they had, er...

Hapless insurance tech biz SSP Worldwide is now on its tenth day of a services outage, which has had a huge impact its broker customers. The company was hit by a power blackout on 26 August that fried its HP Storage Area Network. Halfway through the process of a restoration six days later, the business was struck with further …

  1. Doctor Syntax Silver badge

    Were they insured?

  2. Doctor Syntax Silver badge

    "Apparently SSP has a mirror data centre and it could have switched its operations there, but the decision was taken to try to repair the main one as the mirror was some 15 minutes behind"

    Did they take time to think what the mirror was for?

    It all sounds like a PHB stepping in and deciding what was the best. The fact that the underlying problem recurred suggests they didn't get to grips with what it actually was before going ahead with an attempted restore.

    1. TonyJ

      Suppose the problem here is that if they'd flipped the switch, how many people who may have thought they'd bought insurance, hadn't? Suppose it comes down to whether, during their outage, they were still delivering rates (assume not, given the scale).

      Gotta agree though...surely flip the switch, fail the thing over and fix the original issue at leisure. Then work out a way to communicate to potential customers that they _might_ not be covered.

      1. Doctor Syntax Silver badge

        "Suppose the problem here is that if they'd flipped the switch, how many people who may have thought they'd bought insurance, hadn't?"

        How does that compare with the current situation?

    2. ecofeco Silver badge

      The real question is why was the mirror 15 minutes behind in the first place?

      EPIC FAIL across the board.

      I would pull my business from them immediately.

  3. Anonymous Coward
    Anonymous Coward

    We dumped SSP 3 years ago ...

    I work for a major UK insurer, and we (well I) spotted problems 5 years ago, we began a project to extricate ourselves 3 years ago, which was completed Q1 this year ...

    Not a bad thing to have on my c.v.

    1. Anonymous Coward
      Anonymous Coward

      Re: We dumped SSP 3 years ago ...

      Posting anonymously for once...

      Same here.

      My main client switched to a different provider last year. SSP has been having issues for some time - trying to merge its product range as it bought other companies. My client was originally with MCS then Sector.

      First the software support went down hill, then the infrastructure support followed.

      The move to Solihull was supposedly the holy grail for DR solutions.

      Then they stopped doing the regular DR tests (as far as I was concerned).

      Time to jump ship.

      1. ecofeco Silver badge

        Re: We dumped SSP 3 years ago ...

        It seems in my career that most M&As are to make up for the severe failing of the company doing the buying and instead of adding improvement and using the knowledge of the company being bought, they keep the people who are the problem and created the inept corporate culture in the first place.

        Like so many companies and people I talk too, they think a technology solution will fix their cultural problem.

  4. Stoneshop

    the mirror was some 15 minutes behind

    That's not a mirror, that's a backup of sorts.

    The HP SAN gear I know is easy enough to run as a real remote mirror set. And if you don't want that because you want to be able to cut the link in case some data at the primary site gets corrupted and you want to keep the secondary site from joining in the fun, you have to have some kind of cutout switch you can activate, which obviously requires you (or some monitoring stuff) to detect the problem within the replication delay.

    The primary konking out because of a power failure would have been taken care of with a real mirror.

    1. ecofeco Silver badge

      Re: the mirror was some 15 minutes behind

      Financial companies of all stripes are notoriously skint when it come to operations and especially IT.

  5. Baldy50

    Where are the profits going?

    Well the profits are there for some new or better kit and experienced IT personnel to run it!

    Time to invest on what puts the food on the table!

    1. ecofeco Silver badge

      Re: Where are the profits going?

      But they won't. They'll buy only the bargain basement solution in both hardware and skills.

  6. Steve Davies 3 Silver badge

    So TITSUP is becoming the new norm then?

    As they (and HP) have completely messed up, if I were a client, I'd be working with my lawyers to extacaste my company from the contract and be working with another hopefully more reliable supplier already.

    The total cock up here should serve as a lesson for the bean counting PHB's out there who think that 'we don't need no shitty DR site', and 'we don't need to do an DR testing'.

    1. ecofeco Silver badge

      Re: So TITSUP is becoming the new norm then?

      Every day my friend. Every day.

  7. ecofeco Silver badge

    10 days?

    Epic, epic FAIL.

  8. Anonymous Coward
    Anonymous Coward

    Quadruple redundancy!

    One working, one backup, one being repaired, and one that just broke down 10 minutes ago during repair of number 3.

    If it´s good enough for Nuclear Power Plant Computers, and Main Cooling System, it is good enough for me.

  9. Adrian Montagu

    Power Outages

    Power outages can have strange effects. If you have a large number of inductive loads like electric motors, the sudden disconnection of supply can result in a collapse in the magnetic fields and produce a reverse electric current a "back EMF (Electro-Motive-Force)". Any connected on this newly isolated circuit can be affected. This EMF will run in reverse along the neutral wires which may not be protected against surges. Anything above about 35 volts will zap a logic circuit device. The result is insiduous faults which may not show up at all, show up in 2 years or show up in 2 seconds. It becomes necessary, if this is the case, to replace ALL connected logic circuits to ensure proper running. I have also seen problems occur as a result of a lightning strike to the ground 150 yards away. This raised the ground voltage and fed current in via the earth wires which are connected to neutral wires in various places. The result is that the over voltage gets in and causes insiduous faults.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like