back to article DXC spills AWS private keys on public GitHub

Miscreants racked up a $64,000 bill on DXC Technologies' tab after a techie accidentally uploaded the outsourcing firm's private AWS key to a public GitHub repo. It was red faces all round as the business opened up on the classic crypto key fumble in a PDF memo to staff, the contents of which were seen by The Register. " …

  1. Anonymous Coward
    Anonymous Coward

    AWS has no billing controls...

    For some reason you can’t specify a limit your monthly bill on AWS. You can only put in email alerts when you reach certain levels.

    It’s almost like they want this accidental overspend to happen... I can’t imagine why...

    1. Anonymous Coward
      Anonymous Coward

      NOT

      They can thank their god that the attackers quickly notified them (by racking up a unusual bill) instead of using the key material to access DXC client's data for years. And secretly selling the data to China or the like.

    2. BagOfSpanners

      Re: AWS has no billing controls...

      And somewhat ironically, they charge you for sending you email alerts about the amount they are charging you.

      As others have said, DXC got off lightly. It's likely that large numbers of their staff will have learned useful security lessons from this, which would otherwise have required expensive training.

    3. defiler

      Re: AWS has no billing controls...

      For some reason you can’t specify a limit

      For the reason that Jeff has rockets to pay for!

      Easier to make money if you can bill without limit and then grudgingly offer a part-refund if you're feeling lenient.

  2. Stoneshop
    Devil

    Be aware of your surroundings and stay alert

    Yes, that. The world needs more lerts.

    Do not post about the company on social media if you are not authorised to do so

    Given El Reg's clientele this forum would better qualify as antisocial media, so posting here (authorised or not) would be OK

    1. Anonymous Coward
      Anonymous Coward

      Re: Be aware of your surroundings and stay alert

      And it's in the public interest. LOL.

  3. Anonymous Coward
    Anonymous Coward

    "create secure areas to store these details means the project lost two to four weeks delivery time"

    Err no, the project didn't "lose" any delivery time,.

    In reality, the project was previously running two to four weeks *early* because they'd decided to skip the essential security design and setup.

    1. Anonymous Coward
      Anonymous Coward

      Indeed

      They have the temerity to perform beancounting when they had a blunder with potential massive damage to their customer. For years and lots of millions of revenue. Of the CUSTOMER.

  4. AndGregor
    Thumb Up

    Cost

    How much Monero does 64k worth of AWS earn someone ?

  5. JMiles

    Techie dummy?

    "a member of the technical team created a personal space on the public Github"

    We always laugh about how silly the average user is with their online security but to read that a techie made this type of error is beyond amazing.

    Honestly, I would automatically fire any techie that had no understanding of the implications of publishing internal corporate material on a public resource. They shouldn't need training to tell them that's a bad idea FFS!

    1. rmason

      Re: Techie dummy?

      They don't mean "technical".

      They mean "a developer".

      I make my living providing, mainly basic, technical support to developers. Our business makes software, i'm internal sys admin.

      Developer != technically adept or knowledgeable about IT kit etc. you'd think sitting in front of one for however many years would breed familiarity, but no.

      Someone posted "the thing they need to make AWS things work" up to github. They (clearly) had no understanding of what the thing in question actually *was*.

      1. Anonymous Coward
        Anonymous Coward

        Re: Techie dummy?

        I find it interesting how non IT some of my fellow dev colleagues can be for basic stuff. But also I find it annoying how friends and family think that just because I toil for an ISP I can "sort their windows out" and "fix their wifi"...

        1. Sir Runcible Spoon
          Happy

          Re: Techie dummy?

          " just because I toil for an ISP I can "sort their windows out" and "fix their wifi"..."

          Can't you? Or would just rather not have to do it?

          1. david bates

            Re: Techie dummy?

            Ah - do it once and you're on the hook forever. For them, and their relatives, and their friends....

        2. Muscleguy

          Re: Techie dummy?

          Works in lots of ways. My first Postdoc I entered a lab filled with molecular biologists whose knowledge of biology was frankly woeful. There was an older version of me so together we taught them all enough to get them by. As a trade we learned molbio.

          1. Munchausen's proxy
            Pint

            Re: Techie dummy?

            "As a trade we learned molbio."

            Fundamental rule of molecular biology - If some is good, more is better.

            Leading to the definition of a molecular biologist:

            A biochemist who uses enzymes in stoichiometric amounts.

      2. JMiles

        Re: Techie dummy?

        I get that most developers will have limited knowledge on IT kit. However, if a developer has any business working with access keys / encryption tokens / etc. then they should understands the ins and outs of how they work and how to protect them (I don't suggest they need to understand the maths of it all) but otherwise they're just monkeys working with tools they don't understand.

        Oh wait. I just realised. DXC. I take it all back - monkeys working with tools they don't understand = normal.

        1. Anonymous Coward
          Anonymous Coward

          Re: Techie dummy?

          DXC = Developer-cross-Clowns?

    2. Mark 85

      Re: Techie dummy?

      By "member of the technical team" does everyone assume some level of tech expertise? It's possible it was an intern, admin (secretarial type), or even a paper-pushing manager of some sort.

  6. Anonymous Coward
    Anonymous Coward

    "But culturally there was an impact"

    So *someone* got fired then... despite the Biz not fully training *someone*, its like they have money to throw at law suits.

    1. Mark 85

      So *someone* got fired then... despite the Biz not fully training *someone*, its like they have money to throw at law suits executive bonuses

      FTFY

  7. Anonymous Coward
    Anonymous Coward

    Leader in security services

    From the DXC website:

    Recognized as a leader in security services, DXC Technology help clients prevent potential attack pathways, reduce cyberrisk and improve threat detection and incident response. Explore our solutions and learn how your enterprise can realize true gains in security readiness.

    Not so convincing following this blunder.

    1. macjules

      Re: Leader in security services

      Not to mention how proud they are of their AWS "Partnership" .. does that extend to "can we have our $64,000 back please?"

      1. Michael Strorm Silver badge

        Re: Leader in security services

        macjules: "can we have our $64,000 back please?"

        I guess that would be The $64,000 Question, then.

    2. Sir Runcible Spoon

      Re: Leader in security services

      To be honest, considering the blunder, their response to it is beyond my expectations for that particular organisation by several orders of magnitude. Go figure.

  8. Dwarf

    AWS Security 101

    1. Learn how the AWS IAM role based access works (its in the first chapters of the training for a reason)

    2. Don't publish your keys on-line

    They should engage with a security consultancy to get some advice on this.

    Oh, hang on a mo ....

    1. Anonymous Coward
      Anonymous Coward

      Re: AWS Security 101

      A few additional steps:

      3. Don't publish your keys on-line, someone WILL find them

      4. REALLY - don't publish your keys on-line, someone WILL find them and rack up a "huge" bill

      5. Once you publish you keys on-line and someone racks up a huge bill, pay it and consider it an important lesson. Cancel old keys, get NEW keys and go to step 1.

      If you hit step 5 more than twice, cloud isn't the issue...

  9. Mr Dogshit

    "A security incident exposure matrix was established"

    Oh, well done boys, you created a spreadsheet.

    Good job DXC doesn't do loads of US government defence work, eh?

    1. GetFuckedKid
      Facepalm

      Re: "A security incident exposure matrix was established"

      From what you just said you clearly have no understanding of how major IT corporations work, one blunder like this in a small area makes you automatically assume that none of their government work is secure either?

      Let's be real. In the grand scheme of things this is a small blip in comparison to major ball drops by companies like Accenture who actually made a considerable amount of sensitive data public. Whereas a few scripts to kick off server builds in AWS which the user creating them wouldn't have had access to is minimal. Next time get some background or at least read what was said about the incident in the post and not come to ridiculous conclusions with no leg to stand on.

      1. Mr Dogshit

        Re: "A security incident exposure matrix was established"

        A. That's not what I said.

        B. I've worked for both CSC and EDS.

  10. Anonymous Coward
    Joke

    This may be malicious....

    ..so they are checking the backgrounds of all disgruntled ex-employees. The checks are expected to complete by 2097.

    1. Anonymous Coward
      Anonymous Coward

      Re: This may be malicious....

      At an annual cost of £500,000. Charged to the client.

      Footnote: it wasn't a disgruntled ex-employee, it was a current senior employee ensuring they hit their revenue targets.

      Hopefully this is a joke...

  11. Sir Runcible Spoon
    Facepalm

    Doh!

    Legacy CSC colleagues lost confidence in our ability as a team to maintain secure information and even complete the work required. This also resulted in difficult interactions between colleagues on calls.

    So, some numpty from old HP then.

  12. Snowy Silver badge
    Facepalm

    [quote]"Various secure variables (cryptographic keys that allowed access to DXC procured Amazon Web Services resources) were hardcoded into a piece of work being shared between multiple teams and with the project architect."

    Then on September 27, a member of the technical team created a personal space on the public Github, and the code was loaded to this unsecured repository that allowed individuals as yet unknown to access and use it.[quote]

    Hard Coding cryptographic keys into anything sounds like a very stupid thing to do the unsecured repository just let it out into the wild.

  13. This post has been deleted by its author

  14. James 47

    How did an external person manage to ssh into the machines in order to get them to do anything not pre-build into the AMIs?

    1. Anonymous Coward
      Anonymous Coward

      The keys were used to create new VM's, I'm guessing for the purpose of mining bitcoin as a way of quickly turning the keys into something of value that can easily be cashed in once the unexplained usage is discovered.

  15. Anonymous Coward
    Anonymous Coward

    NOBODY PANIC...

    ...the shareholders and management will be protected. They'll just add half a dozen more critical delivery and a few of the best sales staff to the next round of weekly redundancies to cover the cost and to make the managers feel like they fixed it.

  16. David 55

    I am shocked...

    that they found out about it at all. At least some beancounters are still paying attention. I bet if someone was using the keys to silently steal confidential information they wouldn't have caught it in a million years.

  17. Anonymous Coward
    Anonymous Coward

    Erm.... https://www.theregister.co.uk/2017/11/09/dwp_dxc_tech_contracts_ending/

    Multiple DXC sources that either worked on the contracts or were close to them confirmed the changes. "The DWP decided to in-source," one insider told us.

    Another claimed the Hosting was being moved to Crown Hosting, "DWP has expressed a desire to be free of all DXC contracts by March 2018 when the desktop [agreement] comes up for renewal. Anything that can be virtualized is being placed in AWS." Not everyone we spoke to agreed with this summary.

  18. Anonymous Coward
    Anonymous Coward

    Actually, They Got Away CHEAP

    If the attackers had used a more subtle tactic, they would still have access to lots of AWS instances run by DXC on behalf of the client.

    The attackers would lie low and periodically check whether there exists data on the servers to be taken and sold to the Chinese competitor of DXC's client.

    That situation would probably have gone unnoticed for years, if they simply tipped off the developer to remove the code (after downloading all of it). The developer would have "assumed no damage" and kept the episode for himself.

    So: DXC have more luck than competence.

  19. Anonymous Coward
    Pint

    Massive regulations and no qualified staff?

    This is what you get if you cater to less qualified people and then try to make up for all that by enforcing a massive list of regulations. So massive that I honestly can't blame some for not fully following up on it (did they even understand the whole thing?).

    My stance is to get people qualified for the job, but unfortunately that often doesn't work. Because corners need to be cut. You can get a 30 - 40 year old veteran who is worth their money and would never make mistakes like this (for starters because they understand the concept of a public repository) but it also means their paycheck will be a little bit higher than the rookie you can get for less.

    And instead of training said rookies you just make them feel important and lay out a whole list of do's and don'ts for them to follow, sometimes (more than often) not even bothering to ensure that they understand why you laid out those rules and why they matter. Because... You need to feel important too of course and your will is law. So they have to comply because "you told them to" and that's all they need to know.

    That's not exactly creating a healthy working condition, and then this is often the result.

    Makes you wonder how much you could have saved if you had hired more qualified yet a little more expensive developers / engineers. Can't be more than this.

  20. Anonymous Coward
    Anonymous Coward

    Tribal Frankenfirm

    "Incredibly, DXC admitted a full probe had revealed some in the team were "not briefed on the compliance standards and have not received adequate security training"."

    That must be the most sarcastic "incredibly" seen for a long time! ;-)

    Also, why would anyone with half a brain put the "Legacy CSC colleagues lost confidence... ...resulted in difficult interactions between colleagues on calls" statement in writing, even internally? Bound to leak, and whilst bad enough in itself, it also exposes that DXC is still far from one organisation and the tribal warring is far from contained. Frankenfirm, indeed!

    1. Lysenko

      Re: Tribal Frankenfirm

      Even more bizarrely, they're admitting that the "Legacy" (excellent morale builder, right there) staff are the ones who actually understand security and that their bright, thrusting, wave of the future colleagues were responsible for the cock up; in other words, competency is officially designated as a "legacy" attribute. I sense the malign presence of "DevOps", probably in conjunction with a member of the Maldives Basketball Association.

      1. whileI'mhere

        Re: Tribal Frankenfirm

        It is ambiguous. "legacy CSC staff" could be as opposed to "bright, thrusting wave of the future colleagues" or it could be as opposed to "legacy HPE colleagues". Either way, "one DXC" is clearly fractured and the splints are not yet working.

        1. Anonymous Coward
          Anonymous Coward

          Re: Tribal Frankenfirm

          CSC was already an unhappy patchwork of old, new, EDS, HPE from a decade ago, and various client staff (mostly ex-Civil servants) who'd been slurped up on various accounts. Many of whom felt they'd been lied to, either when hired or TUPE'd over, about the nature of the salary on offer.

          Add to that an ongoing turf war between project staffing and central resources and one of the most fecklessly incompetent layers of mid to senior management you'd ever encounter, it's only a surprise this sort of thing doesn't happen more often.

    2. BeakUpBottom

      Re: Tribal Frankenfirm

      I thought that, but then maybe the grammar is ambiguous. It's incredible they admitted it.

  21. Anonymous Coward
    Anonymous Coward

    AC as I was dumb.

    I did this on my personal AWS account a few weeks ago.

    Earlier in the year I'd been learning Terraform and, as a quick hack, had put my AWS credentials into the throwaway code. Months later I'd been doing Terraform properly, with the credentials held in ~/.aws

    As it was crash-and-burn code, I'd been checking it into a public github account. Last thing before leaving I checked in the earlier code as well, feeling pleased with how I'd progressed in that day.

    The problem with AWS is that the billing is very far from real time - this is why they can't offer monthly price caps, the information is not available to do that. I had a Cloudwatch alarm set up to email me when I spent over $10 in a month - it went off the following morning at 10am, by which time the miscreants had racked up $6,000 with xlarge instances in each location. AWS issued a credit for the amount - yes I'd been dumb, but the exact alerting mechanism to protect and alert you if you've been dumb takes many hours to function. This is OK if you've left a tap running in the bathroom, but no help if someone turns a firehose down your chimney.

    So clearly in this case the developer did A Very Bad Thing, putting corporate code into a public github account. Reprehensible. Other developers had already done A Very Bad Thing hard coding credentials into code - not least, this prevents you from rotating your keys.

    Nobody comes out well, and a wild guess says that AWS offered to refund the $64,000 if DXC went public as a cautionary tale.

    1. defiler

      Ouch - I bet you had sweaty palms.

      Let's be honest, we're not human if we've not made some stupid mistake at some point. Like installing an Exchange Server 5.5 patch in the sure and certain knowledge that it'll tell you when it's finished and leave you with a restart button for the evening. Then it closes. Then all the other windows close. Then the taskbar disappears. Then you're sitting there feeling clammy, staring at the screen, mouth agape, and gradually going paler and paler. Then the phone rings...

      Still, you learn. And when the junior guy suggests just cracking that patch on just now instead of having to waste so much more time in the evening, you can cut him off and tell him you war stories. :)

    2. Anonymous Coward
      Anonymous Coward

      "...a wild guess says that AWS offered to refund the $64,000 if DXC went public as a cautionary tale."

      Interesting speculation. Most corp's (or in DXC's case, 'corpse') would swallow the $64k and avoid the commercial embarrassment. But at DXC every dollar towards Mikey's personal "I win 'cos I got more prizes" fund must be saved, and damn the wider consequences. If true, clearly Amazon knows its client very well indeed, and knew they could not resist.

  22. This post has been deleted by its author

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like