Sign in / up

back to article Dev put AWS keys on Github. Then BAD THINGS happened

Bots are crawling all over GitHub seeking secret keys, a developer served with a $2,375 Bitcoin mining bill found. DevFactor founder Andrew Hoffman said he used Figaro to secure Rails apps which published his Amazon S3 keys to his GitHub account. He noticed the blunder and pulled the keys within five minutes, but that was …

COMMENTS

Post your comment
House rules Send corrections
Add to 'My topics' unstar *
  1. Warm Braw

    What about pre-payment?

    Aside from the obvious issue of storing credentials securely, why isn't there a pre-payment model for these services so that you can limit the financial damage to the amount of your outstanding credits?

    This has always struck me as the achilles heel of cloud services - DDoS someone's trivial and seldom-visited website and you can stiff the owner for huge bandwidth and compute charges.

    I wouldn't hand over my credit card to anyone on the understanding that unlimited amounts could be charged to it.

    15 0 Reply
    1. Buzzword
      Reply Icon

      Re: What about pre-payment?

      Because the last thing you want your customers to see is "Sorry, this site has been suspended because they didn't have enough credit in their account." People are wary enough about buying stuff on unknown websites, without having messages like that pop up.

      0 0 Reply
      1. Jason Bloomberg Silver badge
        Reply Icon

        Re: What about pre-payment?

        "Sorry, this site has been suspended because they didn't have enough credit in their account."

        Except they don't have to put up a message of that kind. All they have to do is report the unavailability, not the reason why.

        Even if pre-payment isn't desirable it would be reasonable enough to put a cap on things, trigger an authorisation request from the payer to keep the service running and incurring costs. That could be opt-out for those who want to take the risk.

        Customers are going to demand things like that if Amazon stops refunding money for carelessness and mistakes.

        2 0 Reply
        1. Lusty
          Reply Icon

          Re: What about pre-payment?

          "Even if pre-payment isn't desirable it would be reasonable enough to put a cap on things, trigger an authorisation request from the payer to keep the service running and incurring costs. That could be opt-out for those who want to take the risk."

          FWIW Microsoft allow caps and warnings on Azure.

          1 0 Reply
    2. Ben Tasker
      Reply Icon

      Re: What about pre-payment?

      Aside from the obvious issue of storing credentials securely

      And, as appears to be relevant here - actually bothering to set up non-privileged keys. If they were spinning up EC2 instances (and the dev seems surprised by it) then either he was using a key with permission to do so (i.e. it's been configured in IAM) or more likely was using his root keys, granting the attacker unlimited access.

      Wonder whether he remembered to purge the keys from his commit history, a 5 minute window is pretty short...

      1 0 Reply
      1. bigtimehustler
        Reply Icon

        Re: What about pre-payment?

        The article specifically states they were S3 keys, so presumably he knows they were not the root keys. Still I think your right that those keys had too many permissions for what they required and the correct roles for them should have been set up.

        2 0 Reply
        1. Ben Tasker
          Reply Icon

          Re: What about pre-payment?

          Yup hadn't purged them fully (though he's now revoked) - https://github.com/andhof-mt/shriek/commit/799a62ed075954eac673322b9f69963ad815c4d0

          @bigtimehustler

          Looking at his post, I'm not sure they were just S3 keys, though it's hard to say for sure. Certainly can't find any reference (based on a _very_ quick google) to being able to fire up EC2 instances through the S3 API - though if it is true, that's some spectacularly bad design by Amazon.

          But yes, either way, they definitely had too many privileges. Mind you, if you look at the average S3 tutorial online, the various authors all seem to think that creating limited privileges in IAM is too complex and skip over it.

          2 0 Reply
          1. Alan Brown Silver badge
            Reply Icon

            Re: What about pre-payment?

            "Mind you, if you look at the average S3 tutorial online, the various authors all seem to think that creating limited privileges in IAM is too complex and skip over it."

            Perhaps Amazon should be all over such tutorials like a badly fitting shirt.

            0 0 Reply
    3. Graham 24
      Reply Icon

      Re: What about pre-payment?

      You get buy pre-paid "credit cards" with a fixed amount of money on them. They are intended for people who need a credit card, but can't get one for credit history reasons, but need the use of one. I don't see why a cloud service wouldn't accept one.

      Alternatively, if you use a genuine credit card, as opposed to a debit card, they usually have a credit limit built-in. The maximum financial damage might be higher than you would like, but it won't be unlimited.

      0 0 Reply
      1. Richy Freeway
        Reply Icon

        Re: What about pre-payment?

        They don't charge it straight to your card, they invoice you monthly. So a limited credit card would just result in an unpaid bill.

        4 0 Reply
      2. Ben Tasker
        Reply Icon

        Re: What about pre-payment?

        To be fair, if you're that determined to 'Pre-Pay' your Amazon account, it's fairly straight forward.

        Buy £50 of Amazon gift cards

        Apply them to your account

        Don't register a card against the account

        Whenever your bill comes out they'll subtract from the credit, though you can probably expect some shitty emails if you do go over.

        0 1 Reply
        1. frank ly
          Reply Icon

          Re: What about pre-payment?

          Pre-payment credit cards are not accepted by organisations that would want to apply debits to your card without your explicit permission for each transaction (or the card issuer will not allow them to be registered for this). I know this is true for Paypal since I tried to register one This is to prevent you from running up a large bill before they try to debit a card that only has a small amount of credit on it.

          0 0 Reply
          1. Anonymous Coward
            Reply Icon
            Anonymous Coward

            Re: What about pre-payment?

            I routinely use with PayPal a "virtual" credit card (it's a service from my bank, it gives me a valid credit card number different than my true one) set to a max amount, and expiring after n months.

            I never had issue registering and using it with PayPal. After all, whatever card you use, I guess those services checks you can pay before accepting the transaction.

            0 0 Reply
    4. Alan Brown Silver badge
      Reply Icon

      Re: What about pre-payment?

      This has always struck me as one of the largest problems of websites and it was worse 25 years ago.

      The Slashdot effect used to take down servers regularly, because of the financial impacts.

      0 0 Reply
  2. Anonymous Coward
    Anonymous Coward

    > "I only had S3 keys on my GitHub and they were gone within five minutes!"

    Assuming they got comitted to a git repo, I wonder if there were still in the version history. Seen that happen a few times on internal repos - people delete the file from HEAD, but nuking the history totally is much harder.

    1 0 Reply
    1. Ben Tasker
      Reply Icon

      Bang on the mark, in fact there are (at least) two different sets of API keys in his commit history, spread across 4 different commits.

      Note - are, not were - it's all still there. So the bot had much more than a 5 minute window to purloin them.

      2 0 Reply
  3. William Boyle

    This is why

    This is why you don't leave the house keys under the door mat!

    1 0 Reply
  4. Tom_

    Poor strategy from the bad guys?

    I wonder if a better strategy than spinning up the maximum number of instances would be to run them with much lower load, perhaps gradually increasing it over time. That way you may be more likely to slip under the radar and get more out of it before anyone notices.

    0 0 Reply
    1. Midnight
      Reply Icon

      Re: Poor strategy from the bad guys?

      Unless somebody else also finds the same keys and then it's time to play either "Prisoner's Dilemma" or "Tragedy of the Commons".

      2 0 Reply
  5. Anonymous Coward
    Anonymous Coward

    Two simple ways, which are common sense...

    1. Don't store credentials alongside code, ever. Change the code so that it picks up the credentials elsewhere, outside of what you commit to public (or any) repositories.

    That's best practice, if not common sense.

    Failing that, at least learn the ropes...

    2. Use .gitignore to avoid pushing sensivie files.

    Blaming it on the used framework is lame. But hey, everybody with little to no clue can call themselves "developer" these days and share the result of that for free with world and dog.

    3 0 Reply
    1. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: Two simple ways, which are common sense...

      It is easily done, I accidentally committed a signing key to my local repo early last year before I realised that my .gitignore wasn't quite right.

      It's the downside of automatically staging new files - though personally I think it's better than the alternative.

      The first key difference is that I checked my history and purged it from my local repo before I pushed it.

      (The second, possibly more important detail being that the repo I push to is also private.)

      0 0 Reply
  6. Destroy All Monsters Silver badge
    Paris Hilton

    "S3 keys", huh?

    Well, you CAN buy a keyfob for AWS access.

    0 0 Reply
  7. Daniel Voyce

    Storing stuff in environment variables FTW

    1 0 Reply
  8. Sirius Lee

    It is going to be the case that keys are going to be posted. The question is, why does AWS allow the default to be that someone who compromises an account is able to start 20 monster E3 instances in all 8 regions?

    This happened to me (no, my keys have never been public and AWS staff were unable to find any) and AWS did remove the credit. However it took a lot of correspondence to have them set the number of available instances in all regions of my account except 1 to zero and in the region I use, set it to 6 instances (3 running, 3 spare).

    In my correspondence I likened AWS to a credit provider who is delinquent in their responsibilities by letting creditors run up massive bills without even trying to limit the scope of their credit.

    I recommend to any other AWS users that in addition to following the advice to cycle keys regularly they also contact AWS support and ask them to prevent instances from running especially in regions they are unlikely to use.

    2 0 Reply
  9. sparkso

    This happened to me ! I got a $2336 bill for EC2 which i didn't use . I only use S3.

    I hope i can get my refund back.

    My developer did the same thing, mistakenly made the git public for a few hours. holy crap i am so depressed now.

    0 0 Reply

POST COMMENT House rules

Not a member of The Register? Create a new account here.

Forgotten password ?

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like