IT Pro confession: How I helped in the BIGGEST DDoS OF ALL TIME • The Register Forums

Thursday 28th March 2013 14:33 GMT DS 1

You are right.. and wrong

Firstly, brave of you to actually outline your edge of network box and its somewhat out of date configs - thus making it a target - for this - 50 lashes :)

DNSSEC and DNS work require skills and training, things businesses of many a note cease to provide. They have reached a point where like everything else, they can't be bothered to do right.

And 50 lashes for you claiming guilt. You did not start this attack, nor did you ask for it. There are however people who are responsible, and they are not being dealt with. Put the blame where it really lies. Cyberbunker and its miscreants.

21 1
1. Thursday 28th March 2013 14:48 GMT LarsG
  
  Re: You are right.. and wrong
  
  The Biggest DDOS attack...... Until the next time
  
  2 2
2. Thursday 28th March 2013 15:56 GMT Trevor_Pott
  
  Re: You are right.. and wrong
  
  Network edge is getting upgraded. Have you seen these? I think I'm in love.
  
  0 1
  1. Thursday 28th March 2013 16:57 GMT Aitor 1
    
    Re: You are right.. and wrong
    
    Quite nice low power sever, still I find it a bit expensive (<1000$), but my guess is that you will use it non stop for 6 years.. and use the old one as backup , right?
    
    1 0
    1. Thursday 28th March 2013 17:28 GMT Trevor_Pott
      
      Re: You are right.. and wrong
      
      Yeah. Actually, the "old one" is actually 2 Atoms. (The primary and the cold spare.) So I would have 2 spares on the shelf to back up the shiny. That said, it would cost far more if I were to try doing the exact same thing but with CISCO on the box...
      
      2 0
      1. Saturday 30th March 2013 14:47 GMT leexgx
        
        Re: You are right.. and wrong
        
        well the good thing is you at least had the sense to email you once your DNS was under attack/high load was happening, most who setup and DNS and forget about it, most do not even notice it as they have high speed connections normally
        
        most other DNS providers just send an angry email when they are sending out large amounts of data when in fact they should be fixing there own DNS to stop it from doing that (cloudflare has had lots of angry emails when an DNS amplification attack is going on, when it is there Own fault for setting up the DNS incorrectly)
        
        1 0
      2. Sunday 31st March 2013 17:28 GMT Fatman
        
        Re: with CISCO on the box...
        
        You know someone has to pay for those executive bonuses and excessive marketing costs, don't you????
        
        0 0
  2. Friday 29th March 2013 06:27 GMT MondoMan
    
    Re: new edge not so much...
    
    It's only available in black??? You would think all those gamers who grew up to be sysadmins would have forced the industry to offer some nice UV fluoro-bling!
    
    1 0
  3. Friday 29th March 2013 06:58 GMT Mike Pellatt
    
    Re: You are right.. and wrong
    
    I've been in love with Supermicro for ages :-) They seem to get server design Right, especially mid-range, unlike Dell (wot, only 2.5" drives in your 2U server ?? I'm not made of money)
    
    0 0
3. Thursday 28th March 2013 16:12 GMT robin48gx
  
  Re: You are right.. and wrong
  
  Ah I thought DNS worked by looking at /etc/hosts and then always doing a recursive if it was not in there.
  
  The plot thickens, thanks for explaining this point.
  
  2 0
This post has been deleted by its author
1. Thursday 28th March 2013 15:04 GMT Professor Clifton Shallot
  
  Re: interesting
  
  Yes, more of this sort of thing, please, Reg.
  
  21 0
Thursday 28th March 2013 14:54 GMT Anonymous Coward

You're lucky...

That your servers didn't reside on networks from TransIP, LeaseWeb (Dutch hosting providers) or even GoDaddy. Because setting up a recursive DNS server completely open for all to (ab)use gives them the right to terminate your connection entirely until you "fix your shit", and to my knowledge they will exercise that right too.

DNS administration isn't for everyone. No offense but there are some mistakes you shouldn't make, this only tells me that you didn't go over your entire setup as carefully as you should have. Also makes me wonder if you really take (/ took?) DNS as seriously as you should.

Because that's the problem with the Net these days: ignorance. "Nah, I'll fix that "tomorrow" because there's no way they'll abuse a server as small as mine".

bzzzzt.

6 6
1. Thursday 28th March 2013 16:23 GMT Tom 38
  
  Re: You're lucky...
  
  Yep, top fail. You should never run an authoritative DNS server as a cache, you should run separate instances of them on different interfaces if you require both DNS caching/recursive lookup services internally and authoritative DNS externally.
  
  If DNS isn't your main job, you might look at easier to use alternatives to BIND. BIND is really powerful, but some of that power is the ability to shoot yourself in the foot. Something like djbdns is much more thought out and less error prone for the novice than BIND.
  
  8 0
  1. Thursday 28th March 2013 19:58 GMT Anonymous Coward
    
    Re: You're lucky...
    
    I would highly recommend using DJBDNS over BIND.
    
    For a start, the tinydns server itself, apart from being extremely fast and low impact on the server, doesn't respond to recursive queries, and doesn't have any of the numerous bugs and flaws that BIND has. I believe Dan Berstein is still waiting to pay out on his promised reward to the first person to report a verified security hole - http://cr.yp.to/djbdns/guarantee.html.
    
    Secondly, the function of recursive lookups is provided by dnscache, a companion daemon which you only expose to your internal network. I use addresses in the private range, and have it listening only on an internal NIC.
    
    6 0
    1. Friday 29th March 2013 09:55 GMT Destroy All Monsters
      
      Re: You're lucky...
      
      Oh yes. After BIND turned out to be an incomprehensible mess that moreover opened up the server to hacks back in the '00s, I turned to djbdns. Easy, comprehensible, straightforward. It had some bizarre quirks and illiberal behaviour due to the grammar nazi attitude of Bernstein which earned me complains from the front office in some rare cases.
      
      Though patching it at each install to make it work with modern compilers was a pain. Way to go putting up that kind retarded license prohibiting third-party distros, DJB.
      
      I hear licensing has been fixed in the meantime and improvements are rolling out but, well, I'm using PowerDNS now. Recommended and you can inject LUA scripts.
      
      1 0
      1. Friday 29th March 2013 15:10 GMT PyLETS
        
        DJBDNS now public domain.
        
        Since 2007. And nicely supported in Debian package format. Had to compile it myself before then.
        
        https://en.wikipedia.org/wiki/Djbdns
        
        0 0
Thursday 28th March 2013 14:58 GMT ukgnome

This sort of article is why I continue my readership of the Reg.

A very informative article.

26 2
Thursday 28th March 2013 15:00 GMT Alex Brett

As I understand it this wasn't actually a DNS amplification attack as you described in the article, instead they were sending DNS requests with the source address spoofed to be the target, causing the DNS servers to send its response directly to the target. As the request is quite small it isn't too difficult to send lots of them, and by targetting the request appropriately you can get the response to be quite large, thus causing the DDOS.

This also means that often simply turning recursion off in BIND is not sufficient, as in the default configuration it will likely (depending on which version of BIND) still return the list of root servers as a referral instead of simply refusing the query. The list of root servers is quite a large response on its own, and thus can be used in this attack.

The magic line you need to add is "additional-from-cache no;" - this will stop that behaviour.

7 2
1. Thursday 28th March 2013 18:19 GMT pixl97
  
  Yes, it was an amplification attack. The attacker sends a small packet, bytes generally, the server replies with a larger packet. 512 bytes with the old behavior and much larger packets with the large udp packet behavior. Any request that sends back more data then send to the wrong host can be described as an amplification attack. It's the magnitude of the DNS response that makes it so effective.
  
  You are right on BIND, the correct response is not to return large amounts of data if you don't have the answer.
  
  4 1
2. Friday 29th March 2013 15:10 GMT PyLETS
  
  @Alex Brett
  
  "As I understand it this wasn't actually a DNS amplification attack as you described in the article, instead they were sending DNS requests with the source address spoofed to be the target, causing the DNS servers to send its response directly to the target."
  
  This aspect of the attack would be largely defeated by BCP 38, also known as RFC 2827 . Basically, spoofed UDP packets pretending to be from the party being attacked would not get through an ISPs customer-facing routers, unless sent from the same network as the party being attacked.
  
  Sure, DNS configurations also need tightening up here as Trevor and some of the comments describe, but this is also something the ISPs need to get to grips with, and start offering worse peering terms to those which don't .
  
  1 0
  1. Saturday 30th March 2013 01:31 GMT Oninoshiko
    
    Re: @PyLETS
    
    While that's great IN THEORY, and I would love for ISPs to jump on this, as a DNS operator you can't depend on the ISP to do this. We should be checking for this kind of thing as early as possible, yes, but we also need to sanity check at later points for casses where some don't get it right or don't care.
    
    Are the ISPs part of the problem, yes, but Trevor was not part of the solution (until recently). Hats off to maning up about it, and solving it.
    
    0 0
  2. Sunday 7th April 2013 05:56 GMT Anonymous Coward
    
    Re: @Alex Brett
    
    I wholeheartedly agree.
    
    There's no reason that I should have to block inbound traffic from 172.16/12, 10/8, 169.254/16, or 192.168/16 on the outside interface of my firewall, but I do.
    
    0 0
Thursday 28th March 2013 15:02 GMT IT Hack

Well done for not putting your head in the sand and actually doing a post mortem and actually publishing it. Kudos.

20 2
Thursday 28th March 2013 15:11 GMT koolholio

All relational to root

DNS amplification Is more to do with forwarding queries to root servers than recursive lookups, if you wish to have a look through some whitepapers, although recursive lookups can play a part of it, hopefully you enabled DNS spoofing/DNS cache pollution protection too and spool size to limit queries to an adjustable level?

Rate limitation only works so far given it can be small but many... the counter to this is big but few

What about 3 Mobile's dns resolvers that queries/matches the ip listed in the SOA record and checks that against the A record... and will fail to resolve if one cannot be resolved successfully... as is, with section 2.4.2 of here: http://www.iana.org/procedures/nameserver-requirements.html

1 0
Thursday 28th March 2013 15:38 GMT Christoph

See also SANS

The SANS Internet Storm Center has a good article on this

0 1
1. Thursday 28th March 2013 17:37 GMT Gordon Fecyk
  
  Ugh, "Cybergeddon."
  
  Just the link makes me cringe in disgust: "The great DDoS Cybergeddon of 2013." We're still here, aren't we? No one was hurt, right? Maybe spam increased for a few nanoseconds while Spamhaus was unreachable for a short time? Do I need to warm up my snow blower to deal with the deluge of junk e-mail?
  
  Seeing as I'm posting this to a Europe-connected network that was supposedly strongly impacted by this, I'd say this was a storm in a teapot. But SANS has a history of sensationalizing internet events. They want to be the weather.com of cyberspace and it's embarrassing.
  
  1 0
  1. Thursday 28th March 2013 20:27 GMT GregC
    
    Re: Ugh, "Cybergeddon."
    
    Did you actually read the article? They make it very clear early on that:
    
    "Here is what did not happen: the Internet did not come close to coming down, not much real impact was felt outside the victims and those in close Internet-proximity to them and we were all still able to get to pinterest and see cat pictures online."
    
    I got the distinct impression that the title was meant to be ironic, the piece itself was (like Trevor's, I should say) both informative and interesting, especially to someone who doesn't know the subject well (ie, me!)
    
    3 0
    1. Thursday 28th March 2013 20:58 GMT Trevor_Pott
      
      @GregC
      
      If my blitherings are interesting then I fear a walkabout outside where the daystar is might be advisable. It seems you need some of those photons that the great big ball of fusion in the sky spits out to help you create some vitamin D and jumpstart the "removing crazy" subsystems. That or oh look it's beer o'clock on a long weekend, bye!
      
      1 0
      1. Thursday 28th March 2013 22:13 GMT GregC
        
        Re: @GregC
        
        No need to worry on my account, all things in moderation! I'm now geeked out for the week and will indeed be enjoying the sunshine at the weekend.
        
        Nope, hang on. I suspect I'll be frozen solid by the time Sunday's done...
        
        2 0
        
        Thursday 28th March 2013 23:33 GMT Trevor_Pott
        
        Re: @GregC
        
        So long as you have a good weekend, sir, then all is good. Cheers and beers!
        
        0 0
    2. Friday 29th March 2013 19:46 GMT Gordon Fecyk
      
      Then SANS needs to stop cheapening Armageddon. And Pearl Harbor. And September 11th.
      
      Catch phrases that invoke disgust:
      
      "Digital Pearl Harbor."
      
      "Digital Armageddon."
      
      "Digital 9/11."
      
      No one died due to any event reported using these catch phrases. Real people died in the real events. That is my problem: Cheapening the memory of real life events that killed real people.
      
      If they want to stop disgusting me, SANS needs to stop using them.
      
      1 5
      1. Friday 29th March 2013 22:00 GMT Destroy All Monsters
        
        I shall have to demoralize you...
        
        Unless you are a born-again X-tian, Armageddon is not a "real event".
        
        You may also notice that "Pearl Harbor" and "9/11" were attacks on Amurrica used at full efficiency to rally the rubes behind fascistic agendas of full-scale domination, so in a sense "digitalizing" them is fully in-line with we are looking at here.
        
        You are also missing "Digital Attack on the Maine" and "Digital Gulf of Tonkin".
        
        2 3
        
        Friday 29th March 2013 22:16 GMT Trevor_Pott
        
        Re: I shall have to demoralize you...
        
        Far more tragic would be the digital death of our beloved Playmonaut; a tragedy of intertubes proportions which would cause Register readers to rise up in droves against the evil aggressor.
        
        R.I.P. Playmonaut.
        
        0 0
Thursday 28th March 2013 15:44 GMT joeW

A++ read

It's worth 50 bits of click-bait about Stephen Fry or Freetards.

13 0
Thursday 28th March 2013 15:56 GMT The Vociferous Time Waster

Wait...

You put all your traffic to your 'datacenter' through *one* server? I bet you run SBS as well.

1 1
1. Thursday 28th March 2013 17:30 GMT Trevor_Pott
  
  Re: Wait...
  
  There are redundancies. It's actually a cluster of 2 devices. I didn't really want to get that deep into it though. I wanted to talk about the DNS not setting up a cluster in CentOS.
  
  1 0
Thursday 28th March 2013 16:23 GMT John Deeb

never "forget" any edge system!

Since my first time as victim of a hack was because of out-to-date BIND somewhere later in the nineties when I was really just a junior apprentice admin, I'd never "forget" what was running at the edge somewhere. It was Christmas morning though and I had to try to investigate from home using shaky ISDN dialup, no physical transport being available and trying to remember various IP addresses since all DNS was down. But the lesson I learned there was that the first machines I'd upgrade or keep at least to a recent level would be the ones opened up to the Interwebs like that. That's the only flaw I could find in the article's analysis, to have the priorities really ass backwards as it was stated that almost everything was gone over to CentOS 6 apart from one of the main attack vectors in the network! Not knowing or investigating BIND's defaults seems nearly irrelevant (although might have helped). Better have a proper upgrade policy and inventory in place. That's where proper management and security starts but definitely will not end. But the scrubber might have seen like a hassle to upgrade?

2 0
1. Thursday 28th March 2013 17:32 GMT Trevor_Pott
  
  Re: never "forget" any edge system!
  
  RHEL 5 is still under active support. There's nothing wrong with using it in live production. The system is updated religiously. The hardware refresh would have taken it to 6, but that got knocked back by about 6 months into April.
  
  So while I may have forgotten which OS the little blighter was running (my mental filing system had ticked it over to "CentOS 6" already) it was still running a maintained, patched, and secure OS.
  
  0 0
  1. Friday 29th March 2013 16:02 GMT foo_bar_baz
    
    Re: never "forget" any edge system!
    
    RHEL (and CentOS) are meant to be used in enterprise environments where things don't change quickly. This means Red Hat maintains an "old" version of a software component way after others have upgraded, religiously backporting patches to make sure it's secure. In other words stuff like BIND and openssh will have ancient version numbers, but still be up to date.
    
    If you want cutting edge, either compile the software yourself or use another distro. I like CentOS myself.
    
    0 0
Thursday 28th March 2013 16:45 GMT Gordon Fecyk

Here's a workaround on Windows Server

This was unexpected; Thanks for coming forward. It made me brave enough to come forward with a similar workaround I did on my Windows Server setup.

I publish a small handful of domains on that HP Microserver I bragged about a few weeks ago. One of those domains is an Active Directory domain as well as a publicly visible domain, and they handle internet e-mail and other internet things, so the domain controllers and dependent servers need to do recursive queries on the DCs running DNS, as well as host the DNS zones that make AD possible.

Windows doesn't have an IP access list saying who can do recursive queries and who cannot. But nothing stops you from copying the zones to another non-DC DNS server and disabling recursion on that.

I already have a reverse proxy server for various things; I just added DNS to it and port-forwarded DNS connections to it instead of to one of the DCs. Then I set up secondary copies of all my zones on it, and disabled recursion. I now have an edge DNS server that doesn't allow recursive queries and still acts authoritatively for my zones. I can still permit zone transfers from it to authorized servers outside as well, and do notifications of zone changes.

Sure, this is, 'duh, captain obvious' stuff for some. Who would have thought DNS would be used as a DDoS vector though? If Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott can come forward with this, so can I.

1 0
1. Thursday 28th March 2013 17:45 GMT Trevor_Pott
  
  Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott
  
  Surely I'm not that negative! I like stuff that actually works just fine. I get tetchy when it doesn't do what it is supposed to. Or costs more than having a human do the same job. Or I haven't had coffee...
  
  9 0
  1. Thursday 28th March 2013 20:18 GMT Gordon Fecyk
    
    Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott
    
    That was the impression I got here and here. Who in these forums brought up the impact of first impressions?
    
    0 0
    1. Thursday 28th March 2013 20:54 GMT Trevor_Pott
      
      Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott
      
      I calls things like I sees 'em. Good or bad. I don't hate any technology - except the things that let you robo call people in the middle of the night - but I do hate it when technology is badly implemented. A great example is saying "Trevor hates Windows.' I don't. Not even a little.
      
      Oh, I hate lots of things about how Windows 8 and Server 2012 have been handled, but this doesn't mean I hate all of those operating systems. Nor does it mean I think they can never be made to not suck in future releases. I fact I have litterally begged Microsoft to make the relevant changes...because I think Windows is a useful tool.
      
      I hate Microsoft's licensing department. I love Microsoft's storage team. I hate some of the very strict rules that Spiceworks has surrounding community interaction, but I love the opportunity it presents me to interact with other sysadmins and vendors.
      
      You really, really, have to do an awful lot to get on my personal hate list. Even Oracle isn't on my "no buy, ever" list. And they take hostages! Sony, on the other hand, will not see a single dollar from me ever after that rootkit fiasco. That is how you make Trevor hate you. That right there.
      
      Technology is a tool. Corporations are groups of people each with individual hopes, dreams, goals and ideals. If technology sucks then I'll pan it. If it's great, I'll praise it. If it's boring, I probably won't even write about it. I'm harsh. I'm honest. I'm as up front as I know how to be. That's my job after all...
      
      ...biting the hand that feeds IT.
      
      13 0
      1. Sunday 31st March 2013 17:43 GMT Fatman
        
        Re: Trevor, "I hate Windows/Java/Flash/PDF/QT/TheWorld," Pott
        
        Sony, on the other hand, will not see a single dollar from me ever after that rootkit fiasco. That is how you make Trevor hate you. That right there.
        
        You can add me to the club of people who have a severe disdain for anything SONY for exactly the same reason (a rootkit infection that killed a system).
        
        0 0
2. Friday 29th March 2013 00:49 GMT koolholio
  
  Re: Here's a workaround on Windows Server
  
  http://support.microsoft.com/kb/198408 <<< theres some hidden gems within!
  
  0 1
3. Friday 29th March 2013 08:16 GMT Anonymous Coward
  
  Re: Here's a workaround on Windows Server
  
  If you just "need to do recursive queries" for "internet e-mail and other internet things", then you should consider removing all the root servers from your root hint list, and just using your (local and) upstream provider(s).
  
  This doesn't make the problem go away -- you can still get rogue external requests -- put it translates them into a problem for your upstream supplier. In a DDOS situation, (1) your upstream supplier will have (hopefully) respond from cache, and (hopefully) have better quallified 24 hr support, who can pull the plug.
  
  0 0
Thursday 28th March 2013 16:56 GMT Colin Miller

filter on sane sender address?

Do ISP filter *outbound* traffic to ensure that the sender's address is sane?

The filtering could be done around the NAS level, or at the peer-exchange (LINX).

0 0
1. Saturday 30th March 2013 01:41 GMT Oninoshiko
  
  Re: filter on sane sender address?
  
  Most, no.
  
  should they? yes.
  
  1 0
Thursday 28th March 2013 20:25 GMT J.G.Harston

Kessel Run?

"it averaged 500Kbit/ses"

For how long? That's a speed, not a volume.

1 0