back to article 'I crashed AOL for 19 hours and messed up global email for a week'

Welcome again to Who, me? In case you've missed previous editions of the column, it's a confessional in which readers share their stories of having broken stuff. Badly. This week, a fellow reader named "Bert" asked you to "Cast your mind back to the mid-90s when America Online (AOL) was the biggest online service and the …

  1. Ilgaz

    I wonder if he is the same guy

    I remember one guy explained something about AOL mail inner workings on Slashdot, a usual "citation needed" nerd asked "How would you know?", he replied "Because I coded it."

  2. Anonymous Coward
    Anonymous Coward

    http://edition.cnn.com/TECH/9608/09/aol.fallout/index.html

    " The Internet is developing into a mainstay, and cyber junkies are finding there are plenty of other on-line services out there, many with on-line fees significantly less expensive than AOL. "

    Those were the days

    1. Paratrooping Parrot
      Holmes

      This brings back memories. I remember the early days using bbcnc, cnn and cnet, although this was at university. Icon for reminiscing.

  3. Anonymous Coward
    Anonymous Coward

    One of my network colleagues came into the office one morning to announce that the internet had finally reached a tipping point of public awareness in the UK.

    He had just followed a van down the A1M motorway - which had a company URL painted on the back door. Probably late 1990s.

    1. Korev Silver badge
      Pirate

      In the late 90s there were a couple of cars in Truro, Cornwall which had email addresses printed in a very large font on. I guess they must have thought they were the coolest guys on the planet :)

      As Penzance isn't far away -->

      1. Prst. V.Jeltz Silver badge

        I was in USA in 95 , and noticed all the movie posters had URLs on them

      2. Aladdin Sane
        Trollface

        In the late 90s there were a couple of cars in Truro, Cornwall which had email addresses printed in a very large font on. I guess they must have thought they were the coolest guys on the planet

        FTFY.

        1. SkippyBing

          'In the late 90s there were a couple of cars in Truro, Cornwall'

          Typical northern metropolitan elite with their private transport.

          1. The Oncoming Scorn Silver badge
            Coat

            But

            Truro, Cornwall is in the South West (They still can't do a cream tea correctly though).

            1. Aladdin Sane

              Re: But

              It may be in the south west of Great Britain, but it's still north of The Lizard.

              And nothing tops clotted cream, you heathen.

              1. jake Silver badge

                Re: But

                "nothing tops clotted cream, you clot."

                FTFY

            2. SkippyBing

              Re: But

              'Truro, Cornwall is in the South West'

              If you come from the right side of the 50th parallel Helston is dangerously northern.

        2. Korev Silver badge
          Devil

          The guys with round wheels were unbearably smug too....

          the work of the devil them -->

    2. Badbob

      Probably fully formatted on the vehicle as:

      http://www.company_name.co.uk/index.html

      I had a friend who operated a mobile disco service, and considered himself to be quite “with it” on the internet. He wasn’t bad, as he was perfectly capable of setting up a hosted domain and coding a rudimentary HTML page complete with every flashing and marquee effect the W3C had made available. An early adopter of internet advertising, he had posted his URL on his brand new vehicle INSTEAD OF his phone number.

      His business pretty much evaporated overnight. For two reasons. One of which was that it was 1997 and no-one that seemed to require his services owned a computer, let alone a modem. Secondly, the URL on his van had omitted all the non alphabet formatting... it said along the lines of httpwwwmobilediscocouk.

      1. Prst. V.Jeltz Silver badge
        Facepalm

        "INSTEAD OF his phone number."

        just wow

        1. Anonymous Coward
          Anonymous Coward

          When I was at university in Oxford in the early 80s there was a company selling audio equipement who used their phone number as their business name with regular ads in the student what's on listings with the tag line "you know our name, you know our number" ... which some of us referred to as "you don't know our number, you'll never find it!"

        2. David Nash Silver badge

          "INSTEAD OF his phone number."

          Do people get phone numbers from random vans in the street anyway?

          I would normally look it up, back in the day in a phone book, these days online of course.

          Vans don't come into it.

          1. DropBear

            Re: "INSTEAD OF his phone number."

            From random vans? No idea, never tried. From a passing cab? Hells yeah!

            1. jake Silver badge

              Re: "INSTEAD OF his phone number."

              One of my license plate frames had my Berkeley bang-path address across the bottom in about 1980. Still does, in fact. The top line reads "Go Bears!". I got all kinds of shit about it when I got to Stanford ;-)

              1. Anonymous Coward
                Anonymous Coward

                Re: "INSTEAD OF his phone number."

                One of my license plate frames had my Berkeley bang-path address across the bottom in about 1980

                "bang-path" - we don't wish to know about your extra-curricular activities sir.

          2. Gomez

            Re: "INSTEAD OF his phone number."

            Actually, this morning I took a photo of the local sandwich shop delivery van in Tesco's car park as it had the number on the side and we might need a buffet next week!

  4. ArrZarr Silver badge
    Go

    With hindsight

    Out of curiosity, would anybody be able to explain how you could fix/prevent this if the only load balancers available were wholly insufficient for the task?

    20/20 hindsight is allowed but you only have the existing resources of 1995/6

    1. Aladdin Sane

      Re: With hindsight

      Yes.

    2. TRT Silver badge

      Re: With hindsight

      A the true face of capitalism. Theory would suggest that as there is a gap/need in the market for some meatier load-balancer, someone will invest the capital to exploit that gap. Reality suggests that it's easier to find a cheap kludge than capital.

    3. Prst. V.Jeltz Silver badge

      Re: With hindsight

      did the story actually tell us how the outage was resolved?

      1. TRT Silver badge

        Re: With hindsight

        I was wondering what triggered the cascade failure in the first place.

    4. Frederic Bloggs

      Re: With hindsight

      A better solution would have been exim, seeing as this resource exhaustion tale has all the hallmarks of a sendmail shop. In the 1990's a friend of mine was running a forum mailing list for a very well known computer software company and was suffering the same sort of problem caused by sendmail's 1 msg then 1 delivery -> 1 process to deliver it "paradigm". A bit of a problem on a forum with 5000 people on it. He changed to exim and the problem went away.

      1. jake Silver badge

        Re: With hindsight

        exim wasn't really ready for prime time until around 1998, some would say 2000.

        1. Trixr

          Re: With hindsight

          Correct - Exim was "experimental" for a fair few years. Postfix came out in 1998, but I didn't use it till v2.2.

          smtpd_timeout was a lovely thing, not to mention all the smtp client timeouts going the other way (no waiting forever tying up a process waiting for a receiving MTA to respond).

    5. Tom 7

      Re: With hindsight

      Commercial load balancers were a bit shit but you could do something with a fast PC with a handful of network cards installed. They had FDDI on the outside and even then you could install versions of Linux that could then fan out that FDDI with a host of machines with 10Mbit ethernet on the inside, either simple round robin or something slightly more intelligent.

      1. jake Silver badge

        Re: With hindsight

        All they had to do to figure out load balancing was ask one of the major Usenet outfits. Wouldn't have changed the main problem, though ... AOL's mail system was a home-grown clusterfuck that had Internet email grafted onto it as an afterthought. The guy that ran it (not this "Bert" character, but rather someone I'll call "L") told me on more than one occasion that he wished he could rip it all out and start over with something sane like sendmail (!!), but the PTB wouldn't let him.

        qmail and postfix had nothing to do with AOL's issues. qmail was written to address security issues that weren't seen as a problem when sendmail was written; postfix was written to be an easier alternative to sendmail. qmail's bones were laid down in late 1995, before the AOL meltdown. postfix was an IBM-Watson reearch project about a year later, and I have it on good authority that IBM didn't give a rat's ass about anything AOL was doing.

        1. CrazyOldCatMan Silver badge

          Re: With hindsight

          qmail was written to address security issues that weren't seen as a problem when sendmail was written

          Indeed. My home[1] use of sendmail was *extremely* short (as in a matter of days) I then switched over to this new, secure and lightweight qmail (and later added on ezmlm rather than listserv for mail list handling)

          From the article - remember this 1996 when servers were feeble – the server would probably reboot

          I've *never* had one of my linux boxes reboor itself because of swap exhaustion. Probably because I use the above-mentioned qmail (and postfix on FreeBSD boxes) rather than sendmail. Both of which handle queuing a hell of a lot better than sendmail does (not difficult!)

          [1] I foolishly later took a job herding Sun boxen - all of which used sendmail. So I got to experience the joys later. Including on making it play nicely with Exchange 5.5 - which advertised that it supported ESMTP. At which point the Solaris sendmail tried doing batch SMTP delivery - which Exchange couldn't handle and so silently discarded the emails. Fixed by telling sendmail to ignore ESMTP announcments for the internal set of IP addresses that the various Exchange boxes used. Those were the good old days of packet-switched frame relay networks..

    6. SImon Hobson Bronze badge

      Re: With hindsight

      I had a thought about this ...

      They could have fiddled with the DNS to get a poor mans load balancer. Set the mx to (say) a.domain.tld with (say) a ttl of 3 hours. After (say) half an hour, change the mx to b.domain.tld, also with a tld of 3 hours. After another half hour, change to c.domain.tld. And so on. You could script the DNS updates to automate it.

      Then each resolver would cache just one of a.domain.tld, b.domain.tld, etc and so (using the numbers originally given would try and contact only one of 5 different MXs. Different resolvers would cache different records depending on the timing of when they last fetched the records. That was definitely doable back then.

      If they had geographically distributed servers then they could also have done some conditional DNS stuff to present different MXs to different area - can be done with BIND using views, but I don't know whether that feature was available then.

  5. Inspector71

    1995/96, using Claris Emailer on my Quadra 700, so exciting.......sigh....

    1. Muscleguy

      '95/'96 we both had JANET email access at work and we had a rented property with a single phone point just inside the door so our Centris 650 was offline. It had to wait for the move up here to Dundee end of '98 and into our own house with a Telewest cable account and a separate phone line to get it online.

      Eudora was our mail app of choice, at work for me as well as at home. Having two email addresses, one at home and one professionally seemed luxurious and somewhat decadent.

  6. Anonymous Coward
    Thumb Up

    Sendmail hacking to the rescue

    I was working as a data network administrator at an Australian university when this happened. Luckily I had written local hacks into sendmail to do a form of exponential back off when emails were unduly delayed. At the time we hosted a number of listervs and other various sundries, as well as 25,000 users doing their normal thing.

    So when the outage hit our mail queue grew to tens of thousands of emails, a fair number of which were to the mailing lists (hi Pavement fans mailing list) enquiring if anyone else hand noticed the outage and asking others to reach out to users who weren't answering; my response, "Are you helping? Good, well stop." Our poor Sparcserver20 reached a load average of 128 but it stayed up, one of only a handful of Aussie university mail servers that didn't bounce at least once during the outage. I know some other university mail admins null routed the email to AOL via DNS lame delegation hacks. All our email to AOL eventually delivered about 48-72 hours later.

    Did I mention I hand wrote our sendmail.cf file?

    1. Muscleguy

      Re: Sendmail hacking to the rescue

      I vaguely remember it now, because of the effect on ListServe. I had a UK JANET account at work back then and other than the effects on email and listsserve we had ringside seats. That big fat, optical JANET pipe was as serious luxury. I could send multi megabite email attachments (science data) and apart from having to confirm that I wanted to do that it would go. Had to be to another JANET account though.

      1. TRT Silver badge

        Re: Sendmail hacking to the rescue

        Ah... I can remember the days when science data used to fit on a SSSD Floppy... and 10MB seemed to be an infinite limit for a mail attachment.

        1. Anonymous Coward
          Anonymous Coward

          Re: Sendmail hacking to the rescue

          "10MB seemed to be an infinite limit for a mail attachment"

          When I was rputting together my first PC, Tiny Computers sold you the box but you had to add a HDD and I remember thinking as I bought a 40MB HDD that despite buying the smallest HDD available I was gettting something so vastly huge (twice the size of the disk on my work PC) that I was never likely to fill!

          (Though going back another 5-10 years I can remember when adding a 4kB RAM *card* - that's really 4096 bytes - to the 6800 processor at school was a huge deal as it meant we could play StarTrek!)

          1. DropBear

            Re: Sendmail hacking to the rescue

            "I bought a 40MB HDD [...] I was never likely to fill"

            Considering floppy disks used to carry 1.44 megs meaning 40 megs held less than thirty disks, and even back then a single floppy held basically not a whole lot of anything, that sounds a tad bit optimistic if said PC was meant for anything beyond Haiku storage.

            1. Anonymous Coward
              Anonymous Coward

              Re: Sendmail hacking to the rescue

              "Considering floppy disks used to carry 1.44 megs meaning 40 megs held less than thirty disks, and even back then a single floppy held basically not a whole lot of anything, that sounds a tad bit optimistic if said PC was meant for anything beyond Haiku storage."

              Actually, small (physically - 5.25 inch) floppies used to hold 160 kb, compared to the 2nd generation full sized (8 inch) floppies which held a massive 250 kb.

              I eventually provided office applications (word processing, spreadsheet, database) for an entire office (one floor of a 20 story office tower) running on a computer with a 5MB hard drive, enabling us to ditch the stand alone word processing machines.

            2. Dog11
              Childcatcher

              Re: Sendmail hacking to the rescue

              Considering floppy disks used to carry 1.44 megs meaning 40 megs held less than thirty disks, and even back then a single floppy held basically not a whole lot of anything, that sounds a tad bit optimistic

              You had 1.44M floppies? You kids didn't know when you had it good. We had 180K floppies, and were grateful for it, didn't have to hump cartons of punchcatds around anymore.

              1. Anonymous Coward
                Anonymous Coward

                Re: Sendmail hacking to the rescue

                You had had 180K floppies? I had 85K ZX Microdrive cartridges. Although sometimes you could get one that would format to 88 or even 89K....

                1. Sargs

                  Re: Sendmail hacking to the rescue

                  Formatting a microdrive cartridge would stretch the tape and increase its potential capacity. If you were a really boring nerd, you could write a program that would repeatedly format the cartridge then measure its capacity, stopping after two formats gave more-or-less the same value. Or the tape snapped.

                  Kids of today- tell 'em you used to literally physically stretch your storage to get a couple of extra kilobytes out of it, they'll ignore you and hope you go away.

            3. jake Silver badge

              Re: Sendmail hacking to the rescue

              1.44 megs? Luxury ...

              Seriously, my first 8" floppys were 256K ... My first 5¼s were 160K. The system ran off one of those disks. In those days, the thought of a 40meg HDD at home was purely in the realm of fantasy. (By way of reference, in mid-1980 an 18 meg NorthStar drive cost in the neighborhood of $4,200 ... in 1980 dollars. About a year and a half later, Apple debuted a 5 meg drive for $3500 ... People lusted after these GIGANTIC storage devices.)

              1. DropBear

                Re: Sendmail hacking to the rescue

                Oh boo-hoo. And punched cards held mere bytes. Yeah, the 8" floppies on our CP/M Z80 boxes held stupid little, but it's irrelevant - by the time buying a HDD for a home computer was a thing that even non-NASA personnel could reasonably do, nobody used anything other than 1.44 floppies; compared to which 40 megs were a luxury, but a vewwy-vewwy modest luxury indeed. Nobody I knew walked around with less than a full box of floppies by then, and when your existing data instantly takes up over a third of your allegedly humongolicious new HDD, starting to longingly ogle one at least four times as big before you even installed this one is what you do, not expecting it to never fill up.

                1. Joe Werner Silver badge

                  Re: Sendmail hacking to the rescue

                  I do beg to differ: our first pc, an IBM .... 80something-or-other had a harddisk. I think something like 20MB? And two 5.25" floppy drives.

                  My first PC with the non-floppy 3.5" floppies had a 250MB hdd. That was... in the early-to-mid 90s

                2. jake Silver badge

                  Re: Sendmail hacking to the rescue

                  By the time buying a HDD for a home computer was a thing that even non-NASA personnel could reasonably do, the IBM XT came with a 10meg HDD and a 5¼ 360K floppy. 1.44s were a number of years in the future. Nobody I knew walked around with a full box of floppies, unless they had just purchased them.

                3. Anonymous Coward
                  Anonymous Coward

                  Re: Sendmail hacking to the rescue

                  "by the time buying a HDD for a home computer was a thing that even non-NASA personnel could reasonably do, nobody used anything other than 1.44 floppies"

                  Nope. My family's second computer had a hard drive and a 5.25" floppy drive. Played many games off of 5.25" floppies. I can't remember if the first one had a HDD; I do recall it had two 5.25" floppy drives.

    2. Anonymous Coward
      Anonymous Coward

      Re: Did I mention I hand wrote our sendmail.cf file?

      I once had a general sysadmin-type inteview where I was asked if I'd ever hand-editted a sendmail.cf. I think it was a trap, but couldn't be sure. However, I had done it, just once - I'd added a line (or commented one out, I forget exactly) to block the open relay that was there in the default, so I gave that reply.

      Mind you, I didn't get the job. Maybe they needed gung-ho and/or expert sendmail hackers? Or was me even thinking about going near a sendmail.cf file with a text editor too crazy for words? I suppose I will never know.

      1. Anonymous Coward
        Anonymous Coward

        Re: Did I mention I hand wrote our sendmail.cf file?

        dnl

        I remember when Allman & co decided that sendmail.cf was too hard and what would really make it easier was getting everyone to write it...in m4. Cos yeah, we all dream in preprocessor languages.

        dnl

    3. Anonymous Coward
      Anonymous Coward

      Re: Sendmail hacking to the rescue

      "Did I mention I hand wrote our sendmail.cf file?"

      You couldn't figure out how to work m4 macros either?

    4. CrazyOldCatMan Silver badge

      Re: Sendmail hacking to the rescue

      Did I mention I hand wrote our sendmail.cf file?

      I take it that now far enough in the past for you to have recovered from the sanity loss?

      (I've written a few in my time. Mostly by copying ones from other people and amending them..)

  7. Mystic Megabyte
    Devil

    Dig

    For those too young to remember, AOL used to bombard us with "free" CDs. The worst mistake that you could make was to put one in your PC. Trying to remove AOL was almost impossible, it wrote itself into multiple registry locations. (possibly hundreds!)

    Future archaeologists will find an entire strata of discarded AOL CDs :(

    1. Anonymous Noel Coward
      Happy

      Re: Dig

      One time, during college, my friends and I drove to a computing goods store, picked up a couple of free AOL cds and took them back to Campus to use as frisbees.

      Good times.

    2. Anonymous Coward
      Anonymous Coward

      Re: Dig

      My friend decorated his bedroom with AOL CD's, shiny side out.

    3. FrogsAndChips Silver badge

      Re: Dig

      A guy had set up a website to collect unwanted AOL CDs and promised to deliver them back to AOL as soon as he'd reached 1 million to protest against the huge waste. As far as I recall, he'd only reached a few 100k by the time every ISP had stopped sending CDs anyway.

      1. phuzz Silver badge

        Re: Dig

        Jason Scott has been trying to digitise every AOL cd. I'm not sure how far he's got, but future historians will thank him. Probably not for this, but they will thank him.

        1. Alan W. Rateliff, II

          Re: Dig

          The floppies are more fun. I come across a couple every so often digging through what-not I have not seen in years (probably should just torch the stuff and be done with it.) Toss 'em into DOSBox just for kicks or for real torture run them in PCTask on the ol' Amiga.

    4. John H Woods Silver badge

      Re: Dig

      "Future archaeologists will find an entire strata of discarded AOL CDs"

      Ah yes, the Eternal September line of 1993, up there with the K-T* and P-T boundaries

      * oops, showing my age: K-Pg for those with hair on their head rather than their chin

      1. W.S.Gosset
        Windows

        Re: Dig

        >"Ah yes, the Eternal September line of 1993, up there with the K-T* and P-T boundaries"

        BIFF!

        Ah yes, check out that glorious Internet Oracle .sig.

        Sind3y! B1ffsm4nia! B1FF R00LZ!!

        I miss the Internet Oracle.

        1. jake Silver badge

          Re: Dig

          That's Usenet Oracle, you heathen!

          After pondering deeply, the Oracle decrees: Citing Wikipedia when there is a perfectly good Jargon File entry is grounds for immediate banishment, only revocable by printing an ASCII art Snoopy on a line printer and hand delivering it to the Harvard Science Center's observatory whilest waving a rubber chicken.

          n.b. Biff was not a barker, that is a baseless, malicious lie!

    5. Tom 7

      Re: Dig

      I still have some that I tie onto strings to hang over my brassicas as bird scarer's. They hate AOL almost as much as me!

    6. Graham Jordan

      Re: Dig

      The downside to no longer getting CD's through the door is now my kid will never know the awesomeness of putting an AOL CD in the microwave and watching that mother fucker light up.

      1. Swarthy
        Happy

        Re: Dig

        Ah, I still remember the smell and awesome patterns they made. We used the nuked discs as coasters.

      2. CrazyOldCatMan Silver badge

        Re: Dig

        awesomeness of putting an AOL CD in the microwave

        It stinks like hell though - and I'm sure that the vapourised plastics are probably really, reaaly bad for you.

        (Doesn't work for modern CD-R's quite as well - they use a dye layer now rather than a metallised film)

    7. Gene Cash Silver badge

      Re: Dig

      "AOL has proven that they have the ability to send a CD or three to every chordate on the planet once a month."

      -- Douglas Henke

    8. Mark 85

      Re: Dig

      Ah, fond memories of those CD's. They made cheap coasters for coffee cups and the like. When had a few boring moments at work, they became Frisbees. One of my friends was an avid target shooter and would use them as targets. Yeah.. I shudder to think of the AOL CD strata layer.

      1. /dev/null

        Re: Dig

        I still have a bag of cover-mount floppies and CDs from 1990s computer mags, mainly because I've never found a good way to recycle them. Alas, I have plenty real coasters and no need for bird-scarers,

    9. Anonymous Coward
      Anonymous Coward

      Re: Dig

      "For those too young to remember, AOL used to bombard us with "free" CDs. The worst mistake that you could make was to put one in your PC. Trying to remove AOL was almost impossible, it wrote itself into multiple registry locations."

      Registry locations? I recall AOL floppies showing up everywhere, and I am reasonably sure they only needed MSDOS...

      Very few people had CDROM drives. I managed a special deal available for Microsoft employees and their families that could net a 2x CDROM drive, complete with the SCSI card, for only $400. No other non-business users I knew had them for years after that.

      1. jake Silver badge

        Re: Dig

        Early AOL floppys came with a "runtime" version of PC/GEOS that could be modified to become bootable. I know a couple of folks who used this as their primary GUI ...

        For some unknown reason, I've been in the habit of burying "time capsules" of miscellaneous industry tat since 1993. One of those archives contains AOL floppys, the next one in the series contains AOL CDs.

    10. CrazyOldCatMan Silver badge

      Re: Dig

      Trying to remove AOL was almost impossible

      One company I worked for we had a real charmer for a marketing director (took pride in the fact that he could make his secretary cry, refused to answer his own emails and generally treated anyone lower-grade than him with utter contempt - especially IT..).

      We had a *very* strict 'thou shalt not install non-work software on thy PC rule' - breahing of which was a disciplinary offence and could lead to dismissal.

      Said director had finally been given a laptop so he 'could work from home'. After a week, he stormed into the Desktop support cube and threw the laptop at us saying "it doesn't work". Eventually, we managed to get out of him that the corporate dialup (which was flaky on a good day) no longer worked.

      Delving into the reasons why, it soon became obvious that he'd install AOL. And his home person finance software. And had an 'interesting' collection of images (this was the late 90's so nothing too amazing but they still drove a chieftan tank through the corporate guidelines).

      The presence of the AOL dialler meant that there was no way whatsoever to get the corporate dialler to work. Even uninstalling it failed so we informed said director that he needed to save his information off the laptop as we were going to have to rebuild it.

      Once we had the laptop back, we nuked it from orbit.. Corporate dialup now worked again.

      Two days later, he was back in our cube screaming that he was going to get us all sacked because we'd deleted all his finance data - turns out that the home finance programme saved all its data in the programme directory (as was common in the late 90s) and that he hadn't bothered to back that up so all his data was gone. He went off to HR while I went to have a chat with the local site director (a really nice guy who had had this marketing director foisted on him by headquarters but nevertheless outranked him).

      Site director apparently tore very large lumps out of Marketing director and told him to amend his ways or he would get relocated to the smallest, most rural backwoods US location that could be found and left there to rot (company policy was that directors *never* got sacked - even if the site they worked at got closed then they just got found an essentially-meaningless job elsewhere).

      He never spoke to us again - any interaction was via his secretary (who also benefitted from his enforced change in attitude).

  8. John Crisp

    I remember.... and feeling vaguely smug that I had decided to get a Demon account instead of AOL.

    1. Paul Chambers

      .....the paucity of early ISP choice...

      JANET, Compuserv, Demon, then force9 (later to become plusnet), for me. I have since BT'd, but I'm better now thanks (A&A if someone else is paying, but plusnet still used, and yes I know they are BT owned...but they are not BT. You can ring them, for a start. With A&A, or more specifically AK, I installed an early-ish consumer level voip system - network alchemy, I think - back in the days when the Rev came out to install the equipment on the cabling I had pre-installed. I had a nokia 9000 and the time and AK had just got a brand new 9110 - so it would be 1998). I was gutted when Demon went under. I first hosted my own website in the Demon days..and it's still running now...but hosted under very different circumstances.

      Aol was always one to be avoided. The volume, and quality, of the advertising was the clue.

  9. Anonymous Coward
    Anonymous Coward

    Hmm

    I seem to recall that would be about the time my organisation rolled out Lotus CCMail after which I pointed out it cached your User and password in plain text in the .INI file (which didn't go down well, apparently messengers are still not liked...)

    Was also about the time I got my first 18.8baud modem and screeched onto the internet with CompuServe along with hideous numeric email address.

    In more recent times I was involved in one of the many early internet bank launches. You will remember these as several crashed at launch due to the number of reporters wanting to report how they would crash at launch...

    this case though I identified that one of the key web handling programs was a flagrant copy of he reference code in the OS vendors manual, and guess what, it didn't work and didn't release resources on exit. Therefore repeatedly crashing.

    Turn up on customer site miles away from home after being called to help, getting late, just told buy some new pants and shirts on expenses, you are not going home until it's fixed... Wasn't even my code!

    1. Anonymous Coward
      Anonymous Coward

      Re: Hmm

      "[...] screeched onto the internet with CompuServe along with hideous numeric email address."

      When our company gave us access to the internet all our email addresses were derivatives of the company's X400 format address - which were neither short nor memorable.

      In spite of the company having a short brand name - it was several years before we got fred.bloggs@zzz.com

      1. Chris King

        Re: Hmm

        I worked at one of the last JANET X25 sites before the service closed in 1997 - the internal network was TCP/IP but one Unix host had Coloured Book Protocol support for JANET access to the Uni over the road.

        Getting mail from SMTP (internal) to X25 (JANET) to X400 hosts was... interesting.

        Sometimes, someone even managed to get a reply to me.

        Just not very often.

        (Hmm, maybe El Reg needs a "What did YOU do in the War, Grandad ?" icon...)

        1. Anonymous Coward
          Anonymous Coward

          Re: Hmm

          "Getting mail from SMTP (internal) to X25 (JANET) to X400 hosts was... interesting."

          SMTP may have taken some of the fun out of email. I think bang paths had character. I wonder if uucp will make a comeback for interplanetary email transfer?

    2. Alister

      Re: Hmm

      and screeched onto the internet with CompuServe along with hideous numeric email address.

      Strangely, I came across my old Compuserve address only the other day, it was 101360.1534@compuserve.com

      It's dead now, of course.

  10. wolfetone Silver badge

    "Bert told us that his errors led to hate mail from noted distributors of unsolicited commercial email"

    Shows how much conviction he had in AOL for emails, he obviously didn't use the service himself.

    1. Prst. V.Jeltz Silver badge

      hows that obvious?

      especially in light of the next paragraph:

      "He assumes he got plenty of angry emails to but was blissfully aware of just how many people hated him – because their emails couldn’t get through!"

  11. Anonymous South African Coward Bronze badge

    This perked me up. I needed to hear somebody else's biggentale of biggenwoe :)

    Ahhhh, the trials and tribulations of being a sysadmin...

  12. MarkelHill

    Similar but different at Yahoo

    On or about 2002 a similar failure occurred at Yahoo when a new MX was added that inadvertently exceeded the max UDP packet size and thus caused all DNS queries to fall back to TCP.

    The sudden influx of huge numbers of TCP connections swamped the DNS servers such that our update mechanisms failed when we attempted a reversion. We couldn't even ssh into them to make a manual fix. Furthermore the servers also stopped responding to UDP queries!

    We had to get our data-centre folk to disconnect each server from the network so they could log in at the console and apply a manual fix to unwind the mess.

    While no mail should have been lost it caused queries to *.yahoo.com to timeout for a number of hours so presumably many millions of dollars of lost ad revenue during that time.

    Unsurprisingly the instigator of the change lost their DNS update privileges but they did keep their job and went on to become a VP.

    1. Robert Carnegie Silver badge

      Re: Similar but different at Yahoo

      Seems unfair that a reasonably unexpected failure of a routine update means that you're disqualified from maintaining that system in future. On the other hand, you probably would want someone else to do it in future, because you've used up your "get out of cockup free" card there.

    2. Anonymous Coward
      Anonymous Coward

      Re: Similar but different at Yahoo

      "went on to become a VP."

      Al Gore?

      1. FrankAlphaXII

        Re: Similar but different at Yahoo

        I was thinking Marissa Meyer, but she eventually made CEO

    3. Anonymous Coward
      Anonymous Coward

      Re: Similar but different at Yahoo

      Ah the old "promote him so he can't do any more damage" trick.

  13. Joe Harrison

    Compuserve

    I was summoned to the top people one day and told that henceforth I would be working on something called Novell NetWare. As the official support channel from Novell was via Compuserve I was given an account and a dialup modem.

    I found it amazingly useful and you could do all sorts of things such as book flights electronically which was unheard of back then. Unfortunately the bosses forgot to tell me it cost something like 25 dollars an hour to use, with predictable results :(

    Afterwards I found that if you waited until 7PM it "only" cost 7 dollars so I used to login at 7:01 and typed my fingers off trying to cram everything in.

  14. Anonymous Coward
    Anonymous Coward

    In hindsight

    I bet he wishes he'd taken AOL email down not for 19 hours, but for 19 years.

    1. Anonymous South African Coward Bronze badge

      Re: In hindsight

      hehehehe :)

  15. StripeyMiata

    BlackBerry 2011

    >It's hard to top Bert's story for sheer scale

    Only one I can think of is the BlackBerry outage of 2011 which I believe was caused in Reading. So someone here might know about it.

    1. Anonymous Coward
      Anonymous Coward

      Re: BlackBerry 2011

      Reminds me of The Great Tea Trolley Disaster of '67.

  16. Anonymous Coward
    Anonymous Coward

    I know of a rumor, a person who shall remain unnamed,

    broke up with their girlfriend and decided to DDOS her.

    He and a friend bought a botnet and proceeded to DDOS her..

    She was in Ghana and their DDOS took the whole of Ghana off the internet...

    Whoops!!!!

    1. Steven Raith

      Heard similar...

      I was told a similar story by an ex network-op of mine - as I recall (and feel free to correct me if I'm wrong) but it was something silly like Ghana (or at least, the state I was told about) having a very small netblock allocation as a country, or only having a single gateway in, or the whole lot being proxied through one server, or other similar 'brick wall' hard limit connectivity problem that up until then, hadn't been a problem.

      Then 'something' occured that made their IP range get flagged up, the whole internet went down for the country.

      I think the story I was told was about a Gulf state (Jordan? Qatar?) but it's apocryphal anyway, so it might be the same tale.

      Anyway, reminiscence over, back to work, sigh.

      Steven R

      1. Anonymous Coward
        Anonymous Coward

        Re: Heard similar...

        'Something' occured that made their IP range get flagged up, the whole internet went down for the country

        This sort of thing used to occasionally happen for the NHS. You'd end up having to answer captchas to use Google, and get snotty "suspicious usage pattern" messages from other websites.

        I haven't seen it in a while, so maybe the proxies have been properly sorted out now. Who am I kidding.

  17. albegadeep

    AOL memories

    Well do I remember AOL. We had it back in the '90s. Used a 2400 bps modem to connect, and everything took FOREVER. My parents finally got fed up and bought a 28.8 kbps modem - and got no increase in speed. We rapidly switched to a local ISP, which was WAY faster.

    1. Boothy

      Re: AOL memories

      My first modem was also 2400 bps, it was a 'loan' modem from where I worked as they'd just bought a 28.8kbs one.

      I just used it for BBSs etc at the time. I remember one local computer shop (PCs, Amigas, Atari STs etc) had things like stock-lists on a dial-up BBS system. You could even pre-order items via the BBS, and then go pick them up. (No payment system etc back then). They even gave you a free Compuserve email address (wow! ;-) ).

      Once I got proper Internet access (Demon), I went and bought a USRobotics external 56k modem. Was in an aluminium case, almost exactly the same size as an external 3.5" floppy drive. This was all on an Amiga, didn't get round to switching to PC till the late 90s!

      1. Stoneshop

        Re: AOL memories

        My first modem was also 2400 bps, it was a 'loan' modem from where I worked as they'd just bought a 28.8kbs one.

        Ohh, lookshoorey. My first modem was banging rocks togethera bit of veroboard with a NE565, a NE567, a CD4066 and an opamp, plus a bunch of resistors, capacitors, switches and some trimmers, lovingly soldered into a device that could whistle 300 bits per second down a phone line, and listen to them coming the other way too. You needed to dial the number yourself on a Real Telephone. It's still somewhere in a box. Then a Racal-Decca 1200 baud modem, which too required External Dial Assist. A 9600 baud one, previously used for Remote Diagnostics at DEC FS, and finally a 28k8 V32.something before ISDN arrived with its whopping 128k (and still 64k when you got a call).

      2. CrazyOldCatMan Silver badge

        Re: AOL memories

        My first modem was also 2400 bps

        Pah. The youth of today. My first modem was a 1200/75 non-autodial (about the size - and weight - of two housebricks!).

        I well remember embiggening my parents phone bill by dialling to Almac BBS in Scotland (we lived in London). There was some... discussion about the on-line time and my parents took to occasionally lifting their phone extension in order to knock me offline..

    2. Anonymous Coward
      Anonymous Coward

      Re: AOL memories

      Ahh, yes, AOL Tymnet nodes. Capped out a 4800 baud in most cases. Oh, and then they eventually told us all to go away. Not sure if that was an AOL decision or if Tymnet was folding, or what.

    3. Anonymous Coward
      Anonymous Coward

      Re: AOL memories

      "Well do I remember AOL. We had it back in the '90s. Used a 2400 bps modem to connect, and everything took FOREVER. My parents finally got fed up and bought a 28.8 kbps modem - and got no increase in speed. We rapidly switched to a local ISP, which was WAY faster."

      Don't be silly. Things got very fast when I traded in my fast (300 bps) modem for a 1200 bps hot rod.

    4. FuzzyWuzzys
      Happy

      Re: AOL memories

      My first was a 600bps modem I knicked out of skip from a warehouse that was closing down, sometime around 1989. The modem had big rubber acoustic couplers and my Dad had to "borrow" a suitable phone from work that would fit the coupler pads! Then this thing called the "Internet" appeared around 1991, my mate at Uni said it was the best thing ever and...it was utter crap! Lots of boring pages through this thing called a browser, tons of very boring text only pages and you needed Windows. No pirated software, no message boards worth bothering with and I went back to BBSs for a anther couple of years as you could use BBS with software that ran on DOS without needing a whopping 4MB of memory to run Windows.

    5. swm

      Re: AOL memories

      I loved the AOL floppy disks they distributed.I would format them and use them at conferences etc. to give people files. They would look at the disks and laugh.

      The CDs were pretty much useless though.

  18. anthonyhegedus Silver badge

    I had an RSVP with my email address on my wedding invite in 1996

    I thought it would be really cool to include a wedding address on my wedding invitations sent out in 1996. Nobody RSVPd by email of course because none of our friends had an email address at the time.

    I still thought it was cool.

    It wasn't AOL either!

    1. Anonymous Coward
      Anonymous Coward

      Re: I had an RSVP with my email address on my wedding invite in 1996

      My father-in-law, in the early '80s, sent an email to everyone on the 'net at the time, regarding the completion of a 9-month-long batch job. Included a few details such as height and weight, and ended it with "IT'S A GIRL!" - it was my wife's birth announcement. Came across the printout recently. I may frame it.

  19. theblackhand
    Coat

    Ethernet ports

    "There was no way we could have crammed all that traffic through a tiny 10Mbps Ethernet port." So he didn’t."

    If he'd just used a regular sized Ethernet port (at least assuming it was UTP...) rather than some non-standard tiny one, he could be running 10Gbps by now...

    I'll get my coat.

  20. Valerion

    Compuserve

    This happened only a few days before I started at Compuserve. The people running the introduction training course made a very big deal out of it!

  21. Anonymous Coward
    Anonymous Coward

    I managed to follow a change management ticket that was wrong when working for Amzn and managed to get the entire Milton Keynes distribution centre staff an afternoon off ;-)

    1. Korev Silver badge
      Pint

      I hope all the Amazonians were grateful and you were properly rewarded -->

  22. a_yank_lurker

    AOL = America OffLine

    When we first got on the Internet we were told by the computer store staff to avoid America OffLine and go with a true ISP which we did.

  23. Anonymous Coward
    Anonymous Coward

    My Hero

    We all admire a Go Big or Go Home type of guy. Who will ever be able to top this (after a suitable period of time has passed)?

    1. Korev Silver badge
      1. Anonymous Coward
        Anonymous Coward

        Re: My Hero

        And that didn't take long. What a great story.

  24. Frozit

    TBH

    This just sounds like they were on the bleeding edge of email systems, and something was going to die, somewhere.

    The fact that they couldn't get a load balancer strong enough to handle their volume tells you something was going to give.

    An honest attempt was made to address the issues, and it failed. Meh.

  25. Anonymous Coward
    Anonymous Coward

    AOl. training wheels for the net. Around a decade ago the decide to let people use their system for free hoping people would pay. They have gone back to the paid walled garden model. Who in thier right mind would pay for AOL ?

    1. Mark 85

      I still have my AOL email account as having friends and family scattered about the world it would just be a PITA to change. Still "free" although the spam filters suck. I'll add that a few tech friends from back in the day still have theirs' for similar reasons.

  26. Anonymous Coward
    Anonymous Coward

    when I was a sysadmin at PSINet, we'd inherited some UUCP servers from the EUNet GB days. Despite it being a popular service with a big reseller, we didn't want to continue it, so we used Y2K as an excuse to shut it down.

    Now, I'm kind of sad we did.

    Posting anonymously because there are probably people still angry about that!

    1. Anonymous Coward
  27. Anonymous Coward
    Anonymous Coward

    I once took out a live, Tier 1 Telco system

    I did it with an SQL query. I’d been doing lots of fairly long (to run) queries without issue, and forgot busy Hour was actually going to be busy, that day (customer had just migrated a load of their subscribers but still, we weren’t expecting it to be so busy). The whole situation went like this:

    1. Loyal AC starts the query running.

    2. And pops out to the local Chinese chippy, for some chips and curry.

    3. And arrives back to find the company’s directors (all four of the trilogy) in the server room at 5:30PM.

    Turns out one of our processes wasn’t resilient to a rather important table being locked too long.

    AC - that one doesn’t go my CV.

    -

  28. J.G.Harston Silver badge

    Somewhere I've still got a sweater with my university departmental email address on it: info@uk.ac.stir.cs

  29. Tim99 Silver badge
    Windows

    Grumble grumble

    When I started using electronic mail in the 1970s we knew our recipients personally - If the message was important, we used to telephone them to see if they had received it; or to ask them to log in and read it. I only had to make a *couple* of transatlantic calls...

  30. Anonymous South African Coward Bronze badge

    Back in that time, email in South Africa was only available to a select few - and we had to dial up to CompuServe (ick) or our local BBS in order to retrieve our email.

    Fun times.

    Then it started to gather momentum, and more people got their@email.com addresses - mine was ****@mweb.co.za for a long while, but when they got too uppity with their pricing, I went over to google.com

    And today I'm contemplating the move away from google.com to something else where my emails won't be perused by bots in order to serve advertisements targeted at myself.

  31. sisk

    I remember that day and it's all Bert's fault! Well....no, really I don't remember that day. And given what I was using email for back then, I'm fairly certain that no negative impact would have found its way into my life. I might have missed a turn of two of my play-by-email D&D game (yes, play by email D&D in 1996. Have I mentioned I was am a massive nerd?) but that would have been it.

  32. Criggie

    No love for mail via fidonet?

    I was 3:770/105.369 but that's decades dead. Ued a 1200 baud Netcomm modem which was powered by a 9V battery. That got expensive quick, till I saved up for a PSU. Still got the modem too.

  33. W.S.Gosset
    Coffee/keyboard

    Jamming ISP mail servers

    I once did this when outraged by receiving SEVERAL UNWANTED SPAM MAILS at work. 1996 or early 97. So banged out a quick oneliner in shell to send BIGNUM emails (I seem to recall 10 million) back to him saying "Stop spamming me!" or somesuch. Then got on with my job, fierce with the pride of the righteously revenged.

    Got a half-aggro half-WTF phonecall around close-of-business from our main sysadmin in Mountain View, CA. Our London boxes all connected to head office there, and thence via their ISP to the world.

    Apparently I brought down the biggest silicon valley ISP's mail server. After apparently a lot of drama recovering, they'd given our man a serious shouty earful and a half for hosting a spammer in-house, and threatened to drop the company. :) Oops. All were soothed once I explained, albeit with some stern finger-wagging.

    15ish years later the boot was on the other foot. Doing some part-time sysad work at a little ISP/hostedservices shop, and a BadActor cracked a client's website and kicked off a massive spam-spray (a real one this time). The queue was BIGBIGNUM and had filled the queue volume, all in the space of a few minutes. Too many files for "rm *" to work and the main man was having a major stress attack because he couldn't see a way out that didn't involve dropping services for the duration. Quick little shellscript along lines of "while true;do; DELFILES=`ls PATTERN | head 200`; rm -f $DELFILES;done " then sit back and wait for the race condition to resolve in our favour.

    1. jake Silver badge

      Re: Jamming ISP mail servers

      "Apparently I brought down the biggest silicon valley ISP's mail server."

      That would have been Netcom.The only major crash I can remember Netcom having back then was a misconfiguration in the BGP code, pushed out to all border routers. Seems to me somebody fat-fingered a stray "&" where it shouldn't ought to have been ... But that was perpetrated by MAE-East, not San Jose. They also had issues with majordomo and bounce floods/loops in that time frame, but not enough to crash the system. Mail was already fairly robust by 1996 ...

      1. W.S.Gosset

        Re: Jamming ISP mail servers

        No, it was a SV ISP, not a national. Remember, siliconvallistas are very parochial.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like