back to article Snapchat coding error nearly destroys all of time for the internet

The current release of the Snapchat app on iOS contains a coding error which is flooding the internet's Network Time Protocol (NTP) pool. NTP is one of the oldest internet protocols, and has for decades been used to synchronise computers to within milliseconds. The NTP pool is a network of volunteers' servers which …

  1. Andrew Moore

    Timely...

    There's a leap second being added on New Years Eve...

  2. Paul Crawford Silver badge

    WTF is an App doing quering network time?

    Really, what is the Snapchat app doing? Timekeeping is an OS-level task, and only that should be syncing the server/PC/phone/telidildonic dildo/etc to real-time, and user level programs can get their time from the OS by whatever means the OS supports.

    1. Dan 55 Silver badge

      Re: WTF is an App doing quering network time?

      I bet it's some bobbins hand-rolled protection (e.g. you can't set the OS clock forward or back to read messages you shouldn't or something), in which case it would be more of a service for everyone if they could set up their own time server and spam that.

    2. DNTP

      Re: WTF is an App doing quering network time?

      Laziness, outsourcing, or both.

      Seriously, though, what if a user deliberately set their device to an incorrect time zone, thus unlawfully interfering with an advertiser's monetization of the user's metadata? Clearly the App Developer has to be On Guard against this kind of Economic Terrorism by implementing their own badly coded resource abusing snitchy permission snaffling spyware network time query assessment.

      1. The Man Who Fell To Earth Silver badge
        FAIL

        Re: WTF is an App doing quering network time?

        Screw the apps. The cell phone provider usually wants the phone's time synced to them.

        Again, this just shows what crappy programmers Snapchat used to write their rubbishy app. Apple should black list them. Google Play as well.

        1. ThomH

          Re: WTF is an App doing quering network time?

          In a former life, working on an app that displayed time-dependent data, we found that some very negligible quantity of people had devices with the wrong time, most likely because they have their iPad, iPod Touch or iPhone without an international SIM, then when they land they adjust the time to wherever they are, not realising that it's much easier just to adjust the time zone. They've had to disable automatic time setting to get to that option. They end up with a device that says the same time on it as the clock on the wall so you try telling them they've done it incorrectly.

          I guess somebody at SnapChat decided they don't trust users not to have disabled the built-in OS time synchronisation.

    3. Ken Hagan Gold badge

      Re: WTF is an App doing quering network time?

      Regardless of what Snapchat *thought* they were doing, the fallout from this would indicate that Apple's/Google's response should be to block out-going NTP traffic from every process except the one that the OS uses to set the time for the device.

    4. Anonymous Coward
      Anonymous Coward

      Re: WTF is an App doing quering network time?

      > Really, what is the Snapchat app doing?

      Well, you could have followed the links provided in the article and would have found the answer from the horse's mouth: "The Snapchat mobile app does not currently have a need to talk with NTP

      servers, it was an error."

      Nice to see that they made up for the trouble by adding to the pool in the South Hemisphere.

  3. Anonymous Coward
    Alien

    Could this explain why

    My bank account keeps showing deposits from my wifes account (with a different bank) two days before she actually makes them??

    NOT a joke.

    1. DNTP

      Re: Could this explain why

      Is your wife perhaps an astronaut, or involved in any other occupation which involves travel at relativistic velocities? Have you ever seen her leave for work with a futuristic spandex body suit under her street clothes? Are there closets in your house that seem mysteriously larger on the inside?

      (My GF and I used to do time traveler roleplaying involving lots of spandex, we never accessorized the experience with 'paychecks from the future', she took the spandex when we broke up.)

      1. keithpeter Silver badge
        Coat

        Re: Could this explain why

        "(My GF and I used to do time traveler roleplaying involving lots of spandex, we never accessorized the experience with 'paychecks from the future', she took the spandex when we broke up.)"

        @DNTP and all

        My time travellers wear frock coats and tartan waistcoats. Serviceable in most centuries for chaps we find.

        Coat: mine's the one with the penny whistle in the pocket.

        ntp: almost worth setting up a cheap server somewhere. How hard is it to run one of these?

        1. ElReg!comments!Pierre

          Re: Could this explain why

          > How hard is it to run one of these?

          Not at all. Easier than FTP for example, easier than any but the most basic static webserver too. Basically install ntpd, configure it with 4-7 local time servers (if you can edit a text file, easy peasy ; the hardest part may be to look up 4-7 local time servers, but the NTP pool page has a nice list of these on the setup help page), and of course declare your server to the pool so that it can actually be used by others.

          You do need a static adress, or at least one that doesn't change more than once a year, and an always-on server, but that's not really a concern for most Reg readers I would think. The pool website does say that layer is unimportant (even layer 4 servers can join the fun!) but I won't be doing that for mine, for personal preferences.

          Optionally you may want to redirect the port 80 traffic (web traffic) to the main website to redirect misled visitors, but I will be having a local page instead (with a link to the main website, but also a photo of the server an fun facts about the raspberry Pi).

          Apparently the traffic you can expect is barely above the noise from hole-pokers and webcrawlers that keep hitting on anything net-connected these days.

          1. ElReg!comments!Pierre

            layers, strata (Re: Could this explain why)

            By "layer" I obviously meant "stratum", i.e. network distance from a physical time-tracking device. Just now realised that the "layer" term could cause confusion with the OSI layers.

        2. ashdav
          Happy

          Re: Could this explain why

          "mine's the one with the penny whistle in the pocket."

          I'll see your penny whistle and raise you a sonic screwdriver.

        3. Anonymous Coward
          Anonymous Coward

          Re: Could this explain why

          It's easy to setup a local NTP server, and if you run a bunch of clock-less hardware, like my many RPi servers, you should do it without question. Unless you are doing some hard-science projects and require extremely precise time updates, like your own stratum 1 source, etc. Just pick a server that is always going to be on, I use my Raspbian NFS server for that. It serves up files to the Kodi players, and it also serves up NTP to any and all other devices on the local network. Check out your /etc/ntp.conf man page for the setup details. My local NTP server has all the normal pool of NTP servers listed, and is set to broadcast to my local network. All other devices get pointed to this server's local IP only, and viola; you can have as many local devices up asking for time, and they all avoid going out and bothering the volunteer servers. You can check the progress of NTP using the handy command 'ntpq -p localhost' which tells you what servers you are communicating with, and other NTP statistics. Be kind, rewind, and don't have extra traffic, or waste free services, when you don't really need to.

        4. Francis Boyle Silver badge

          Re: Could this explain why

          "My time travellers wear frock coats and tartan waistcoats."

          Quite right too. On the other hand there's this.

          (Yes, I'm dating myself but some things are never forgotten.)

          1. Sam Liddicott

            Re: Could this explain why

            If you want to date yourself you need time travel.

            Or a mirror.

      2. Stevie

        Re: Could this explain why

        Low Earth Orbital velocity, which can be barely achieved with the current state of the art I might add owing to all the so-called scientists renaming stuff instead of getting proper rocket science done, hardly qualifies as "relativistic".

        Just because going up and down the well futzes up yer digital watch doesn't imply a meaningful red shift in the rear-view mirror.

    2. Known Hero

      Re: Could this explain why

      Could it be a direct debit you set up, and the bank is sending it early so it clears on the date requested for some weird reason ?

    3. Anonymous Coward
      Thumb Up

      Don't complain

      You earn more interest this way, so long as the debits aren't also backdated two days!

  4. ElReg!comments!Pierre

    Oh well, I s'pose I could thow a RasPi zero at it... not IPv6 unfortunately, as my home ISP doesn't support it and running it at work would require no end of administrative paperwork, probably for most of 2017, if it is even approved at all.

    1. Anonymous Coward
      Anonymous Coward

      You can have both IPv4 and IPv6, even if your ISP says they don't support it yet. They are just saying they can't hand your hosts directly connected IPv6 address themselves. You can turn on IPv6 via your host or your router configuration, if you have access to it and it has the support built in. Most do. Anyway, the point being that your host, and also independently your router, can encapsulate the IPv6 packets inside of IPv4 and most modern networks know how to unpack and forward them along, no need to bother waiting for your ISP to offer IPv6 service to your home. You can just turn it on and with minimal configuration have it working. Check out: https://en.wikipedia.org/wiki/IPv6_transition_mechanism for the many methods being used to let IPv6 traffic work in a mostly IPv4 world, without having to home-run an all IPv6 infrastructure to your doorstep.

  5. Anonymous Coward
    Anonymous Coward

    "...multiple Snapchat installs on iPhones..."

    hmmmn, sneaky fuckers with fake profiles.

  6. Frank Bitterlich
    Happy

    I have a suggestion...

    "We are also open to any suggestions on how we can help with the present traffic."

    How about Snapchat pays for the damage by contributing a few NTP servers to the "perennially under-resourced pool"?

    1. Brian Miller

      Re: I have a suggestion...

      D-Link settles dispute with 'time geek'

      Yes, it's good corporate netizenship to provide more NTP servers.

      1. Len
        Thumb Up

        Re: I have a suggestion...

        That same "time geek", Poul-Henning Kamp, is also the driving force behind the replacement for NTPD called ntimed. If you're into time keeping and related things (securing bread and butter ancient internet protocols for instance) this is a very interesting presentation:

        https://ma.ttias.be/ntimed-ntpd-replacement/ (review by an attendee)

        http://phk.freebsd.dk/_downloads/FOSDEM_2015.pdf (the full presentation)

    2. DarkOrb

      Re: I have a suggestion...

      They were standing servers up the other day to help, so your assumptions are incorrect.

      The speed they can get added is limited by the pools health checking functions.

    3. Anonymous Coward
      Anonymous Coward

      Re: I have a suggestion...

      > How about Snapchat pays for the damage by contributing a few NTP servers

      They already have.

  7. Lee D Silver badge

    Already run two NTP pool servers, both IPv4 and IPv6.

    What does The Reg do?

    Oh, sorry, we can't even manage IPv6 or SSL yet. It's a bit much expecting an IT site to give back past running a Facebook page.

  8. Anonymous Coward
    Anonymous Coward

    Hey look at me, at me, at me

    That's anti-Social Media A to Z.

    Life was better before the likes of Facebook, Twitter and the rest came on the scene.

    1. Anonymous Coward
      Anonymous Coward

      Re: Hey look at me, at me, at me

      Well take solace in the fact the same people who think everyone wants to see their lunch and a pic of the dump of it after expelling are the same people that actually think the sharing and gig economy are not ripping them off.

  9. allthecoolshortnamesweretaken

    "Oh, is that the time?"

    "No, time is an abstract concept. This is a watch." (Mike, The Cool Person)

    1. Anonymous Coward
      Boffin

      "Oh, is that the time?"

      "No, time is an abstract concept. This is a watch." (Mike, The Cool Person)

      Mike is not up with the latest ideas in physics, in which "time" doesn't exist, it is a human construct which derives from observing natural processes. A watch is just a mechanical amplification of a natural process. So yes, for someone observing it, it is "the time".

      1. Doctor Syntax Silver badge

        "Mike is not up with the latest ideas in physics"

        Yes, but Mike is being The Cool Person. He probably hasn't time to do both.

      2. Anonymous Coward
        Anonymous Coward

        "Mike is not up with the latest ideas in physics, in which "time" doesn't exist, it is a human construct which derives from observing natural processes. A watch is just a mechanical amplification of a natural process. So yes, for someone observing it, it is "the time"."

        Time is an emergent property not an entity in it's own right - Space-Time is a useful construct, but does not match with the physics at the sub-atomic scale.

        And anyway - if time isn't real, how come I remember yesterday and not tomorrow.

    2. Velv

      Time is an illusion. Lunchtime doubly so

      (Douglas Adams)

      1. Fatman
        Joke

        RE: Lunchtime doubly so

        At this (in)famous employer called WROK PALCE Inc., the Boss slave driver in charge named Jim screams at his employees (fondly referred to as 'lakkeys') that he doesn't """pay""" them to eat lunch, and that they should get their slacking asses back to WROK. Guess who wants to put a few pounds of Semtex in HIS Christmas stocking???

  10. ElReg!comments!Pierre

    pool.ntp.org

    With a project of this nature, and a rather plain-looking websit (as should be, content over form etc), surely the registering page would be using old-school, tried and true, "barebone" forms, and not some stupid JS, Shirley. If only to allow wanabee volunteers to register their server from the server itself over ssh, as part of the setup process. Well, guess again...

    "Javascript is required to login."

    Yeah right. How do you you spell "right after I see Stan buying cross-country skis" in JS?

    1. ashdav

      Re: pool.ntp.org

      Javascript is built into your browser.

      What's wrong with that?

      Are you confusing it with Java?

      1. patrickstar

        Re: pool.ntp.org

        Obviously his browser doesn't run Javascript since he sees that message and is unable to login, so, no.

        Either he's running a browser without Javascript (those exist) or he has disabled it (perfectly good reasons to). Next question?

        Since when is it acceptable that basic functionality on web sites, that could trivially be implemented server-side and/or fall back to non-web-development-hipster mode, requires Javascript?

        1. Lee D Silver badge

          Re: pool.ntp.org

          Since the very first days of Javascript.

          Over 99% of web users have Javascript enabled. It's not unreasonable to assume it's present. And, in and of itself, it's no more a security risk than parsing HTML. It might suck up CPU cycles or move things around that you don't want to but good luck browsing ANYWHERE with Javascript turned off entirely.

          Seriously, I have click-to-run plugins and all kinds of things to improve my browsing experience but not for one second have I ever thought I could get away with disabling Javascript entirely.

          And do you not have an easy way to whitelist it for one domain? Sounds like a crap browser to me.

          1. This post has been deleted by its author

            1. Anonymous Coward
              Anonymous Coward

              Re: pool.ntp.org

              Also I am not saying web developers shouldn't use JS. Just for the majority of sites they should be able to build them such that the web site degrades gracefully if the client doesn't or is choosing not to support Javascript or HTML5.

              1. Anonymous Coward
                Anonymous Coward

                Re: pool.ntp.org

                > Also I am not saying web developers shouldn't use JS. Just for the majority of sites they should be able to build them such that the web site degrades gracefully if the client doesn't or is choosing not to support Javascript or HTML5.

                Nice in theory, but the cost of doing that skyrockets and so does the complexity for all but the most trivial sites (many of which already do run just fine without JS).

                With increased complexity, affecting both the client and server ends of the service, comes increased attack surface and therefore increased vulnerability, resulting in a net *decrease* in security across the user population.

                It is also likely that there will be usability and cost penalties due to the loss of XHR-type requests (increased lag and bandwidth).

                Disabling JS for security reasons was sensible advice ten years ago, but is rather questionable nowadays.

                Performance-wise, it's a closer call, especially with single-process browsers such as Firefox (which I'm using now), and I suppose it comes down to which sites one tends to visit. Still, as has been pointed out, selectively enabling / disabling may be an option.

                The last major objection concerns accessibility. Sadly I am not up to speed on the current accessibility scene so I cannot comment on that. Perhaps someone familiar with the subject can chip in.

          2. Anonymous Coward
            Anonymous Coward

            Re: pool.ntp.org

            >Over 99% of web users have Javascript enabled. It's not unreasonable to assume it's present. And, in and of itself, it's no more a security risk than parsing HTML

            Posted long rant but here is the TL;DR version

            1. It should be possible to browse the web much easier with a fast lightweight browser like Dillo that has far less dependencies than FF or Chrome (attack surface matters especially when OpenBSD brings in half the gnome desktop as dependencies for FF, and Chromium also brings in a ton and also requires the more dangerous wxallowed mount option on /usr/local).

            2. Javascript is a bigger security risk than simple HTML especially if we are not also including HTML5. If nothing else your browser has to support both so again this an attack surface increase. Also look at FF security vulnerabilities to see how many are JS or HTML 5 related. Quite a few.

            3. Security and privacy matter a lot to some people and JS on the web is not great for either. At best a compromise can be reached where some JS is allowed such as with Noscript and Tor Browser. I am saying a compromise shouldn't be necessary for the majority of especially information only sites.

          3. ElReg!comments!Pierre

            Re: pool.ntp.org

            > Over 99% of web users have Javascript enabled

            General idiocy notwithstanding, we're talking infrastructure here, not end user. All the machines involved are headless servers.

            1. Anonymous Coward
              Anonymous Coward

              Re: pool.ntp.org

              > General idiocy notwithstanding

              Please do not be rude.

              > we're talking infrastructure here, not end user. All the machines involved are headless servers.

              Out of curiosity, are those HTTPS services for which you do not have the private keys? Else, is something along the lines of "ssh -CL8000:localhost:80" not an option? In my experience, I tend to find that faster and more convenient than running the browser on the remote side, especially on high latency channels (e.g., satellite).

          4. patrickstar

            Re: pool.ntp.org

            Unlike HTML, Javascript is an actual Turing complete language. Parsing it isn't the hard or risky part.

            And even for bugs that aren't in the actual JS engine, exploitation generally becomes much quicker and/or more likely when you have the ability to actually run code that controls it. Nowadays with near-universal adoption of DEP and ASLR this has become significantly more relevant than it was in the days where lots of bugs meant you could just fire off a payload blind based on the build/OS and reasonably expect your shellcode to execute.

            Think of a bug somewhere (there are lots of examples in all modern browsers) which, through some magic ritual, allows arbitrary memory read/write. Without JS, this would require

            a) the ability to have the browser somehow calling back with the data read, and

            b) the ability to have the browser somehow calling back to get "instructions" on what to read and write next based on the outcome of a)

            This will typically be a multi-step process unless you get really lucky with the data that can be read initially.

            With JS enabled, this is frequently trivial. Without JS enabled, this is frequently impossible. Even for the attacker best-case scenario. Even when it isn't, it's gonna be a lot more time-consuming and noisy.

            Back in the days before lots of complex browser APIs/JS engines, client side exploitation of memory corruption bugs was rarely considered practical - single shot, limited control of things like memory layout, etc. Instead server side stuff was the usual target - often multi-shot (forking/respawning services), often possible to control memory layout well (even though it could require a large amount of data transferred over the network, like the TESO BSD telnetd exploit) and generally interact in a multi-step process to leverage things like memory/information leaks.

            Now we have DEP/ASLR and client side exploitation, which is arguablly a lot harder to protect against, is suddenly both practical and widely done, while server side stuff where you generally can't fire off Turing complete instructions and have it follow them, is considered hard.

            The former isn't just because the exploit writer kung fu has improved but largely "thanks" to the insane things clients now allow you to do.

            And I don't recall when I encountered the first site actually requiring JS for basic functionality, but it certainly was much later than 1996.

      2. ElReg!comments!Pierre

        Re: pool.ntp.org

        Either you know of a web browser that can render JS (bleh) over ssh or you did not read my post. If the former, I don't want to know about it.

      3. ElReg!comments!Pierre

        Re: pool.ntp.org

        > Javascript is built into your browser. What's wrong with that?

        When admin'ing servers over ssh my browser of choice is w3m (but I've been using Links, sometimes). What's wrong with you?

        1. ashdav

          Re: pool.ntp.org

          @ ElReg!comments!Pierre

          I simply asked a question (which you did not answer).

          You have cited an individual case which does not require javascript, which I accept.

          As the original article implies general web browsing I think my statement still stands.

  11. Doctor Syntax Silver badge

    "An investigation by the perennially under-resourced pool discovered"

    The investigation found the list of servers apparently compiled in from a library on Github, a library from which they have now been removed in the current version.

    What sort of eejit compiles in stuff like that? Haven't they ever heard of configuration files?

    1. John Brown (no body) Silver badge

      "What sort of eejit compiles in stuff like that?"

      Probably same sort of people who live link to scripts on 3rd party sites without considering that the script might change or disappear.

      1. Anonymous Coward
        Anonymous Coward

        >Probably same sort of people who live link to scripts on 3rd party sites

        Had some millennial poster on here insist indignantly they only do that during the nightly build (back when some 3rd party LTrim script or whatever was suddenly removed in one of these hipster craptastic web frameworks breaking nearly everyone's build) not realizing that is still a terrible idea.

  12. ksb1972

    So why

    didn't Apple's review of v9.44 catch this nasty bug before they let it onto the App store?

    Which of course also highlights the poor testing Snap Inc themselves do.

    The bigger they get, the lazier they become me thinks.

    1. Velv
      Boffin

      Re: So why

      It's not that simply to test to the scale of 150million daily users.

      1. Anonymous Coward
        Anonymous Coward

        Re: So why

        > It's not that simply to test to the scale of 150million daily users.

        Would the person who downvoted the referenced comment please step forward and share their knowledge of this subject with us, in the spirit of learning from others' experiences and all that? Thanks.

        1. tfewster
          Facepalm

          Re: So why

          15, 15,000, 15,000,000 - Makes no difference, you test the functionality and use-cases, not a proportional number of users.

  13. GrapeBunch

    We need a Fast Show skit

    with "Boutros Boutros Boutros Boutros Boutros Boutros ... Golly". Thank you, Paul Julian Whitehouse.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon