back to article Boffins want to stop Network Time Protocol's time-travelling exploits

Among the many problems that exist in the venerable Network Time Protocol is its vulnerability to timing attacks: turning servers into time-travellers can play all kinds of havoc with important systems. Complicating the problem is that timing attacks are enabled by the protocol itself, which makes it hard to change. Now a …

Page:

  1. A Non e-mouse Silver badge

    So how's this different from specifying multiple servers for the normal NTP client? When I do that and one server is out of whack compared to the others, you know what? The NTP client rejects the dodgy server as a time source.

    1. Richard 12 Silver badge

      Numbers

      You don't want to just reject one.

      Say you query three servers and your system is set up to reject a single outlier.

      If the attacker can affect two of the servers, your system will reject the good one!

      If you query ten and reject the outliers, an attacker has to delay 6 of them to succeed.

      1. juul

        Re: Numbers

        As long as the client on have one connection to the internet it is too easy to manipulate

        1. DavidD

          Re: Numbers

          Same problem exists in the new proposal.

          It doesn't matter how many servers are queried, if they are all queried over the same internet connection then there is a single point of failure/interception.

    2. Mark in CA

      It's totally different. Specifying multiple servers today means you are still relying on the result from only one server. What is being suggested here is always relying on the result of tens of servers, or more. This is not unlike how scientists today determine what "real" time is, by querying all the atomic clocks in their netowrk of such devices around the world, tossing out outiers and then performing a weighted average.

  2. hammarbtyp

    Time NTP was upgraded(See what I did there!)

    To be honest its about time NTP was replaced with something a bit more fit for purpose. Every month we have a issue with NTP due to mis-configuration or lack of understanding

    Any new algorithm should include a modern security infrastructure using certificates to verify time sources, provide greater accuracy in LAN environments by piggy backing on IEEE 1558, more control on the skewing, improved configuration tool and better monitoring interfaces.

    1. Nick Kew

      Re: Time NTP was upgraded(See what I did there!)

      All that infrastructure defeats the whole purpose of NTP: a lightweight protocol. Add a certificate, and handling it becomes a bottleneck that injects a whole new timing attack vector, quite apart from causing packets to timeout.

      1. hammarbtyp

        Re: Time NTP was upgraded(See what I did there!)

        All that infrastructure defeats the whole purpose of NTP: a lightweight protocol.

        We have SNTP for that

        Add a certificate, and handling it becomes a bottleneck that injects a whole new timing attack vector, quite apart from causing packets to timeout.

        Not necessarily. If the NTP next gen was modular and scalable you could target it to whatever device you wanted. Anyone who has had to wade through the standard NTP code base will tell you that it is huge mess of spagettified code which needs a rethink and rewrite.

        Security wasn't a big thing when NTP was invented, but now is the primary concern. if you are required to get information off-site, you need to have trust, this requires encryption, PKI, etc. You can do this with a VPN if necessary, but the overhead is much the same.

        If you want you can use your edge devices to get the NTP signal and then relay it to other devices using a lower overhead protocol/transport mechanism. It would just be nice if NTP supported all the various use cases

        1. Paul Crawford Silver badge

          Re: Time NTP was upgraded(See what I did there!)

          Lets face it - if your really REALLY depend on time to < 100ms or so accuracy (which seems to be the thing here - as I think trying to delay the NTP out/return by much more than that will lead to rejection anyway) you should get your own GPS receiver to have your own stratum-1 source.

          Sure it is a cost but you can start from £100 (for a Raspberry PI and a GPS expansion board (e.g. from uptronics), antenna, plus a funky case) or get 1U servers for around the £1-3k mark depending on hold-over accuracy and battery back up features.

          And you are doing what that needs super-accurate time? It is not a consumer problem as typically Windows machines are out be much more due to SMTP and (last I checked) ~1 week polling so if you are looking at fraud in the £M region from 100ms of fiddling why are you trusting it all through a single ISP, etc?

          1. david 12 Silver badge

            Re: Time NTP was upgraded(See what I did there!)

            I don't think that <0.1s time is a really demanding standard. The real issue is with people who depend on time, but don't have a place for GPS receiver. Which is small security system devices. The <0.1sec statement is just an indication that their system works well enough to be useful.

            1. Anonymous Coward
              Anonymous Coward

              Re: Time NTP was upgraded(See what I did there!)

              GPS receivers are only one of many potential reliable time sources - and probably the newest. There's plenty of radio clocks (much easier to fit than GPS clocks IME, many countries have at least one tower somewhere broadcasting a time signal of some description - https://en.wikipedia.org/wiki/Radio_clock#List_of_radio_time_signal_stations), you can buy a nice rubidium clock (or if you're really flush you can get a caesium clock with a rubidium backup), you can get a direct connection to an atomic clock at the National Physical Laboratory if you really want to.

              1. Anonymous Coward
                Anonymous Coward

                Re: Time NTP was upgraded(See what I did there!)

                And how reliable are GPS and radio clocks? I seem to remember reading a detailed description once of the signal used by radio clocks and seeing no mention of cryptographic authentication, so that would presumably be easy to fake. I don't know about GPS. Can anyone tell us?

                Another thing about GPS: it doesn't use UTC, does it? So you'd have to, firstly, make sure your systems don't fail when there's a genuine leap second, and, secondly, make sure an adversary doesn't feed you bogus information about leap seconds.

                The next version of NTP should make it easier to get TAI and UTC in a straightforward way from the same server, rather than have to work one out from the other using a document taken from a different server. The clock_gettime system call on Linux lets you ask for TAI and UTC, but very few systems seem to be set up so that this works properly. (Go on, try it!)

                1. This post has been deleted by its author

                2. Paul Crawford Silver badge

                  Re: Time NTP was upgraded(See what I did there!)

                  "And how reliable are GPS and radio clocks?"

                  Generally pretty good, but not totally spoof-proof which is why you normally have more than one receiver (for hardware redundancy) on site and also use a decent number of NTP servers for confirmation as well. I think Meinberg offer servers with both LW and GPS sources for added certainty.

                  "Another thing about GPS: it doesn't use UTC, does it?"

                  No, GPS internally uses an atomic time scale that was in-sync with UTC in 1980 as well as providing the stepped UTC-GPS offset to get UTC today. Any decent GPS module also provides the pending leap second information as well, but sadly these days quite a lot of cheap GPS modules only use NMEA strings to communicate and they don't report leap second information. Also the companies behind them seem to be populated by muppets that don't understand the products or service they are selling.

                  1. A Non e-mouse Silver badge
                    Holmes

                    Re: Time NTP was upgraded(See what I did there!)

                    Also the companies behind them seem to be populated by muppets that don't understand the products or service they are selling.

                    To be fair, you could say that about most companies.

                3. Anonymous Coward
                  Anonymous Coward

                  Re: Time NTP was upgraded(See what I did there!)

                  AC asked, "how reliable [secure] are....radio clocks?"

                  Not even slightly. There's trivial Arduino code floating around to create legacy baseband time code waveform as used by WWVB. As you surmised, it's plain text.

                  Transmitting it at 60 kHz RF is equally trivial. So it's an easy prank to reset all your neighborhood radio clocks to whatever comedy time you like.

                  There's another more recent phase shift waveform layered on top. That'd be another stage to your prank. A few more days effort.

                  1. Paul Crawford Silver badge

                    Re: Time NTP was upgraded(See what I did there!)

                    "Transmitting it at 60 kHz RF is equally trivial."

                    At close range, yes. But not from far away as it takes a pretty big antenna to get any sort of radiating efficiency at 60kHz.

                    It all comes down to your risk assessment, while anyone on the other side of the world can poke at your systems via the Internet, getting up close and personal to fiddle radio clocks carries a higher cost and risk of being caught. Having a combination of sources allows you to pick out dodgy clocks (the "false tickers" in NTP parlance) and more than one radio type adds another layer.

                    But if you do see yourself at risk of a serious, planned and coordinated timing attack and it is of value you can get your own atomic clocks ("low cost" are Rb + GPS adjustment, or if you really must have the bast a few companies make hydrogen masers).

                  2. Bill Stewart

                    Re: Time NTP was upgraded(See what I did there!)

                    Hah - I should try that, not for nefarious reasons, but because I've got a WWVB clock that has trouble getting signal in my house unless I stick it in an upper window. (At the moment, this means retrieving it from behind a dresser, because the cat also likes that window and knocked the clock off the windowsill.)

          2. Peter Gathercole Silver badge

            Re: Time NTP was upgraded(See what I did there!)

            Unfortunately, things like Blockchain, and a lot of historical trading and other financial systems absolutely need reliable sub-second accuracy in order to record the absolute time of transactions to make sure that a successful sequence is recorded. It is here that, for example, making a transaction look like it happened later (or earlier!) than it actually did could invalidate the transaction (think if someone were able to delay your registration of a newly mined bitcoin, and claim it as their own merely because they could subvert the time your system apparently mined it).

            I worked in the electricity distribution industry some time back, and they had a requirement for accurate sub-second time as well, not that I ever asked why ( the fact that I was compiling the xntpd source to include the RCC8000 time clock tells you how long ago that was).

            1. Brian Miller

              Re: Time NTP was upgraded(See what I did there!)

              Unfortunately, things like Blockchain, and a lot of historical trading and other financial systems absolutely need reliable sub-second accuracy in order to record the absolute time of transactions to make sure that a successful sequence is recorded.

              True, and PTP (IEEE 1588-2002) is designed for high-accuracy synchronization.

              I've had the "fun" of setting up configurations of NTP clusters, and making sure they were actually staying accurate. NTP can, and will, go wonky when the configuration isn't right. I've seen a cluster, with uplink, go out of sync over the weekend, and the cluster's time was a week ahead of where it should have been. Yes, the cluster's time was in sync with itself, but not with its master.

            2. Loyal Commenter Silver badge

              Re: Time NTP was upgraded(See what I did there!)

              think if someone were able to delay your registration of a newly mined bitcoin, and claim it as their own merely because they could subvert the time your system apparently mined it

              From my understanding of this, that person would have to get in between your 'mining' the coin and everyone else on the blockchain network hearing about it, and also 'mine' that coin themselves.

              In practice, this means solving the same block themselves in that time interval, since the 'solution' also involves the id of the 'wallet' solving it. Since the Bitcoin network is set up so that the total global processing power is at a level where one block gets solved every ~10 minutes (by adjusting the hashing difficulty on each block based on the last), and assuming that the attacker's window of opportunity is ten seconds (orders of magnitude higher than it actually would be), that equates to having processing power approximately equivalent to 60 times the global total processing power in order to have a 50% chance of pulling off such an attack.

              This ignores, of course, the fact that if you had 60 times the computational power of the network, you would have far surpassed the 50% needed to take control of Bitcoin (the so called 50% attack). The integrity of the network depends on everyone agreeing that a given block was 'mined' by a given 'wallet' and assigning it to them. You'd need 50%+ of the network to take control of that quorum.

          3. John Smith 19 Gold badge
            Facepalm

            if your really REALLY depend on time to < 100ms or so

            Read the article.

            It's all about how tampering with the system time standard enables the attack of other subsystems.

            Which it does.

          4. -tim
            Pint

            Re: Time NTP was upgraded(See what I did there!)

            "Sure it is a cost but you can start from £100 (for a Raspberry PI and a GPS expansion board (e.g. from uptronics), antenna, plus a funky case)"

            We did that with the £40 uputronics GPS hat. I thought it was about 4 times better than the old server we had been using and then I looked closer at the numbers and it appears about 4,000 times better than the older one which was a decade old server that spent its days saving CCTV data on spinning rust. The GPS sits in the warehouse on a beam under one of the plastic skylights. The problem with the hat is it confused FreeBSD boot process since that didn't like the NMEA strings and the 1PPS driver in NTPd can't cope with adjusting the local NMEA clock so for a non-Internet NTP server, you want two Pis and one with a battery back clock to keep the time when power gets cycled.

            1. Paul Crawford Silver badge

              Re: £40 uputronics GPS

              I think my PI + Uptronics GPS board can work stand-alone (no internet, though normally I use the 4 ntp pool servers as well) using Ubuntu with the PPS enabled. My ntp.conf has this setting:

              # Add the NMEA driver using GPRMC (1) and 9600 Baud (16) mode.

              # Also tell it to assume 117ms delay on RS232 and also to enable the 1pps correction using 'flag1'

              server 127.127.20.0 mode 17 prefer minpoll 4 maxpoll 4

              fudge 127.127.20.0 time2 0.117 flag1 1

              Also needed to edit /etc/init.d/ntp to add on start the commands to create symlinks:

              cd /dev ; ln -s ttyAMA0 gps0 ; ln -s pps0 gpspps0 ; cd /

    2. stiine Silver badge
      Thumb Down

      Re: Time NTP was upgraded(See what I did there!)

      Bullshit. Redhat's already on that bandwagon with Chrony, and its useless. On the 400+ servers that run ntpd, I can use "ntpq -c lpeer" but on the handful of CenOS 7, hell, I don't even remember the command, to determine the time sync status.

  3. Blockchain commentard

    And you need a powerful computer to decrypt *any* encrypted/secure time source, NTP or any other proposal. There go precious milliseconds.

    And if your ISP is anything like mine, latency sucks, so I'm happy if my computers are just showing the correct date !!!!

    1. Warm Braw

      There go precious milliseconds

      You don't need encryption as such, you need a signature. You could take a clear text, signed response, assume it's valid and set up a "shadow" clock then check the signature in spare cycles, copying the shadow clock to the real one if it checks out?

      1. hammarbtyp

        Re: There go precious milliseconds

        You don't need encryption as such, you need a signature. You could take a clear text, signed response, assume it's valid and set up a "shadow" clock then check the signature in spare cycles, copying the shadow clock to the real one if it checks out?

        And how do you know if your signature is from a real server? To do that you need to encrypt the signature and unencrypt it and check the server details.

        To be fair it maybe you only really need to do this at connection, after this you may be able to utilise some sort of secure token

        1. Loyal Commenter Silver badge

          Re: There go precious milliseconds

          And how do you know if your signature is from a real server? To do that you need to encrypt the signature and unencrypt it and check the server details.

          - Ask server for time

          - Receive server time

          - Note local time

          - Decode packet

          - Note local time again

          - Add the difference to time in the packet (and also add half the delay between asking for, and receiving packet as network latency)

          I'm sure it's a little more complex than that, but factoring out the time taken to decrypt the packet is trivial.

          1. Yet Another Anonymous coward Silver badge

            Re: There go precious milliseconds

            So you shift the time by a second in each packet and send 1000s of packets - gradually shifting the server clock a few minutes.

            If you always believe your own clock over the NTP feed then why are you bothering with NTP ?

            1. Richard 12 Silver badge

              Re: There go precious milliseconds

              They aren't.

              That post roughly described how NTP works.

              NTP provides clock updates, it's not a clock. It requires that the device have a local clock that is sufficiently accurate over a period of minutes.

              If it drifted far enough over a few milliseconds that you couldn't measure the time taken to decrypt with sufficient accuracy, then your hardware platform is not suitable.

              At the worst case, you can disable interrupts and manually count the CPU cycles taken. Decryption of a known size payload takes a known time (unless your algorithm is broken)

    2. hammarbtyp

      And you need a powerful computer to decrypt *any* encrypted/secure time source, NTP or any other proposal. There go precious milliseconds.

      And if your ISP is anything like mine, latency sucks, so I'm happy if my computers are just showing the correct date !!!!

      Most embedded processors will support standard encryption protocols and we are talking a very small amount of data here. If your device is not capable of supporting encryption then it should not be connecting to the web.

      For the kind of accuracy NTP provides the overhead is not going to effect accuracy. If you want greater accuracy you should be using 1588 anyway.

      However there is a benefit in having a more modular scalable solution which allows you to make move the dial between security and performance. This should also be defined in the standard source code and allow you to have a core functionality and then choose to include specific features such as encryption into the final solution

  4. Nick Kew

    Consumer-grade 'puters

    If you take a look at the time configuration in a typical consumer computer, you'll see one or two NTP servers nominated.

    Erm, yes. A typical consumer computer is a consumer of NTP. It doesn't need nor expect atomic-clock accuracy. If it's within UDP-packet timeout time of its ISP's ntp server, that's plenty adequate. Or if it just polls time hourly, daily, or probably even weekly, that'll do.

    Need more accuracy? Then you're not a consumer-grade 'puter. You want a competent sysop to configure your NTP with lots of peers, and no doubt other critical setup.

    Methinks this is baked in. The protocol is the quintessential UDP user: better to lose a packet than to use a delayed packet! Configuration allows for different levels of operation: peer network, polling frequency, etc. Dammit, when I first set up NTP I used chrony not ntpd, precisely because of its advertised ability to deal with intermittent connections.

  5. hammarbtyp

    It sounds similar to peer to peer authentication where you establish trust which has been pushed as an alternative to PKI for embedded systems

  6. Anonymous Coward
    Anonymous Coward

    Why not use a GPS dongle?

    Any datacenter selling hosting services should have at least one server set up as a tier 1 GPS source with an actual GPS receiver.

    Then the link(s) to the internet aren't vulnerable to interception/delay, and attackers would have to compromise the router (or whatever) connected to the GPS receiver.

    A major datacenter could have three of them, ideally running three OSes (Cisco IOS, Linux, BSD) to make compromising two of them less likely, so this 'crowdsourcing' idea could work. Before anyone complains about how you might not be able to receive GPS inside the datacenter, they can run coax to an outside antenna.

  7. Anonymous Coward
    Anonymous Coward

    NTP already implements auth...

    I have to admit I'm not entirely sure what the problem is here; NTP already supports setting up basic symmetric keys to verify the connection with a remote NTP server. It's nothing special but it stops MITM timing attacks in their tracks. And if the server at the other end is compromised then you're hosed regardless of any authentication attempts.

    https://access.redhat.com/solutions/393663

    http://blog.ine.com/2007/12/28/how-does-ntp-authentication-work/

    If your internal stuff is dependent on accurate time via NTP, then you should also have at least two internal clocks - as DougS points out it needn't break the bank, you can set up a Rapberry Pi with a GPS + PPS PiHat for less than £100 and that'll give you a "easily good enough" time source for millisecond accuracy (and for most businesses, second accuracy is perfectly good enough). Radio signal receivers are another relatively inexpensive option (and useful in a data centre where you might not be able to install an external GPS antenna). Install three or more and you can generally forego having to look outside your own network for NTP at all.

    Bog-standard NTP and commercial clocks (which are usually just a linux box running some version of bog-standard NTP) should all support auth keys without issue (although TTBOMK windows' NTP implementation doesn't support it). Then you might want to think about using external NTP servers as tertiary sources - and again it's quite easy to fudge them to specify their stratum value as much worse than your internal ones, so even if evil haXx0rz perform timing attacks on a majority of your external tertiary servers your internal servers should still be following your internal clocks.

    Call me cynical but wanting to implement a CA/PKI setup over the top of an NTP-ish protocol sounds a lot like another power grab by the people who want to set themselves up holding the keys in the same way that google seem to want to force everyone away from being able to use their own SSL certificates.

  8. Christian Berger

    It's not an actual problem

    I mean, yes, you can shift the time of servers around, in theory, if you put lots of effort into it and if the server operator doesn't have its own local NTP infrastructure, but in reality that's just a lot of hassle for little profit.

    Clients typically don't care about NTP at all and only implement it's braindead cousin SNTP which gives you a very rough approximation of the actual time and date.

    GPS sounds like a good idea, until you are inside, however for mobile devices, which have GPS anyhow, this is a sensible way to get a rather decent precision of time.

    In some places, like Europe you have the additional possibility of getting your time via longwave transmitters. The DCF77 signal, carries the time in a way you can get your error well down below a millisecond. Other simmilar transmitters will still get you the time to a fraction of a second.

    1. BlartVersenwaldIII
      Windows

      Re: It's not an actual problem

      > Clients typically don't care about NTP at all and only implement it's braindead cousin SNTP which gives you a very rough approximation of the actual time and date.

      I guess it depends on the type and size of clients but it's quiet common even in small windows shops to have:

      a) some of the edge networking kit go out to the various internet NTP servers to query time

      b) present that time internally via an internal NTP server on the same networking kit (usually a requirement of other bits of networked kit e.g. telephone PBX)

      c) one or two windows domain controllers configured to grab their time over NTP (not SNTP) straight from this network kit

      d) all domain controllers sending SNTP to the windows clients

      So whilst clients may not care about NTP and are content to sleep with the braindead cousin, they're frequently indirectly dependant on it by virtue of getting the domain controllers to know what year it is.

      1. stiine Silver badge
        Thumb Up

        Re: It's not an actual problem

        Close. Your better bet is have a pair of INTERNAL network devices polling ntp.org. this way your edge device for NTP. syncing AD/DCs to these two+ internal sources (along with everything else internal) is the way to go, IMNSHO.

    2. Charles 9

      Re: It's not an actual problem

      "In some places, like Europe you have the additional possibility of getting your time via longwave transmitters. The DCF77 signal, carries the time in a way you can get your error well down below a millisecond. Other simmilar transmitters will still get you the time to a fraction of a second."

      The US equivalent is WWV out of Fort Collins, Colorado, which gets its timebase from NIST in nearby Boulder. It transmits several different time signals with varying degrees of precision.

  9. Crisp
    Coat

    Simply fit all computers with sundials.

    Then to calibrate the internal clock, simply let your pc sit in the sun for an hour or two to track the sun across the sky and provide absolute precision.

    No new protocol needed.

    I really think these so-called "boffins" tend to over think these things.

    1. BlartVersenwaldIII
      Alien

      Re: Simply fit all computers with sundials.

      ...until you forget to unplug the sundial, some bright spark installs a Sun server in your data centre, then all of a sudden the sunlight changes completely and your time goes completely out of whack! Unless you move the Sun around the room in the proper way, your sundial will always read the same time.

      1. onefang

        Re: Simply fit all computers with sundials.

        "your sundial will always read the same time."

        They say a stopped clock is correct twice a day.

        1. Charles 9

          Re: Simply fit all computers with sundials.

          "They say a stopped clock is correct twice a day."

          Not if it's missing a hand.

          And as for sundials, what if it rains when the times sync up? Plus one of them is likely to occur at night when there's no sun.

    2. Velv
      Childcatcher

      Re: Simply fit all computers with sundials.

      You're not familiar with British weather, are you...

    3. Lotaresco

      Re: Simply fit all computers with sundials.

      You're under-thinking this. As John Taylor observed, if you are designing something to do a job, it is better to have it doing two things rather than one. Hence in this case, the sundial, excellent as it may be, is second fiddle to a Sumerian Water Clock. We can use the Water Clock for processor cooling as well as a time signal.

      1. BlartVersenwaldIII
        Stop

        Re: Simply fit all computers with sundials.

        > We can use the Water Clock for processor cooling as well as a time signal.

        You're over-under-thinking it! If you did that, different loads will heat the water up to different amounts, resulting in changes in water density and thus changes in the temporal dampenflux. You're better off using the heat from the processor to boil the water so that it can power a steam turbine to generate electricity to wind the pendulum.

        1. Nick Kew

          Re: Simply fit all computers with sundials.

          Once you've boiled the water, you have a nice cuppa tea.

          Damn, where's my Infinite Improbability Drive?

        2. Crisp

          Re: Simply fit all computers with sundials.

          @BlartVersenwaldIII

          A good plan with one flaw. If a pendulum needs winding twice a day....

          How are you going to know when to wind the pendulum?

          1. onefang

            Re: Simply fit all computers with sundials.

            "A good plan with one flaw. If a pendulum needs winding twice a day....

            How are you going to know when to wind the pendulum?"

            I was just pointing out that a stopped clock is correct twice per day, seems the perfect solution to your problem.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon