storm, teacup
Do we now await a whole range of "accidental" outages, in the name of some random's ideological beliefs? How can this happen?
Neil Fenemor @ NANOG sez:
"While they are deliberately blocking Youtube nationally, I suspect the wider issue has no malice, and is a case of poorly constructed/implemented outbound policies on [PT's] part, and poorly constructed/implemented inbound polices on [PCCW's] part."
John van Oppen @ NANOG sez:
"... PCCW allows unfiltered route-announcement capability to a large number of their customers..."
Simon Lockhart @ NANOG sez:
"So, from the tit-bits I've picked up from IRC and first-hand knowledge, it would appear that 17557 leaked an announcement of 208.65.153.0/24 to 3491 (PCCW/BTN). After several calls to PCCW NOC, including from Youtube themselves, PCCW claimed to be shutting down the links to 17557. Initially I saw the announcement change from "3491 17557" to "3491 17557 17557", so I speculate that they shut down the primary link (or filtered the announcement on that link), and the prefix was still coming in over a secondary link (hence the prepend). After more prodding, that route vanished too.
Various mitigations were talked about and tried, including Youtube announcing the /24 as 2*/25, but these announcements did not seem to make it out to the world at large.
Currently Youtube are announcing the /24 themselves - I assume this will drop at some time once it's safe.
It was noticed that all the youtube.com DNS servers were in the affected /24. Youtube have subsequently added a DNS server in another prefix."
Steve Bellovin @ NANOG sez:
"...a number of us have been warning that this could happen. More precisely, we've been warning that this could happen *again*; we all know about many older incidents, from the barely noticed to the very noisy. (AS 7007, anyone?) Something like S-BGP will stop this cold.
Yes, I know there are serious deployment and operational issues. The question is this: when is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done."
Patrick Gilmore @ NANOG sez:
"How many of those [incidents] would be stopped with transit providers filtering their downstreams? Which doesn't require rolling out a new technology like SBGP. And, I would argue, if we cannot even get transit providers to filter their downstreams, there is no way in hell we can get transit providers to filter on some RR or doing authentication on individual prefixes."
Matsuzaki Yoshinobu @ NANOG sez:
"I am in the APRICOT meeting in Taipei now, and met a guy from PCCW/AS3491. I have showed him this thread, and have suggested
1) validating prefixes from downstreams before accept, and
2) setting an inbound prefix-filter to their downstreams."
Michael Dillon @ NANOG sez:
"The real solution to the YouTube issue is for people to pressure other network operators to raise their game and pay attention to how they manage their BGP trust relationships and filter announcements. In addition, more people need to get involved in information sharing arrangements like Routing Registries, MyASN, alert services and so on. "
Stu sez:
So it seems that it's unlikely that a site can be maliciously hijacked in this way. It's simple enough to filter the rogue route announcements, however in this case the world's ISPs did not have to do this to restore YouTube, they simply waiting for PT's upstream to filter the announcements for them. If in the future some group of cyber-warriors wanted to hijack, say the Reg, they would need the co-operation of the Reg's ISP, plus the other upstream providers involved. They would also then need the world's ISPs to ignore this activity. All very unlikely. Because the internet is comprised of thousands of separate networks, managed by separate companies and individuals, it is not possible to maintain control of this type of hijack. And therefore, if attempted, it would amount to nothing more than a stunt.
This incident has highlighted the importance of "transit provider downstream announcement filtering", and has revealed that filtering is not well-implemented at some transit providers. However, no amount of technology will prevent a mistake at a trusted provider from advertising false routes. Pressure applied by PCCW's peers (other transit providers) was sufficient to prompt a fix in this case. If a rogue transit provider consistently advertises false routes, that provider will be filtered by its peers. If this turns into a nasty intractable problem, SBGP will apparently ride to the rescue.