>I have dual WAN (ATT Fiber and TMobile LTE) failover
>for exactly this type of scenario.
1)
Charter had a fiber optic line cut in my town last week.
No problem, I can tether to my iPhone or use the corporate VzW hotspot that they provide specifically so I can do remote support if Charter is down...
Barely any voice signal strength, no data on VzW.
I'm guessing VzW was renting dark fiber capacity from Charter as I know from past tests their Network Layer was separate.
2)
Work has both Charter & Verizon fiber for redundant ISPs, confirmed that they use separate fiber over separate routes to our 2500 employee sized corporate campus.
Started having issues with traffic going to a handful of sites, but some of which were heavily hit by automated processes. Some tracerouting around showed the failures were going out Charter and had a CenturyLink (Level3) router in Boston in common...but the traceroutes also showed that both Charter AND Verizon were using the same Verizon infrastructure between their (separate) local offices and the same facility in Boston...just once it hit Boston the traffic over Charter was routed very CenturyLink for the trip to Atlanta, while Verizon looked like it kept it on it's own network.
I'm just glad I'm not on the network team that had to coordinate a Charter/Verizon/CenturyLink conference call :D 24 hours for us to diagnose and inform Charter (who was getting complaints to the of sporadic problems throughout our area from other customers but hadn't figured it out themselves), 48 more hours before Charter updated the ticket that CenturyLink acknowledged there was an issue on their router, another 24 hours for CenturyLink to fix it.
My conclusion: That the frigging internet works as well as often is impressive, because there is not as much true resiliency as most of think, hope, or fantasize there is. It would take a lot more project management, engineering, and architecture work than most companies will put into it.