BC Winter Floods 2021
On Sunday, 14 November 2021, the southern, central and coastal regions of British Columbia experienced a "pineapple express" storm which brought significant rainfall and intense wind. The result was massive damage caused to our provincial infrastructure over the following two days. Vancouver became cut-off from the rest of the province as all highways leading from Vancouver inland experienced washouts, mudslides, bridge losses, and flooding. Aside from highway damage, TransMountain and Enbridge both reported preventative pipeline shutdowns, BC Hydro experienced grid failures, municipal water and sewer services were incapacitated, and telecommunications were cutoff.
So what does all this mean for the Internet and its recovery? RhiCom purchases Internet transit (thee peering component for providing Internet to customers) from service providers both with core network in Vancouver where the Vancouver Internet Exchange is located. The two diverse routes RhiCom uses follow the Fraser Canyon (under the rail tracks) and the Coquihalla Highway (parallel to the highway). As a result of the washouts, both fibre routes have been badly damaged and have multiple breaks. In fact, as of 19 November, we have had no information or updates regarding the state of the Coquihalla fibre route and assume it to be lost completely for the next several months.
We will use this page to communicate our recovery progress as well as continue to post updates on our Facebook page. We expect recover to be technically complicated, logistically frustrating, and frankly exhausting. After seeing videos and pictures of the damage to our provincial infrastructure, it is anyone's guess how long reconstruction will take as assessments are still ongoing and some areas of damage are not geologically stable. Recovery is further complicated by existing supply chain issues.
UPDATE 12 - Thursday, 25 November 2021 at 9:35
On Monday evening/Tuesday morning I completed the migration of the last few commercial customers back onto our network IPs.
Yesterday morning, some commercial customers reported VPN issues and not being able to connect. This was interesting, as I was also having VPN issues. The problem turned out to be a specific type of DDoS protection service which was erroneously enabled.
As for the Canyon fibre, a lot of progress has been made by repair crews. For the most part, the cable lies under the rail beds (crossing between CN and CP) and they have made it almost up to Boston Bar. The daily reports we get indicate crews are working insane hours and as they repair one section of cable, they find
further damage (likely from the cable being stretched in some places). No estimated time of repair has been provided as of the last update this morning.
Kayle
UPDATE 11 - Monday, 22 November 2021 at 10:25
Good morning. We had some success last night, and I have been moving traffic since. The majority of customers are back onto our normal public IPs. There are still about 100 more circuits to move over and Blue River is transitioning right now.
The temporary re-route to Calgary is about 15ms longer than to Vancouver; however, I have noticed most of the web sites/services we access are still around 30ms - so it is very tollerable.
We will continue to work on adding our prefixes to the third route just in case something else happens (always good to have options!).
The clean up work from last week will likely take me most of this week to complete. If I have to do any equipment resets I will plan to do them between 2 am and 7 am in the mornings.
Updates postings will likely become less frequent; however, I will continue to post any significant changes in this thread.
Thank you all again for your patience and kindness.
Kayle
UPDATE 10 - Sunday, 21 November 2021 at 14:20
Fibre transport technicians are at the Kamloops Central Office working to re-route our traffic East. This is probably the best news we've had so far. Our primary upstream provider has not given me a timeline, but with techs in the building, I'm expecting over the next 12 to 24 hours we might almost be normal.
Once they complete the physical move (fibre jumpers relocated in the CO), then we can test the circuit without impacting traffic. From there, I will probably do a core switch reset (as the thing has been put through its paces) which will interrupt service for five to ten minutes (hopefully before the morning workday starts). Then I will start to move traffic over by disabling the temporary DHCP server and enabling the proper one. I will keep both the current route and the new routes live in parallel. This will "split" traffic between the restored route and the temporary route. As DHCP addresses expire on your routers, they will renew with the core switch and correct details. It takes about an hour or two for this to happen automatically per network segment.
More updates to come in the morning.
Happy SUNDAY!
Kayle
Once they complete the physical move (fibre jumpers relocated in the CO), then we can test the circuit without impacting traffic. From there, I will probably do a core switch reset (as the thing has been put through its paces) which will interrupt service for five to ten minutes (hopefully before the morning workday starts). Then I will start to move traffic over by disabling the temporary DHCP server and enabling the proper one. I will keep both the current route and the new routes live in parallel. This will "split" traffic between the restored route and the temporary route. As DHCP addresses expire on your routers, they will renew with the core switch and correct details. It takes about an hour or two for this to happen automatically per network segment.
More updates to come in the morning.
Happy SUNDAY!
Kayle
UPDATE 9 - Friday, 19 November 2021 at 16:30
Not a whole lot of useful information to offer today. There is still quite a lot of work going on in the background and several hours on the phone trying to get our prefixes out. We've got confirmation emails saying that orders are in process to make that happen.
I will be working through the weekend and will respond to messages as they come in during normal daytime hours. Generally speaking, I'm seeing the normal number of radios online and approximately the same number of IP addresses issued - so I'm thinking that most everyone has some Internet function at home (also, our phone has been ringing much less today).
On the fibre restoration - the last update from the cable owners was that some sections are still under or in the middle of running water and mud flows. I can't imagine that the main transport will be restored anytime soon. We've asked our primary service provider if they could reroute us East to Calgary or an alternate location so that we can have our bandwidth back. If we get a positive response on that request I will communicate that out.
Cheers all,
Kayle
UPDATE 8 - Thursday, 18 November 2021 at 09:13
I can see many users are back online. Commercial users with static IPs - that's a different story. If you are able to temporarily set your routers to DHCP (dynamic), I can get you a functioning route.
Those commercial customers with dedicated vlans, I will build you a new route as well (I've started and completed those with DHCP addresses).
Cable updates:
Coquihalla - no news at all.
Fraser Canyon - six breaks in the cable were confirmed yesterday. One was repaired late last night. Crews are accessing sites via helicopter and high-railers.
Routing updates:
Trouble tickets are submitted upstream and we are waiting for support to add our network prefixes into their access control lists. Once this is complete, then we can start to be normal again.
Today's task list:
- finish getting people online wherever we can
- make some improvements/repairs to communication into our office (we're still working using call forwarding to mobile devices)
- hopefully get our prefixes added upstream and start to swing traffic back into RhiCom IPs (this can be done mostly without actually affecting service any further)
A note from me, Kayle:
Please remember, British Columbia is very much in a state of emergency and there are people doing some very dangerous jobs to try to get even a sense of what the state of our provincial infrastructure is in. I know for us, the stress and lack of sleep is wearing. We are SUPER thankful to those who have reached out and been patient with us. Your kind words help!
UPDATE 7 - Wednesday, 17 November 2021 at 19:23
I don't want to get too excited, but it is working! Sort of...
We were not able to get the BGP interfaces to work correctly so until we can, we have setup a NATed route for most customer networks. This is VERY crude, but I am currently at home connected to the Internet over my normal home WiFi (just like all of you would be normally).
Your routers are going to need to pull a new IP address from the network core. They will do this automatically over the next couple of hours, but if you want to speed it along, you can reset the power to your home router.
Make sure that you don't re-arrange any of your network cables. The Internet port needs to be connected to the LAN port of our power adapater (black or white with a single light on it).
More updates to come tomorrow as we work to get the BGP back up. BGP or Border Gateway Protocol, btw, is thee internet...
UPDATE 6 - Wednesday, 17 November 2021 at 09:52
Moving further along with good news. We have an Internet connection at our core. But I am struggling with a configuration to advertise our customers outbound (this is a me issue, and my 2,331 page manual is a joy to read). Still thinking that we will have something up this morning as I just cleared another hurdle that took me 4 hours to get through.
Cross our fingers, update 7 is a service restored notice... 
UPDATE 5 - Tuesday, 16 November 2021 at 19:06
I'm late on this but we have made some progress! A MW link is physically constructed and we are now working on a configuration/testing. This part is still going to take a bit, but I am hoping that I can have this up early in the morning. Worst case, we may have to do a little more coordination with the folks helping us out here in town.
We have also heard that not everyone is able to complete calls to our phones. It sounds like this is mostly from those with Bell and TELUS mobile phones. Kyle has mentioned a few times that he has answered incoming calls and they immediately hang up. Know we're working on fixing the Internet as hard as we can!
For the fibre transport update: there are two confirmed breaks. One near North Bend and a second at Basque. A crew was able to fly into some shelter locations by helicopter today and start assessing for damage. We have heard no news on the Coquihalla situation.
UPDATE 4 - Tuesday, 16 November 2021 at 12:01
Design is complete for our MW path as mentioned in last update. About to start building it and hoping to have some service restored tonight or early tomorrow morning. Additional assistance is on the way to help Kyle and I build it.
We also received an update about half an hour ago regarding the Canyon cable - the owners are thinking that there are at least two breaks but that their crews have not been able to get close enough to confirm.
I will post the next update around 1600 this afternoon.
UPDATE 3 - Tuesday, 16 November 2021 at 09:40
Decision made, we are going to build a new MW path to another facility in Kamloops and will try to bring up at least some network late today. Service speeds likely won't be normal, but at least we'll be able to do the necessities until the cable is rebuilt. We're seeing videos on instagram of the washout in the Canyon and it doesn't look good, and the Coquihalla situation sounds worse.
UPDATE 2 - Tuesday, 16 November 2021 at 07:07
We have received some news that crews are expected to be able to access the damage on the Fraser Canyon route today. Once we hear the extent of the damage and an estimated time for repair, we will decide what the next steps are.
We have been offered an alternate third path from another company in Kamloops; however, that process will likely take a day or more to construct. If we don't have an update from the fibre crews by 10am today, we will start working on this plan.
Initial Post - 15 November 2021 at 11.55
We are experiecing a serious outage on our network at this time which is likely related to the storms in the lower mainland. Both of our diverse routes to Vancouver have failed and we are unable to reach our own upstream service providers (their phones are not working).
Our team is working to find a root cause and will post updates regularly here.
Kayle
