Upstream Carrier Issues
Incident Report for Level365
Postmortem

I wanted to take a moment and provide an update on the issue that affected one of our voice carriers, Bandwidth (status.bandwidth.com) that may have intermittently affected some inbound calls on the Level365 UC platform. Beginning on Saturday, September 25, 2021, Bandwidth was the victim of a distributed denial of service (DDoS) attack that affected their voice and messaging services. Bandwidth is a leading wholesale distributor of calling services and in addition to Level365, provides similar services to RingCentral, 8x8, DialPad, and many others. 

When choosing vendors we work with, we only select the top companies in their industry. For example, in addition to Bandwidth, we also partner with Lumen (formerly Level 3, CenturyLink), Verizon, AT&T and other speciality carriers for international and unique calling termination areas. 

We build redundancy into all aspects of our network and platform; multiple geo-redundant data centers with high-availability services, multiple redundant backbone connections to public Internet peering points and many other architected hardware and software failover systems to provide uninterrupted services. Unfortunately, no matter how well architected a system is, there is no way to eliminate every single point of failure issues at the carrier level outside our network and platform. While we can use multiple providers to allow for redundant outbound calling, an inbound phone number has to be attached to a single carrier and that carrier is responsible for routing those inbound calls from the public switched telephone network (PSTN) to the Level365 UC platform.  

Regardless where the failure occurred, and in defense of Bandwidth was a targeted act of extortion, the responsibility falls to Level365 to mitigate any carrier outages. Therefore, we will be taking the following steps to prevent this type of issue in the future:

  • Discuss with all of our carriers what protections they have in place to handle an infrastructure attack such as this.
  • Deploying DDoS protections within our own infrastructure to prevent this from happening against us directly.
  • Investigating options that will bypass the traditional PSTN limitation that prevented us from mitigating sooner.
  • Introduce quarterly carrier reviews to ensure that the vendors we utilize are meeting our expectations.

Again, at the end of the day this was our issue because you, our valued clients, were disrupted.  And for those of you that have worked with our support team on problems in the past that involved third parties, you know that we never point the finger or deflect.  We will learn from this issue and make adjustments where necessary, and come out of this more resilient.

Posted Oct 06, 2021 - 08:27 EDT

Resolved
This incident has been resolved.
Posted Sep 27, 2021 - 06:05 EDT
Monitoring
We are receiving mixed messages from our upstream carrier. We believe the issue has been resolved however. We continue to monitor.
Posted Sep 26, 2021 - 20:22 EDT
Update
We are continuing to work on a fix for this issue.
Posted Sep 26, 2021 - 11:49 EDT
Identified
One of our upstream carriers is having an outage. This will impact some inbound calls as well as 911 service. Outbound should be unaffected. We are working to mitigate as much as possible while we wait for the carrier to resolve their issue.
Posted Sep 26, 2021 - 11:49 EDT
This incident affected: Voice Carriers (Bandwidth).