Enhancing Service Resilience: Addressing the Recent Outage and Our Steps Forward
April 18th Circuit Outage at Teraco
On April 18, 2023, 123NET customers experienced an unexpected disruption in service due to a power outage at the TERACO data center. The outage occurred around 9:30 AM and was resolved by approximately 12:00 PM. As a trusted Fiber Network Operator and Internet Service Provider, we at 123NET want to provide a transparent overview of the incident, our response, and the steps we are taking to enhance service reliability.
Timeline of Events and Immediate Response
The service disruption was initially traced back to maintenance on one of the power circuits within the TERACO facility. However, a further complication arose when our APC Automatic Transfer Switch (ATS), which is designed to seamlessly switch between primary and secondary power circuits (labeled A and B), failed to operate as intended. This failure prevented the ATS from switching power sources effectively, resulting in a sustained outage.
Despite the sudden loss of connectivity, our technical team acted swiftly. Upon identifying the source of the problem and the ATS issue, we coordinated closely with TERACO to address the circuit failure and to manually manage the power source switch. This swift action ensured that the outage lasted less than three hours, minimizing the impact on our customers as much as possible.
Challenges Faced and Lessons Learned
The primary challenge during this incident was the ATS’s failure to switch between the A and B circuits, compounded by the lack of real-time communication from TERACO about the ongoing maintenance. Not being forewarned prevented us from proactively managing the situation and informing our customers in a timely manner.
Commitment to Enhanced Communication and Reliability
Understanding the critical nature of internet connectivity, especially in today’s digital age, we are taking significant steps to bolster our operational and communication strategies. Our commitments include:
- Overhauling our internal systems to ensure more robust and effective customer communication, even during outages.
- Enhancing our network monitoring capabilities to detect and address issues more swiftly.
- Strengthening our partnerships with facilities like TERACO to ensure transparency and responsiveness are prioritized.
Looking Forward
The April 18th outage was an unfortunate event, but it has reinforced our dedication to improving every facet of our service delivery. We are actively implementing lessons learned from this incident to prevent similar occurrences in the future. Our commitment to our customers remains steadfast: to provide reliable, fast, and communicative internet services.
We thank you for your understanding and patience during this outage. As always, we value your trust in 123NET and are dedicated to keeping you connected and informed.