--- title: Network Outage Preparedness & Fault Monitoring Guide description: Network Outages are going to happen. The marketing department talks about Zero Outages -- and that's a great goal to have. But as the pragmatic engineering image: https://www.ecg.co/hubfs/Imported_Blog_Media/regular_outage-images-1.png --- [![ECG](https://www.ecg.co/hubfs/lp/cloud-migration/logo-vector.svg)](https://www.ecg.co) - Who We Are Company [**About ECG** Learn about the team behind the networks and services that solve real problems.](https://www.ecg.co/about) [**Careers** Join a fun, collaborative, engineer-owned company.](https://www.ecg.co/careers) Industries We Serve [**Enterprises** Optimize multi-site voice networks while meeting compliance requirements.](https://www.ecg.co/sectors/enterprise) [**Universities** Professional services designed for the unique needs of campus networks.](https://www.ecg.co/sectors/universities) [**Government** Engineering services built for government agency voice challenges.](https://www.ecg.co/sectors/government) [**Service Providers** Deep voice expertise to maintain compliance and keep customers connected.](https://www.ecg.co/sectors/service-providers) - Technologies **Carrier & Trunking Services** SIP trunking, resilience & carrier interconnect **Compliance & Call Authentication** STIR/SHAKEN, 603+ & secure signaling **Unified Communications & Collaboration** UC capabilities & platform expertise **Platform Reliability & Security** Infrastructure, capture & network fundamentals ### Carrier & Trunking Services Design, harden, and secure the carrier side of your voice network — from SIP trunking to encrypted media and compliant recording. [SIP Trunking](https://www.ecg.co/technologies/sip-trunking-consulting-expertise) [SIP Trunking High Availability](https://www.ecg.co/technologies/sip-trunking-high-availability) [STIR/SHAKEN](https://www.ecg.co/technologies/stir-shaken) [603+](https://www.ecg.co/technologies/603-plus-do-not-originate) [eSIM & FMC](https://www.ecg.co/technologies/esim-fmc) [VoIP TLS & SRTP](https://www.ecg.co/technologies/voip-encryption-tls-srtp) [SIPREC](https://www.ecg.co/technologies/siprec-call-recording) ### Compliance & Call Authentication Meet regulatory requirements end-to-end — call authentication, robocall mitigation, lawful capture, and encrypted signaling. [STIR/SHAKEN](https://www.ecg.co/technologies/stir-shaken) [603+](https://www.ecg.co/technologies/603-plus-do-not-originate) [Packet Capture / PCAP](https://www.ecg.co/technologies/pcap-wireshark-packet-capture-analysis) [VoIP TLS & SRTP](https://www.ecg.co/technologies/voip-encryption-tls-srtp) ### Unified Communications & Collaboration Modern UC capabilities backed by deep, hands-on expertise across every major voice platform. [E911](https://www.ecg.co/technologies/e911) [Business Texting](https://www.ecg.co/technologies/business-texting) [Teams Replace Phone System](https://www.ecg.co/technologies/ms-teams-replace-phone-system) [Provisioning & Endpoint Mgmt](https://www.ecg.co/technologies/sip-provisioning-endpoint-management) [Call Control & Features](https://www.ecg.co/technologies/call-control-features) [SIP Trunking HA](https://www.ecg.co/technologies/sip-trunking-high-availability) [SIPREC](https://www.ecg.co/technologies/siprec-call-recording) [vCon](https://www.ecg.co/technologies/vcon-virtualized-conversations) Platform Expertise [BroadWorks](https://www.ecg.co/technologies/broadworks) [NetSapiens](https://www.ecg.co/technologies/netsapiens-support) [Metaswitch Alianza Core](https://www.ecg.co/technologies/metaswitch-alianza-core) [Oracle Communications](https://www.ecg.co/technologies/oracle-communications-support) [Metaswitch Perimeta](https://www.ecg.co/technologies/metaswitch-perimeta) [Sansay VSXi](https://www.ecg.co/sansay-vsxi-session-border-controller) ### Platform Reliability & Security The infrastructure layer underneath your voice services — engineered, monitored, and secured by people who run it every day. [EKS / Kubernetes](https://www.ecg.co/technologies/kubernetes-eks) [Critical Network Fundamentals](https://www.ecg.co/technologies/critical-network-fundamentals) [Packet Capture / PCAP](https://www.ecg.co/technologies/pcap-wireshark-packet-capture-analysis) [VMware & Alternatives](https://www.ecg.co/technologies/vmware-alternatives) [Routing, Switching & Firewalls](https://www.ecg.co/technologies/routing-switching-firewalls) [Freeswitch](https://www.ecg.co/technologies/freeswitch-open-source-call-processing) - How We Do It Services [**Services Overview** Our experience and expertise in voice and data engineering.](https://www.ecg.co/services) [**Network Operations** Keep your voice network running with outsourced network operations.](https://www.ecg.co/services/network-operations) [**Staff Augmentation** Expert engineering support to meet deadlines and fill knowledge gaps.](https://www.ecg.co/services/staff-augmentation) [**IT Consulting & Engineering** Complete short-term projects faster with a dedicated project manager.](https://www.ecg.co/services/it-consulting-and-engineering) - Resources #### [Utilities Launch Delay Calculator Estimate what a potential product launch delay would cost your business. Try the calculator](https://www.ecg.co/cost-delay-calculator) #### [Resource Library Guides, White Papers & Tools In-depth resources on voice engineering, compliance, and AI — built from real client work. Explore resources](https://www.ecg.co/resource-library) #### [Blog Latest from the ECG Team Practical articles on SIP, UC platforms, network operations, and voice AI. Read the blog](https://www.ecg.co/blog) - [Events](https://www.ecg.co/events) [Login](https://portal.ecg.co) [Contact](https://www.ecg.co/contact) Who We Are [About ECG](https://www.ecg.co/about) [Careers](https://www.ecg.co/careers) Industries We Serve [Enterprises](https://www.ecg.co/sectors/enterprise) [Universities](https://www.ecg.co/sectors/universities) [Government](https://www.ecg.co/sectors/government) [Service Providers](https://www.ecg.co/sectors/service-providers) Technologies Carrier & Trunking Services [SIP Trunking](https://www.ecg.co/technologies/sip-trunking-consulting-expertise) [SIP Trunking High Availability](https://www.ecg.co/technologies/sip-trunking-high-availability) [STIR/SHAKEN](https://www.ecg.co/technologies/stir-shaken) [603+](https://www.ecg.co/technologies/603-plus-do-not-originate) [eSIM & FMC](https://www.ecg.co/technologies/esim-fmc) [VoIP TLS & SRTP](https://www.ecg.co/technologies/voip-encryption-tls-srtp) [SIPREC](https://www.ecg.co/technologies/siprec-call-recording) Compliance & Call Authentication [STIR/SHAKEN](https://www.ecg.co/technologies/stir-shaken) [603+](https://www.ecg.co/technologies/603-plus-do-not-originate) [Packet Capture / PCAP](https://www.ecg.co/technologies/pcap-wireshark-packet-capture-analysis) [VoIP TLS & SRTP](https://www.ecg.co/technologies/voip-encryption-tls-srtp) Unified Communications & Collaboration [E911](https://www.ecg.co/technologies/e911) [Business Texting](https://www.ecg.co/technologies/business-texting) [Teams Replace Phone System](https://www.ecg.co/technologies/ms-teams-replace-phone-system) [Provisioning & Endpoint Mgmt](https://www.ecg.co/technologies/sip-provisioning-endpoint-management) [Call Control & Features](https://www.ecg.co/technologies/call-control-features) [SIP Trunking HA](https://www.ecg.co/technologies/sip-trunking-high-availability) [SIPREC](https://www.ecg.co/technologies/siprec-call-recording) [vCon](https://www.ecg.co/technologies/vcon-virtualized-conversations) Platform Expertise [BroadWorks](https://www.ecg.co/technologies/broadworks) [NetSapiens](https://www.ecg.co/technologies/netsapiens-support) [Metaswitch Alianza Core](https://www.ecg.co/technologies/metaswitch-alianza-core) [Oracle Communications](https://www.ecg.co/technologies/oracle-communications-support) [Metaswitch Perimeta](https://www.ecg.co/technologies/metaswitch-perimeta) [Sansay VSXi](https://www.ecg.co/sansay-vsxi-session-border-controller) Platform Reliability & Security [EKS / Kubernetes](https://www.ecg.co/technologies/kubernetes-eks) [Critical Network Fundamentals](https://www.ecg.co/technologies/critical-network-fundamentals) [Packet Capture / PCAP](https://www.ecg.co/technologies/pcap-wireshark-packet-capture-analysis) [VMware & Alternatives](https://www.ecg.co/technologies/vmware-alternatives) [Routing, Switching & Firewalls](https://www.ecg.co/technologies/routing-switching-firewalls) [Freeswitch](https://www.ecg.co/technologies/freeswitch-open-source-call-processing) How We Do It Services [Services Overview](https://www.ecg.co/services) [Network Operations](https://www.ecg.co/services/network-operations) [Staff Augmentation](https://www.ecg.co/services/staff-augmentation) [IT Consulting & Engineering](https://www.ecg.co/services/it-consulting-and-engineering) Resources [Launch Delay Calculator](https://www.ecg.co/cost-delay-calculator) [Guides, White Papers & Tools](https://www.ecg.co/resource-library) [Latest from the ECG Team](https://www.ecg.co/blog) [Events](https://www.ecg.co/events) [Login](https://portal.ecg.co) [Contact](https://www.ecg.co/contact) # Outage-Ready: Networks Ready for Anything By: Mark Lindsey 5 min read Jun 5, 2018 Network Outages are going to happen. The marketing department talks about Zero Outages -- and that's a great goal to have. But as the pragmatic engineering and operations team, you can prepare for outages to prevent them and to remediate instantly. 1. Skip the Pretend Redundancy 2. Detect and Triage Faults 3. PCAP: Packet capture 4. Ready for Remote Testing 5. Logging Enabled & Synchronized 6. Prepare the humans ## Skip the *Pretend Redundancy* Everybody knows you need hot spares. *Pretend-Redundancy* is what you get when you spend twice as much on equipment and software but never test it. Salesmen love it because they sell everything you need to be reliable. Many companies *buy* spare equipment and even plug it in, but actually *implementing working redundancy* is far more challenging. Proper physical design can be elusive, and testing time-consuming. But it's worth it to have a robust network. **Outage-Prepared Networks not only build redundancy, they test their redundancy.** > ***Redundancy in networks*** means that the system: > > 1. Has adequate replicas (e.g., spare servers, spare routers) > 2. Continuously synchronizes all necessary information between replicas > 3. Automatically detects faults > 4. Automatically takes the faulted component offline (so that traffic doesn't keep going to the faulted component) > 5. Automatically selects a replacement component replica > 6. Causes the replacement component to become active or start to take the additional workload # Detect & Triage Faults To prepare for outages, you've got to be equipped to know when the faults are occurring. 1. **Get the basics:** Get the status and health information your devices have to offer using SNMP. Is the Interface up? Get alerts when components go offline.You can this fault detection and reporting with a variety of systems, including [OpenNMS](http://www.opennms.com/), [SolarWinds](http://www.solarwinds.com/), and others. 2. **Measure  the normals:** Beyond the basic good/bad readings, you can define Key Performance Indicators (KPIs) like number of active users, concurrent calls, CPU and memory usage, and set thresholds for what is normal or abnormal. CPU of 50% may be normal, but 90% will be normal for some workloads and abnormal for others. 3. **Get the alerts to a human.** Generating alerts isn't enough: **s**ome responsible human has to get the alert *and do the right thing!* The human analysts also need an escalation path to alert the problems-solvers. 4. **Analyze.**  When you get an alert, you need to troubleshoot it to determine its severity. Fault-detections are imperfect, so you need to determine how true the alert is, and triage the situation for seriousness. Few automated alerts are reliable enough to be trusted. > In the June 2018 Visa Europe outage, the problem was bad hardware. The Network Operations team needed to contact people who could travel to the equipment site. The network monitoring analysts need a reliable communication path to other teams. ## PCAP: Packet Capture ![](https://www.ecg.co/hs-fs/hubfs/Imported_Blog_Media/000741-0001-000050-1-1.jpg?width=330&height=255&name=000741-0001-000050-1-1.jpg)Outage-Ready Networks have packet capture capabilities online throughout the network -- at least at the low-speed disaggregated points. Capturing 100 Gbps Ethernet may be impractical today. But capturing the traffic going in and out of each 1 Gbps load balancer should be achievable. Capturing traffic should not require a dispatch to a physical site: the system should be built to allow troubleshooters to activate packet capture, collect and analyze data within minutes. ## Ready for Remote Testing The Operations staff need the ability to test remotely to replicate problems remotely. - Voice (Phone) services -- It's easy to REGISTER a SIP / IMS phone over the Internet, or to build VPNs, to replicate the experience of users attaching at different points in the network - Web services -- It's easy to use VPN or DNS to route traffic to a particular entry point in the network - Local ISPs -- If you have a global service, using advanced BGP or DNS based routing, you can setup service with local ISPs in your markets and provide test units there to let you test the experience of local users in those markets. After the outage has begun, it's usually too late to build Remote Testing capability. Outage-Prepared organizations build these capabilities in advance. ## Logging Enabled & Synchronized All of our systems have logging. In telecom, though, many of those logs are disabled automatically. To prepare for outages, enable the logging you can, and be sure the logs are useful. 1. **Enable the logging.** Turn on the logging you can afford to enable without crippling the device. Debug logging is often too much. 2. **Synchronize the clocks.** To find out what truly happened in a problem, you often need to know the sequence of events that led up to it. But so often the clocks in systems are not synchronized, so that uncovering what occurred fast enough to remediate an outage is too difficult. 3. **Centralize the logging.** Whenever possible, aggregate all your logs to a central location *bearing in mind you'll lose the logging locations as well.* Centralized logging with an analyzer tool like [Splunk](http://www.splunk.com/) can radically reduce remediation time, but if your centralized-logging sites are down or unreachable, you still need to troubleshoot the problems. I prefer systems that store some logs locally, and send a copy to the centralized log storage. ## Prepare the Humans ![](https://www.ecg.co/hs-fs/hubfs/Imported_Blog_Media/adobestock_101287310-1-1.jpeg?width=5760&height=3840&name=adobestock_101287310-1-1.jpeg)To prepare for an outage, you need all of your staff ready to help. Too many organizations depend on their senior most engineers for outage troubleshooting, but this is a big mistake. You need your full staff prepared to triage and analyze hard problems. Training should focus on how to determine whether each component in your network is working. Components can be servers (like www2, or the DNS server 216.128.200.2) or services (like the SIP SBC at 4.1.2.3 or the REST API at https://foo.com/api/v2/). The 24x7 troubleshooting staff should have: 1. List of supported product. (*Supported* means that *if it breaks, we have to fix it.)* 2. Diagrams of how the product works, for purposes of understanding dependencies. 3. List of components involved in each product 4. Method for testing each component. E.g., 1. REGISTER with SIP to sip:user@4.1.2.3 and make a call to sip:+12292442099@4.1.2.3;user=phone 2. curl https://foo.com/api/v2/get-some-good-stuff | grep good-stuff-v2- 5. Expected outcomes: How to interpret the test results With a training and preparation regime, the staff monitoring the network will be able to isolate > Don't rely ***exclusively*** on your senior technical staff to support you in outage. Train the whole staff in testing ## [Does this ring true? ECG helps with preparing your team and your network to be truly outage-ready.](https://www.ecg.co/contact) [troubleshooting](https://www.ecg.co/blog/tag/troubleshooting) ## Similar posts [![What Is Infrastructure Monitoring? Benefits, Tools, & Use Cases](https://www.ecg.co/hubfs/Blogs/2025/What%20Is%20Infrastructure%20Monitoring%20Benefits,%20Tools,%20Use%20Cases/ecg-feat-whatisinfrastructuremonitoring.png)](https://www.ecg.co/blog/what-is-infrastructure-monitoring-benefits-tools-use-cases) ### [What Is Infrastructure Monitoring? Benefits, Tools, & Use Cases](https://www.ecg.co/blog/what-is-infrastructure-monitoring-benefits-tools-use-cases) [![Google Next Big Service: Plain-Text Transcriptions Of Phone](https://www.ecg.co/hubfs/ecg-site/blog-listing-placeholder.jpg)](https://www.ecg.co/blog/77-google-next-big-service-plain-text-transcriptions-of-phone-call-podcast-and-youtube-videos) ### [Google Next Big Service: Plain-Text Transcriptions Of Phone](https://www.ecg.co/blog/77-google-next-big-service-plain-text-transcriptions-of-phone-call-podcast-and-youtube-videos) [![Staff Augmentation vs Outsourcing: Pros, Cons, & What to Choose](https://www.ecg.co/hubfs/Blogs/2025/Staff%20Augmentation%20vs%20Outsourcing%20Pros,%20Cons,%20Best%20Fit/ecg-feat-staffaugmentationvsoutsourcing.jpg)](https://www.ecg.co/blog/staff-augmentation-vs-outsourcing-pros-cons-best-fit) ### [Staff Augmentation vs Outsourcing: Pros, Cons, & What to Choose](https://www.ecg.co/blog/staff-augmentation-vs-outsourcing-pros-cons-best-fit) ## Need Help With Your Network or AI Project? Get expert advice from engineers who've solved complex problems for carriers, enterprises, and government agencies. [Let's Talk](https://www.ecg.co/contact) [![ecg-logo-white](https://www.ecg.co/hubfs/ecg-site/global/ecg-logo-white.svg "ecg-logo-white")](https://www.ecg.co/) 505 North Toombs St. Suite A Valdosta, Georgia 31601 USA #### Headquarters - 1.866.324.0700 US Toll-free - 1.229.244.2099 International - 1.229.244.2399 Fax - [info@e-c-group.com](mailto:info@e-c-group.com) #### Services - [Overview](https://www.ecg.co/services) - [Network Operations](https://www.ecg.co/services/network-operations) - [Custom AI Development](https://www.ecg.co/services/custom-ai-development) - [Outsourced NOC Support](https://www.ecg.co/services/outsourced-noc-support) - [Staff Augmentation](https://www.ecg.co/services/staff-augmentation) - [IT Consulting & Engineering](https://www.ecg.co/services/it-consulting-and-engineering) #### Company - [About](https://www.ecg.co/about) - [Careers](https://www.ecg.co/careers) - [Blog](https://www.ecg.co/blog) - [Resources](https://www.ecg.co/resource-library) - [Events](https://www.ecg.co/events) © ECG, Inc. [Privacy Policy](https://9353907.hs-sites.com/privacy) ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Mark Lindsey", "url" : "https://www.ecg.co/blog/author/mark-lindsey" }, "dateModified" : "2025-10-23T19:38:39.095Z", "datePublished" : "2018-06-05T04:00:00.000Z", "headline" : "Network Outage Preparedness & Fault Monitoring Guide", "image" : [ "https://www.ecg.co/hubfs/Imported_Blog_Media/regular_outage-images-1.png" ], "mainEntityOfPage" : { "@id" : "https://www.ecg.co/blog/129-outage-ready-networks-ready-for-anything", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://www.ecg.co/hubfs/ECG_hexagons_and_url_borderless.jpeg" }, "name" : "ECG, Inc" } } ```