Phone System Outage - ACL (INC123)

Symptoms - Phones in ACL location were not connecting to the phone controller. No calls could be made or received.

Observations

• 8:00AM - IT was notified the phones were down by both employees and monitoring system
• 9:00AM - IT contacted vendor and rescheduled an on-site visit to replace the drive that failed in the controller
• 11:15AM - Backup was restored on the USB replacement drive and the phone system came back online
• 11:30AM - Final testing was done to ensure all systems are functioning again
  

Resolution - We had to replace a faulty hard drive with a USB hard drive and rebuild the controller from a backup.

Root Cause Analysis - During the backup stage last night (10/15@8PM) the hard drive failed and crashed. This brought the phone controller down with it. No details about what caused the drive failure as the logs were destroyed with the drive. A USB drive was installed temporarily to get the system back up and running.

We cannot replace the physical drive inside the controller as this will void the ‘warranty’ for repair. We are in communication and planning stages to upgrade the physical hardware with new controllers.

Documentation was captured for how to resolve this issue without the need of a 3rd party vendor.