All Locations - Global Network Outage (INC114)

Incident Report for Alene Candles IT

Postmortem

Symptoms:

Networked devices could not connect to the network (internal or external).

Observations:

  • Initial incident was created and troubleshooting began
  • IT discovered that DHCP leases were not renewing to local devices at ACM and ACL
  • IT reconfigured the local Silver-Peak appliance to reset the DHCP leases
  • Silver-Peak was restarted and devices began to come back online‌

Resolution:

IT reset the IP helper addresses in the Silver-Peak controller that pointed devices to the DHCP server. Technical steps: https://alene.itglue.com/3221095/docs/5451199

Root Cause Analysis:

Preventative measures are being handled by an outside project that will reduce the DHCP traffic going to the Silver-Peak.

Posted Jul 21, 2020 - 12:36 EDT

Resolved

This incident has been resolved.
Posted Jul 19, 2020 - 09:15 EDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jul 18, 2020 - 12:17 EDT

Identified

The issue has been identified and a fix is being implemented.
Posted Jul 18, 2020 - 12:12 EDT

Investigating

We are currently investigating this issue.
Posted Jul 18, 2020 - 11:38 EDT
This incident affected: Network Systems (ACBN (Building 5) Warehouse, ACDC (Bedford) Data Center, ACL (Scarborough) Plant, ACM (8860) Plant, ACMB (Meadowbrook) Warehouse) and ERP System (IFS Cloud Access), Help Desk Ticket System, Phone Systems.