All Locations - Company Wide Network Outage (INC115)

Incident Report for Alene Candles IT

Postmortem

Symptoms:

Networked devices could not connect to the network (internal or external).

Observations:

  • Initial incident was created and troubleshooting began
  • IT discovered that DHCP leases were not renewing to local devices at ACM and ACL
  • IT reconfigured the local Silver-Peak appliance to reset the DHCP leases
  • Silver-Peak was restarted and devices began to come back online‌

Resolution:

IT reset the IP helper addresses in the Silver-Peak controller that pointed devices to the DHCP server. Technical steps: https://alene.itglue.com/3221095/docs/5451199

Root Cause Analysis:

Preventative measures are being handled by an outside project that will reduce the DHCP traffic going to the Silver-Peak.

Posted Jul 21, 2020 - 12:35 EDT

Resolved

This incident has been resolved. If you are still seeing issues please submit a Help Desk issue (http://aleneit/portal/)
Posted Jul 21, 2020 - 12:32 EDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jul 21, 2020 - 11:59 EDT

Identified

The issue has been identified and we are working on a resolution at this time. Please refrain from submitting IT Help Desk tickets or emails on this matter. Thanks.
Posted Jul 21, 2020 - 11:56 EDT

Investigating

We are currently investigating this issue.
Posted Jul 21, 2020 - 11:49 EDT
This incident affected: Network Systems (ACBN (Building 5) Warehouse, ACDC (Bedford) Data Center, ACL (Scarborough) Plant, ACM (8860) Plant, ACMB (Meadowbrook) Warehouse) and ERP System (IFS Cloud Access), Help Desk Ticket System.