There is an outage in New York / New Jersey at this time.
The facility are investigating. It is impacting the entire building.
The facility have advised the following:
We are currently experiencing a sitewide power issue related to our UPS systems at EWR1. We have site personnel investigating the issue. We have called our UPS vendor who is currently in route to the site to assist us as well. We will send updates every 15 minutes as we make progress on resolution.
UPDATE 1 from DATACENTRE:
Power remains off at our data center in Secaucus/EWR1 per the local fire marshal. We have had an electrical failure with one of our redundant UPS' that started to smoke and then had a small fire in the UPS room. The fire department was dispatched and the fire was extinguished quickly. The fire department subsequently cut power to the entire data center and disabled our generators while they and the utility verify electrical system. We have been working with both the fire department and the utility to expedite this process. We are currently waiting on the fire marshal and local utility to reenergize the site. We are completely dependent upon their inspection and approval. We are hoping to get an update that we can share in the next hour. At the current time, the fire department is controlling access to the building and we will not be able to let customers in.
UPDATE 2:
After reviewing the site, the fire Marshall is requiring that we extensively clean the UPS devices and rooms before they will allow us to re-energize the site. We have a vendor at the site currently who will be performing that cleanup. We will provide an update at 8:00AM EDT unless something significant changes overnight.
UPDATE 3:
DIRECT UPDATE FROM DATACENTRE -
The EWR Secaucus data center remains powered down at this time per the fire marshal.
We have just finished the meeting with the fire marshal, electrical inspectors, and our onsite management. We have made great progress cleaning and after reviewing it with the fire marshal, they have asked us to clean additional spaces and they have also asked us to replace some components of the fire system. They have set a time to come back and review these requests at 9am EDT Wednesday. We are working to comply completely with these new requests with these vendors and are bringing in additional cleaning personnel onsite to make the fire marshal's deadline.
In preparation for being able to allow clients onsite, the fire marshal has stated that we need to perform a full test of the fire/life safety systems which will be done after utility power has been restored and fire system components replaced. We have these vendors standing by for this work tomorrow.
UPDATE 4:
All servers in New York except for Budget KVM Node 2 and Ryzen Node 1 are back online, even these two servers are being checked and will be back online soon
UPDATE 5:
All servers in New York except Budget KVM Node 2 are back online, we are still working on bringing even this server back online.
UPDATE 6:
All servers are Back online
