Update: All services were back online by 15:55 EDT (19:55 UTC). The outage was caused when the building electricians accidently tripped the main panel breaker while removing the panel cover for further investigation into why the breaker failed yesterday.

The datacenter has contracted with an engineering firm to determine where the weak points are in their power distribution systems and what they can do to improve the power system.

Staring tomorrow evening we will begin moving servers and routers that provide load sharing and redundant routing into another cage. This work will take several days to complete and service outages are expected to be very brief. This work will be done between 20:00 and 00:00 EDT (00:00 - 04:00) UTC tomorrow evening and between 16:00 and 20:00 (20:00 - 00:00 UTC) Saturday and Sunday with the same schedule the following weekend.

Update on the power failure during hurricane Katrina. Lightning struck one of the generators supplying power to the first floor taking out the fuel pump that feeds fuel from the large ground level storage tanks to the generator day tank. The generator continued to operate untill its day tank was empty. To help prevent failures of this nature in the future, additional monitoring and pumps are being installed.

15:10 EDT (19:10 UTC) Power has been restored and services will be back online shortly.

Update: Power was restored around 12:10 EST (1610 UTC) and IMAP access via the webmail clients was restored shortly after that. Full access to the IMAP and POP servers followed and finally SMTP access was restored. We are still working to bring up the backup raid systems.

The cause of the failure was an overheated 208V 3-phase circuit breaker in the distribution panel that supplies our racks. The breaker failure was complete and it had to be replaced.

Our second cage in the same datacenter is powered from a different 208V distribution panel. The redundant servers, routing, and switching, will be moved into that cage starting later this week.

We have lost all power to one of our cages at the datacenter. We will be moving enough servers to our second cage to bring services back online.

Between 0300 and 0400 EDT, 0700 and 0800 UTC, our colocation provider will be upgrading their core routers to Cisco layer 3 switching technology. There will be a few short periods during this time when some customers may not be able to reach our servers.