The datacenter where our racks are located lost power to the first floor around 2230 EST (0330 UTC) taking out most carriers in the center. Power was restored to the first floor around 2300 EST (0400 UTC).

An update will be posted when we know more about what caused the power outage and what steps will be taken to prevent future occurances.

Between 0100 and 0130 EDT, 0500 and 0530 UTC, our colocation provider will be testing failover to backup circuits. There will be a few periods of routing instability during this period. We do apologize for the inconvience this will cause for some customers.

Between 0100 and 0200 EDT, 0500 and 0600 UTC, our colocation provider will be upgrading line cards in one of the border routers. We do apologize for the inconvience this will cause for some customers.

Between 0030 and 0045 EDT, 0430 and 0445 UTC, our colocation provider will be upgrading line cards in one of the border routers. There will be two periods of brief routing instability while routing is moved from and back to the router being upgraded.

Between 1200 and 1230 EDT, 1600 and 1630 UTC, we will be updating the software on the backend IMAP servers in preparation for supporting near real-time replication of IMAP events to the backup IMAP servers.

This update requires restarting the IMAP servers. IMAP and POP3 access will not be available during the update. This update is expected take between 5 and 15 minutes.

Accounts created after 1500 EDT (1900 UTC) today will use an alternate IMAP namespace. In this alternate namespace IMAP clients will see the INBOX folder on the same level as other top level folders. This change will better support IMAP clients that make invalid assumptions about the IMAP folder hierarchy.

The namespace can be changed per mailbox in the Manager.

Effective May 21, 2005, mail that is forwarded to AOL accounts that scores as spam will be silently discarded. This change is necessary to insure that legitimate non-spam mail can continue to be forwarded to AOL accounts by our customers.

Before you push that AOL 'Report Spam' button, please check the message headers to see if the spam was sent directly to your AOL account or forwarded from your domain. Some threshold of spam reports from our IP addresses will result in AOL blocking all mail from those IP addresses.

The routing issue caused 50% to 80% packet loss for about 20 minutes for some customers. One of the routers causing this has been taken offline and its backup has taken over active routing duty. The problem router will be restarted later tonight EST.

We are currently experiencing an internal routing problem at the data center. We hope to have this issue resolved ASAP.

We are testing new versions of the Squirrelmail and IMP webmail clients at our beta test site. Both clients have many new features and configuration options. Please try these new clients and contact support if you have any questions or comments about these new clients.

At 1037 EST, 1537 UTC, the authentication server for the mail storage servers stopped responding to authentication requests. Unable to determine the reason for failure we restarted the server at 1055 EST, 1555 UTC, and it is now functioning properly. We will continue to investigate the cause of this failure.

Update on the Jan 26, 2005 power outage.

Batteries for the failed UPS system arrived by truck from Tampa and were installed late yesterday afternoon. By 1830 EST, 2330 UTC, we were back on UPS power.

Power Outage Update: January 27 0330 EST, 0830 UTC

At 1505 EST, 2005 UTC, a UPS in the data center failed and our systems were without power for approximately 20 minutes. The engineers at the data center attempted to bring the UPS back online without success and utility power was switched to the circuits fed by the failed UPS. When power was restored one router failed to boot and its HSRP backup failed to announce our routes contributing to the problem.

All servers suffered some file system damage requiring another 10 to 90 minutes to repair before the servers could be brought online. Approximately 80 minutes after the power failure the incoming MX servers, the outbound SMTP servers, the MX relay servers, and two mail storage servers were back in service.

The last mail storage server was back online just before 1800 EST, 2300 UTC. The web sites were brought up shortly after all mailbox servers were up and functioning. The Account Manager followed shortly after that.

The UPS failure was not an inverter failure as reported earlier. When the Powerware engineers arrived on site diagnostics showed that the UPS was healthy but the batteries were not up to the task. Throughout today the facility was experiencing short but serve utility power fluctuations causing all of the UPS systems to switch in and out of battery operation and the batteries were not being fully recharged when returning to utility power. This frequent discharge and inadequate recharge caused the batteries in this UPS to fail resulting in this outage.

The data center is expediting delivery of new batteries for this UPS and the other UPS systems in the data center. The new batteries may arrive as early as tomorrow afternoon, Friday.

We sincerely regret the inconvenience to our customers caused by this unfortunate outage.

Due to the impending arrival of hurricane Frances in South Florida, customer service will not be available by phone or email on Sep 3, Sep 4, and possibly Sep 5 if Frances takes a more Westerly direction.

There is no concern that email services will be impacted by hurricane Frances.