Some customers were sent an invoice notice for resellers by mistake. Please ignore that message if you are not a reseller.

IMAP servers ms1.mxes.net and ms6.mxes.net have been switched to their replicas. The incoming mail queues are draining. No mail will be lost.

Several IMAP servers are unreachable. We suspect a power problem. Services are being switched to repliacs.

ClamAV died on one MX server and our monitoring system failed to detect that due to a configuration error. A good amount of mail is queued and the queue is draining. This weekend we will force a failure on all systems to double check the monitoring system. Its running on a new server with new software. Sorry for the mail delay.

This is the new status site software running on a server at johncompanies.com. This software allows us to update this site more easily and the RSS feed will stay in sync.

Postmortem: The nuke plant failure was widely reported by the mainstream media and there is nothing to add to those reports.

The reason our power was affected was that one battery in the battery bank for the third floor UPS failed before the generator could come up to speed. The entire battery bank has been replaced.

Update: LDAP server and SMTP servers are up.
Update: All IMAP servers and IMAP/POP3 muxes are up.
Update: Imap servers ms1.mxes.net and ms3.mxes.net are up.

Working to determine the extent of the problem. Most services are down.

Between 18:00 and 20:00 EST (UTC -0500) ms1.mxes.net, ms5.mxes.net, and ms6.mxes.net will be rebooted to change a kernel parameter. These are the scheduled reboot times.

  ms1.mxes.net 18:00 - Back up and accepting logins at 18:04
  ms5.mxes.net 18:45 - Back up and accepting logins at 18:48
  ms6.mxes.net 19:30 - Back up and accepting logins at 19:33

Downtime for each server will be 3 to 5 minutes. Expect IMAP and POP3 connection failures during this maintenance period. New mail will be queued. No mail will be lost.

We apologize for this service interruption.

At 21:35 EST (UTC -5) IMAP server ms1.mxes.net became unresponsive requiring a switch to its replica. The switch was complete and IMAP/POP3 access restored by 22:00 EST. All queued mail was delivered by 2300 EST. No mail was lost.

We apologize for this service interruption.

Several months back we decided that all new machines would be HP/Compaq and most existing machines would be upgraded to HP/Compaq machines. Today we will switch customers that have accounts on ms1.mxes.net to its replica that has already been upgraded to HP/Compaq. Later in the week or next weekend we will switch to the new HP/Compaq primary.

The switch will take about 60 seconds and during that time customers on ms1.mxes.net will not be able to connect to the IMAP or POP3 server. No mail will be lost.

And last but not least, Happy New Year!

We have a few reports from our customers and a few from one of our colocation customers customer of not being able to reach our network. The problem is very scattered and it appears to be an AT&T problem. All reports are from customers that are connected to the Internet via a cable system provider.

Our bandwidth and IMAP/POP3 connection rate is very normal indicating the problem is not very widespred.

Around 1120 UTC, customers transiting through Cogent were not able to reach some of our servers for about ten minutes. This was a Cogent issue affecting many other Cogent customers.

A reasonable limit to the number of catchall rejects logged will be established. Logging 1,000,000 rejects to a single catchall address is not reasonable.

Messages rejected by MX restrictions for a catchall address will not appear in the reject reports for part of today and going forward. This change will eliminate about 50,000,000 rows from the report database speeding up queries considerably. Catchall addresses were useful when 90%+ of email was not spam, malware, and backscatter. That time is long gone.

Cogent, cogentco.com, is one of our upstream networks and they have some routing issues that are affecting our customers that transit Cogent's network. Some customers in Canada are affected and possibly elsewhere. This issue was resolved by bouncing our BGP session which caused the affected Cogent routers to see the correct routes.

The latest beta release of the Roundcube email client has been installed on a temporary beta site along with PHP5. Many bugs have been fixed in this release of Roundcube.

Earlier this morning a customer reported that they could not connect to one of our webmail servers. They provided a traceroute showing a routing loop where Level3 transits to one of our connectivity providers, Host.net. The problem appeared to be a link load sharing issue, an adjacent IP address in the same /24 network was routed correctly.

Load sharing with equal cost routes is based on the source IP address, the destination IP address, and possibly the source and destination ports. That and the fact that the bulk of our traffic does not transit Host,net, limited the number of customers affected by this issue. SMTP, IMAP, and POP3 access was not affected.

We have seen a number of problems with Level3 routing over the last few months. Last month we dropped our BGP announcements to Level3 to keep as much inbound traffic off of that network as possible. Some outbound traffic will still transit the Level3 network to Level3 customers.

Unfortunately our web services were not in the IP address space that we announce to the Internet with BGP. We have renumbered the webmail client servers and the Manager server into the address space that we announce via BGP. The beta server will be renumbered later this evening.