Over the next several days we will be replacing the SSL certifcates on all web, SMTP, IMAP, and POP3 servers. This is being done in response to the recent publication of a possible attack on MD5 signed SSL certificates. The short story is that these researchers have created a CA, Certificate Authority, signing certificate that can be used to sign end entity SSL certificates that will appear to have been issued by the real CA. The gory details are here.

To exploit this MD5 vulnerability requires considerable cryptography knowledge and a significant amount of computing power to create the fake CA signing certificate. The attacker then has to convince the victim to connect to the fake server via DNS hijacking, social engineering, or with phishing techniques. Financial institutions would be the likely target should generating the fake CA certificate actually be acomplished outside of the laboratory.

Apologies for the delay., its been a trying day

We use the OSPF routing protocol internally to advertise the IP addresses of each service to the border routers providing load balancing and failover. The routers were loosing OSPF adjacency and the assumption was that this was an OSPF bug in the routers or in the routing daemons running on the physical servers. OSPF bugs are not unheard of. It appeared that the OSPF processess in the routers were consuming most the the router CPU.

Much time was wasted shutting down all OSPF daemons and adding static routes to provide access to the IMAP and SMTP servers when the real problem was elsewhere. With OSPF shut down the routers were still seeing bursts of 100% CPU causing periods of total packet loss.

The problem was isolated to to a switch in our first floor rack by disconnecting all trunks to the first floor and to our upstreams and reconnecting one by one. Eveything was then disconnected from the first floor switches and reconnected one machine at a time and tested. This was a time consuming process.

The culprit was a machine in our first floor rack that was spewing packets of some sort that was driving the routers to 100% CPU. Counters on the switches and on the machines themselves were not out of the ordinary hiding the real problem.

We have redundant routing, trunks, switches, with two Ethernet interfaces on each server. With this configuration, the network will survive total hardware failures but not what we experienced today. We are not new to routing and this is the first time a failure like this has been seen.

No mail was lost. The network being down will not cause mail to be lost (unless its an Exchange server but that's not our problem). SMTP is a robust queue and retry protocol. Mail is queued untill it can be delivered to the next hop and a positive acknowledgement of receipt is received. Its worked that way for 20 years.

Webmail is now working.

Mail is back up and beginning to flow - for those using IMAP desktop clients. All other processes should be coming online in the next few hours, if not before. Static routes will be put in place within 40 minutes and should fix the problem We apologize for this extremely unusual interruption. It is a routing problem - no mail will be lost.

Router reload did not fix the problem.

Reloading routers now.

Later today an updated *.mxes.net cert will be installed on the IMAP/POP3 proxy servers and on the customer SMTP servers. Some mail clients will whine when the cert expiration date changes.

Some customers were sent an invoice notice for resellers by mistake. Please ignore that message if you are not a reseller.

IMAP servers ms1.mxes.net and ms6.mxes.net have been switched to their replicas. The incoming mail queues are draining. No mail will be lost.

Several IMAP servers are unreachable. We suspect a power problem. Services are being switched to replicas.

ClamAV died on one MX server and our monitoring system failed to detect that due to a configuration error. A good amount of mail is queued and the queue is draining. This weekend we will force a failure on all systems to double check the monitoring system. Its running on a new server with new software. Sorry for the mail delay.

This is the new status site software running on a server at johncompanies.com. This software allows us to update this site more easily and the RSS feed will stay in sync.

Postmortem: The nuke plant failure was widely reported by the mainstream media and there is nothing to add to those reports.

The reason our power was affected was that one battery in the battery bank for the third floor UPS failed before the generator could come up to speed. The entire battery bank has been replaced.

Update: LDAP server and SMTP servers are up.
Update: All IMAP servers and IMAP/POP3 muxes are up.
Update: Imap servers ms1.mxes.net and ms3.mxes.net are up.

Working to determine the extent of the problem. Most services are down.

Between 18:00 and 20:00 EST (UTC -0500) ms1.mxes.net, ms5.mxes.net, and ms6.mxes.net will be rebooted to change a kernel parameter. These are the scheduled reboot times.

  ms1.mxes.net 18:00 - Back up and accepting logins at 18:04
  ms5.mxes.net 18:45 - Back up and accepting logins at 18:48
  ms6.mxes.net 19:30 - Back up and accepting logins at 19:33

Downtime for each server will be 3 to 5 minutes. Expect IMAP and POP3 connection failures during this maintenance period. New mail will be queued. No mail will be lost.

We apologize for this service interruption.

At 21:35 EST (UTC -5) IMAP server ms1.mxes.net became unresponsive requiring a switch to its replica. The switch was complete and IMAP/POP3 access restored by 22:00 EST. All queued mail was delivered by 2300 EST. No mail was lost.

We apologize for this service interruption.

Several months back we decided that all new machines would be HP/Compaq and most existing machines would be upgraded to HP/Compaq machines. Today we will switch customers that have accounts on ms1.mxes.net to its replica that has already been upgraded to HP/Compaq. Later in the week or next weekend we will switch to the new HP/Compaq primary.

The switch will take about 60 seconds and during that time customers on ms1.mxes.net will not be able to connect to the IMAP or POP3 server. No mail will be lost.

And last but not least, Happy New Year!

We have a few reports from our customers and a few from one of our colocation customers customer of not being able to reach our network. The problem is very scattered and it appears to be an AT&T problem. All reports are from customers that are connected to the Internet via a cable system provider.

Our bandwidth and IMAP/POP3 connection rate is very normal indicating the problem is not very widespred.