We have a few reports from our customers and a few from one of our colocation customers customer of not being able to reach our network. The problem is very scattered and it appears to be an AT&T problem. All reports are from customers that are connected to the Internet via a cable system provider.

Our bandwidth and IMAP/POP3 connection rate is very normal indicating the problem is not very widespred.

Around 1120 UTC, customers transiting through Cogent were not able to reach some of our servers for about ten minutes. This was a Cogent issue affecting many other Cogent customers.

A reasonable limit to the number of catchall rejects logged will be established. Logging 1,000,000 rejects to a single catchall address is not reasonable.

Messages rejected by MX restrictions for a catchall address will not appear in the reject reports for part of today and going forward. This change will eliminate about 50,000,000 rows from the report database speeding up queries considerably. Catchall addresses were useful when 90%+ of email was not spam, malware, and backscatter. That time is long gone.

Cogent, cogentco.com, is one of our upstream networks and they have some routing issues that are affecting our customers that transit Cogent's network. Some customers in Canada are affected and possibly elsewhere. This issue was resolved by bouncing our BGP session which caused the affected Cogent routers to see the correct routes.

The latest beta release of the Roundcube email client has been installed on a temporary beta site along with PHP5. Many bugs have been fixed in this release of Roundcube.

Earlier this morning a customer reported that they could not connect to one of our webmail servers. They provided a traceroute showing a routing loop where Level3 transits to one of our connectivity providers, Host.net. The problem appeared to be a link load sharing issue, an adjacent IP address in the same /24 network was routed correctly.

Load sharing with equal cost routes is based on the source IP address, the destination IP address, and possibly the source and destination ports. That and the fact that the bulk of our traffic does not transit Host,net, limited the number of customers affected by this issue. SMTP, IMAP, and POP3 access was not affected.

We have seen a number of problems with Level3 routing over the last few months. Last month we dropped our BGP announcements to Level3 to keep as much inbound traffic off of that network as possible. Some outbound traffic will still transit the Level3 network to Level3 customers.

Unfortunately our web services were not in the IP address space that we announce to the Internet with BGP. We have renumbered the webmail client servers and the Manager server into the address space that we announce via BGP. The beta server will be renumbered later this evening.

The change to Daylight Saving Time in the US and Canada occurs tomorrow morning, March 11, 3 weeks earlier than previous years. Many of our long lived services must be restarted for those services to use the updated time zone files that describe when DST starts and ends.

Between 20:00 and 20:30 EST today, 01:00 and 01:30 UTC March 11, IMAP, POP3, LDAP, the web servers providing web client service and the Manager service, will be restarted. Those services will not be available for approximately 10 minutes and most likely less.

New mail will be accepted and queued while the IMAP servers are restarted. No mail will be lost.

Our apologies in advance for this service disruption.