We have a few reports from our customers and a few from one of our colocation customers customer of not being able to reach our network. The problem is very scattered and it appears to be an AT&T problem. All reports are from customers that are connected to the Internet via a cable system provider.

Our bandwidth and IMAP/POP3 connection rate is very normal indicating the problem is not very widespred.

Around 1120 UTC, customers transiting through Cogent were not able to reach some of our servers for about ten minutes. This was a Cogent issue affecting many other Cogent customers.

A reasonable limit to the number of catchall rejects logged will be established. Logging 1,000,000 rejects to a single catchall address is not reasonable.

Messages rejected by MX restrictions for a catchall address will not appear in the reject reports for part of today and going forward. This change will eliminate about 50,000,000 rows from the report database speeding up queries considerably. Catchall addresses were useful when 90%+ of email was not spam, malware, and backscatter. That time is long gone.

Cogent, cogentco.com, is one of our upstream networks and they have some routing issues that are affecting our customers that transit Cogent's network. Some customers in Canada are affected and possibly elsewhere. This issue was resolved by bouncing our BGP session which caused the affected Cogent routers to see the correct routes.

The latest beta release of the Roundcube email client has been installed on a temporary beta site along with PHP5. Many bugs have been fixed in this release of Roundcube.

Earlier this morning a customer reported that they could not connect to one of our webmail servers. They provided a traceroute showing a routing loop where Level3 transits to one of our connectivity providers, Host.net. The problem appeared to be a link load sharing issue, an adjacent IP address in the same /24 network was routed correctly.

Load sharing with equal cost routes is based on the source IP address, the destination IP address, and possibly the source and destination ports. That and the fact that the bulk of our traffic does not transit Host,net, limited the number of customers affected by this issue. SMTP, IMAP, and POP3 access was not affected.

We have seen a number of problems with Level3 routing over the last few months. Last month we dropped our BGP announcements to Level3 to keep as much inbound traffic off of that network as possible. Some outbound traffic will still transit the Level3 network to Level3 customers.

Unfortunately our web services were not in the IP address space that we announce to the Internet with BGP. We have renumbered the webmail client servers and the Manager server into the address space that we announce via BGP. The beta server will be renumbered later this evening.

The change to Daylight Saving Time in the US and Canada occurs tomorrow morning, March 11, 3 weeks earlier than previous years. Many of our long lived services must be restarted for those services to use the updated time zone files that describe when DST starts and ends.

Between 20:00 and 20:30 EST today, 01:00 and 01:30 UTC March 11, IMAP, POP3, LDAP, the web servers providing web client service and the Manager service, will be restarted. Those services will not be available for approximately 10 minutes and most likely less.

New mail will be accepted and queued while the IMAP servers are restarted. No mail will be lost.

Our apologies in advance for this service disruption.

At 12:46 EST, 17:46 UTC, we switched servces on ms3.mxes.net to its replica. IMAP and POP access was not available for about 5 minutes. All incomming mail queues have drained.

Accoriding to MSN's Smart Network Data Services reports, mail that our customers are forwarding to their Hotmail accounts is being reported as spam to Hotmail at rates reaching 5% on some days. The effect of our customers reporting spam that they have forwarded to their Hotmail accounts is that now Hotmail thinks most if not all mail sent from our customer SMTP servers and our forwarding SMTP servers is spam.

Effective immediately mail scoring 1.0 and higher that is forwared to Hotmail and MSN is being discarded. If the complaint rates improve considerably we will increase the discard threshold. If complaint rates do not improve we will be forced to discard all mail forwarded to Hotmail and MSN.

We have already taken several steps to improve the Hotmail situation and to date none have made any difference.

For over 6 months mail that is forwarded to Hotmail scoring 6.0 and higher has been discarded.
We joined the Smart Network Data Services program to have access to MSNs's complaint rate data.
We signed and snail mailed the documents necessary to join MSN's Junk Mail Reporting Partner Program which seems another exercise in frustration since the program is for bulk mailers and no bulk mail is sent from our servers, period.
We routed forwarded mail via SMTP servers that do nothing but forward mail.

The root of the problem is probably Hotmail's busted spam system and the fact that many people have decided that mail they no longer want is spam.

We see it every day in the spam reports we get from AOL. Messages containing conversations with several to dozens of replies by each party is reported as spam by the AOL customer. Shipping information for online purchases is reported as spam by the AOL customer. Responses to requests for information is reported as spam by the AOL customer. Why would a typical Hotmail user see things any differently that a typical AOL customer? We suspect there is no difference and what is reported as spam to Hotmail is in fact not spam but is just mail that is no longer wanted.

And just like AOL, Hotmail places the Junk button right next to the Delete button where mistakes are sure to happen.

Update: Hotmail spam samples.

Its a new feature or it has not been very obvious but some samples of reported spam are available from the MSN SNDS reports.

Two forwarded samples were newsletters forwarded by a company we host email for that is in the same business space as the newsletter senders. The newsletters scored less than 1.0 and at the old and new thresholds were forwarded and would be forwarded to Hotmail. Both sender domains are now in our block lists.

The more interesting samples were not bulk but rather personal mail sent via our customer SMTP servers to Hotmail accounts.

I won't be able to attend...out of town...again! Thanks for taking such good minutes, Kxxxx, I guess that's the only way I know what's going on. Only another month or so of craziness, then things will slow down for me.

hi friends Yes, I made it to Turkey and I've been having a wonderful time here at Kaya Village art camp. It's like a workcamp, but there's no work.

Those two messages may have been mis-addressed since most any random character combination will be a vaild email address at Hotmail. Or the Hotmail email address no longer belongs to who the sender thinks it does. Or the Hotmail recipient is a moron. Those messages may be unwanted but they are not spam.

In any case, we think Hotmail has a pretty broken spam system when one person's notion of spam can affect all other users of the system. There is absolutely nothing we can do to prevent a Hotmail user from pushing the 'spam' button whenever they feel like it resulting in email delivery problems from our servers to Hotmail. Hotmail users can change the accepted definition of spam from "Unsolicited Bulk Mail" to "Mail I don't want".

LDAP is an acronym for 'Lightweight Directory Access Protocol'. Most modern desktop email clients support accessing directories with the LDAP protocol. For our purposes the directory is an address book.

Initially each account has an account address book and a personal address book for each mailbox account. Both address books are visible in the IMP4 web clients and in the LDAP only Squirrelmail web client on the beta test site. Personal address books are visible in the IMP4 and Squirrelmail clients on the production site and the beta site.

Personal address books are read/write in the IMP4 and Squirrelmail clients. Mailbox accounts configured for full management access have read/write access to the account book in the IMP4 clients and the LDAP only Squirrelmail client on the beta site. Mailbox accounts not configured for full management acess have read-only access to the account book. Desktop email clients implement read-only access to LDAP directories.

Desktop client access parameters
LDAP Server name	ldap.mxes.net
Personal address book
cn=mailbox_name,ou=auth,dc=mxes,dc=net	Auth, Bind DN, (Account in Outlook)
ou=mailbox_name,ou=ab,dc=mxes,dc=net	Search base, Base DN, (Advanced tab in Outlook)
Account address book
cn=account_name,ou=auth,dc=mxes,dc=net	Auth, Bind DN, (Account in Outlook)
ou=account_name,ou=ab,dc=mxes,dc=net	Search base, Base DN, (Advanced tab in Outlook)

The LDAP password has to be set before you will be able to use LDAP from a desktop client. Mailbox account LDAP passwords can be set in the Manager on the mailbox management page. Choose LDAP Password in the Action select box. Choose Account -> LDAP Password to set the account LDAP password.

Thunderbird and Seamonkey configuration
Address Book -> File -> New -> LDAP directory
Name:	Your Choice
Hostname:	ldap.mxes.net
Base DN:	ou=your_mailbox_name,ou=ab,dc=mxes,dc=net
Port number:	389
Bind DN:	cn=your_mailbox_name,ou=auth,dc=mxes,dc=net
Click OK

Outlook and Outlook Express configuration
Tools -> Accounts -> Add -> Directory Service
Internet directory (LDAP) server:	ldap.mxes.net
My LDAP server requires me to log on:	Check this box
Click Next
Accout name:	cn=your_mailbox_name,ou=auth,dc=mxes,dc=net
Password:	The LDAP password you set for your_mailbox_name
Log on using Secure Password Authentication (SPA):	DO NOT CHECK THIS BOX
Click Next
Do you want to check addresses using this directory service:	Your choice, probably Yes.
Click Next
Click Finish
Click Properties
Click Advanced
Search Base:	ou=your_mailbox_name,ou=ab,dc=mxes,dc=net
Click OK
Click Close

No Windows email client will update or write to an LDAP directory.
LDAP will not magically sync address books.
No Windows email client will display the contents of an LDAP directory.
LDAP directories are only searchable by Windows email clients.
Thunderbird has the option to download an LDAP directory but Thunderbird attempts an anonymous bind to do the download and that will not work.

Planned enhancements for second quarter 06.

• Additional account level address books
• Finer grain account book access control
• Better support for Thunderbird and Outlook data fields

Early in July the following changes are planned

• Access to the IMP4 Private Address Book will be removed. Currently this address book is visible only to mailbox accounts that have existing entries in that address book.
• The current Squirrelmail Personal Address Book will be removed. The LDAP address books will become the only address book available in the Squirrelmail clients.

SyncML beta test

SyncML is an acronym for "Synchronization Markup Language". The SyncML protocol shows promise in keeping data synchronized between desktop clients, web clients, PDAs/cell phones, and other SyncML servers. SyncML support in PDAs and current generation cell phones is widespread. Support in desktop clients is spotty.

Our SyncML server is very much beta and will be in that state for a while. Each SyncML client implements the protocol in a slightly different way leading to interoperability problems. SyncML is more of a concept right now than an established standard protocol. Hopefully that will change.

Please contact support if you are technically oriented and would like to help test SyncML.

Between 18:00 and 18:10 EDT (22:00 and 22:10 UTC) today, the IMAP/POP server with most of our customers mailboxes, m3.mxes.net, will be rebooted to introduce a kernel parameter change. IMAP and POP services will not be available for approximately 5 minutes during this 10 minute maintenance window.

Further investigation shows that in all probability the IMAP server panic on March 10 was due to hitting a compiled in kernel limit and not due to a bug in the fileystem code as was reported after the March 10 panic.

Our apologies in advance for this brief service disruption.