At 12:46 EST, 17:46 UTC, we switched servces on ms3.mxes.net to its replica. IMAP and POP access was not available for about 5 minutes. All incomming mail queues have drained.

Accoriding to MSN's Smart Network Data Services reports, mail that our customers are forwarding to their Hotmail accounts is being reported as spam to Hotmail at rates reaching 5% on some days. The effect of our customers reporting spam that they have forwarded to their Hotmail accounts is that now Hotmail thinks most if not all mail sent from our customer SMTP servers and our forwarding SMTP servers is spam.

Effective immediately mail scoring 1.0 and higher that is forwared to Hotmail and MSN is being discarded. If the complaint rates improve considerably we will increase the discard threshold. If complaint rates do not improve we will be forced to discard all mail forwarded to Hotmail and MSN.

We have already taken several steps to improve the Hotmail situation and to date none have made any difference.

For over 6 months mail that is forwarded to Hotmail scoring 6.0 and higher has been discarded.
We joined the Smart Network Data Services program to have access to MSNs's complaint rate data.
We signed and snail mailed the documents necessary to join MSN's Junk Mail Reporting Partner Program which seems another exercise in frustration since the program is for bulk mailers and no bulk mail is sent from our servers, period.
We routed forwarded mail via SMTP servers that do nothing but forward mail.

The root of the problem is probably Hotmail's busted spam system and the fact that many people have decided that mail they no longer want is spam.

We see it every day in the spam reports we get from AOL. Messages containing conversations with several to dozens of replies by each party is reported as spam by the AOL customer. Shipping information for online purchases is reported as spam by the AOL customer. Responses to requests for information is reported as spam by the AOL customer. Why would a typical Hotmail user see things any differently that a typical AOL customer? We suspect there is no difference and what is reported as spam to Hotmail is in fact not spam but is just mail that is no longer wanted.

And just like AOL, Hotmail places the Junk button right next to the Delete button where mistakes are sure to happen.

Update: Hotmail spam samples.

Its a new feature or it has not been very obvious but some samples of reported spam are available from the MSN SNDS reports.

Two forwarded samples were newsletters forwarded by a company we host email for that is in the same business space as the newsletter senders. The newsletters scored less than 1.0 and at the old and new thresholds were forwarded and would be forwarded to Hotmail. Both sender domains are now in our block lists.

The more interesting samples were not bulk but rather personal mail sent via our customer SMTP servers to Hotmail accounts.

I won't be able to attend...out of town...again! Thanks for taking such good minutes, Kxxxx, I guess that's the only way I know what's going on. Only another month or so of craziness, then things will slow down for me.

hi friends Yes, I made it to Turkey and I've been having a wonderful time here at Kaya Village art camp. It's like a workcamp, but there's no work.

Those two messages may have been mis-addressed since most any random character combination will be a vaild email address at Hotmail. Or the Hotmail email address no longer belongs to who the sender thinks it does. Or the Hotmail recipient is a moron. Those messages may be unwanted but they are not spam.

In any case, we think Hotmail has a pretty broken spam system when one person's notion of spam can affect all other users of the system. There is absolutely nothing we can do to prevent a Hotmail user from pushing the 'spam' button whenever they feel like it resulting in email delivery problems from our servers to Hotmail. Hotmail users can change the accepted definition of spam from "Unsolicited Bulk Mail" to "Mail I don't want".

LDAP is an acronym for 'Lightweight Directory Access Protocol'. Most modern desktop email clients support accessing directories with the LDAP protocol. For our purposes the directory is an address book.

Initially each account has an account address book and a personal address book for each mailbox account. Both address books are visible in the IMP4 web clients and in the LDAP only Squirrelmail web client on the beta test site. Personal address books are visible in the IMP4 and Squirrelmail clients on the production site and the beta site.

Personal address books are read/write in the IMP4 and Squirrelmail clients. Mailbox accounts configured for full management access have read/write access to the account book in the IMP4 clients and the LDAP only Squirrelmail client on the beta site. Mailbox accounts not configured for full management acess have read-only access to the account book. Desktop email clients implement read-only access to LDAP directories.

Desktop client access parameters
LDAP Server name	ldap.mxes.net
Personal address book
cn=mailbox_name,ou=auth,dc=mxes,dc=net	Auth, Bind DN, (Account in Outlook)
ou=mailbox_name,ou=ab,dc=mxes,dc=net	Search base, Base DN, (Advanced tab in Outlook)
Account address book
cn=account_name,ou=auth,dc=mxes,dc=net	Auth, Bind DN, (Account in Outlook)
ou=account_name,ou=ab,dc=mxes,dc=net	Search base, Base DN, (Advanced tab in Outlook)

The LDAP password has to be set before you will be able to use LDAP from a desktop client. Mailbox account LDAP passwords can be set in the Manager on the mailbox management page. Choose LDAP Password in the Action select box. Choose Account -> LDAP Password to set the account LDAP password.

Thunderbird and Seamonkey configuration
Address Book -> File -> New -> LDAP directory
Name:	Your Choice
Hostname:	ldap.mxes.net
Base DN:	ou=your_mailbox_name,ou=ab,dc=mxes,dc=net
Port number:	389
Bind DN:	cn=your_mailbox_name,ou=auth,dc=mxes,dc=net
Click OK

Outlook and Outlook Express configuration
Tools -> Accounts -> Add -> Directory Service
Internet directory (LDAP) server:	ldap.mxes.net
My LDAP server requires me to log on:	Check this box
Click Next
Accout name:	cn=your_mailbox_name,ou=auth,dc=mxes,dc=net
Password:	The LDAP password you set for your_mailbox_name
Log on using Secure Password Authentication (SPA):	DO NOT CHECK THIS BOX
Click Next
Do you want to check addresses using this directory service:	Your choice, probably Yes.
Click Next
Click Finish
Click Properties
Click Advanced
Search Base:	ou=your_mailbox_name,ou=ab,dc=mxes,dc=net
Click OK
Click Close

No Windows email client will update or write to an LDAP directory.
LDAP will not magically sync address books.
No Windows email client will display the contents of an LDAP directory.
LDAP directories are only searchable by Windows email clients.
Thunderbird has the option to download an LDAP directory but Thunderbird attempts an anonymous bind to do the download and that will not work.

Planned enhancements for second quarter 06.

• Additional account level address books
• Finer grain account book access control
• Better support for Thunderbird and Outlook data fields

Early in July the following changes are planned

• Access to the IMP4 Private Address Book will be removed. Currently this address book is visible only to mailbox accounts that have existing entries in that address book.
• The current Squirrelmail Personal Address Book will be removed. The LDAP address books will become the only address book available in the Squirrelmail clients.

SyncML beta test

SyncML is an acronym for "Synchronization Markup Language". The SyncML protocol shows promise in keeping data synchronized between desktop clients, web clients, PDAs/cell phones, and other SyncML servers. SyncML support in PDAs and current generation cell phones is widespread. Support in desktop clients is spotty.

Our SyncML server is very much beta and will be in that state for a while. Each SyncML client implements the protocol in a slightly different way leading to interoperability problems. SyncML is more of a concept right now than an established standard protocol. Hopefully that will change.

Please contact support if you are technically oriented and would like to help test SyncML.

Between 18:00 and 18:10 EDT (22:00 and 22:10 UTC) today, the IMAP/POP server with most of our customers mailboxes, m3.mxes.net, will be rebooted to introduce a kernel parameter change. IMAP and POP services will not be available for approximately 5 minutes during this 10 minute maintenance window.

Further investigation shows that in all probability the IMAP server panic on March 10 was due to hitting a compiled in kernel limit and not due to a bug in the fileystem code as was reported after the March 10 panic.

Our apologies in advance for this brief service disruption.

Server is back up at 03:55 EST (08:55 UTC).

Web services have been moved to a machine running a new kernel with a fix for a bug that most likely caused the file system corruption on the machine running web services on the 21st. The problem on the 21st was compounded by the RAID system not taking a failing SCSI drive offline and rebuilding on a hot spare drive.

The crash this morning was caused by hardware problem. Replacement hardware will arrive tomorrow.

The IMAP server panic on the 10th was most likely caused by the same kernel bug that caused the file system corruption on the 21st. The kernel on this IMAP server will be updated this weekend or next when system usage is low.

At 03:15 EST (08:15 UTC) web services are not available due to a server crash. IMAP and SMTP services are not affected.

20:20 EST Due to a configuration error, the standard HTTP port, port 80, was not permitted through the firewall. The secure HTTPS port, port 443 was allowed and customers using HTTPS were able to access the web services. 20:05 EST Web services restored at 20:02 EST (01:02 UTC). This morning at 10:07 EST (15:07 UTC) a machine developed a disk space problem that was resolved by 10:20 EST (15:20 UTC). There is an ongoing issue with the RAID system that will require shutting down this machine.

The affected machine runs two customer visible services, a server in the SMTP cluster and a web server for the Manager and the production web clients. Taking this machine offline will be transparent to customers except for web services which will be unavaliable for approximately two minutes between 20:00 EST and 20:05 EST (01:00 and 01:05 UTC). Some customers may loose their login session when web services are restored. Our apologies in advance for this disruption.

Update: 18:56 EST (23:56 UTC) The IMAP server holding most of our customers mailboxes paniced resulting in damage to the file system containing the Cyrus metadata. The file system has been repaired but some metadata files have been lost. These files can be reconstructed but some customers will not be able to receive new mail or access No existing mail was lost and new mail is queued on the MX servers.

We have elected not to switch to a replica IMAP sever and we will continue to investigate the cause of the panic.

The affected IMAP server should be back online by 19:10 EST (00:10 UTC). We apologize for this outage.

We are having trouble with one IMAP server. We are working the problem but there is no ETR at this time.

Effective today at 12:00 EST (17:00 UTC), all forwarded mail scoring 6.0 and higher is being discarded.