The results from the second method have been amazing. Without even having to manually classify very many messages Spambayes has been performing at superior levels. Check out Spambayes Wiki for many other training methods. I will discuss easy methods of sorting miss classifications in the Mutt and Squirrel Mail sections.
Spam System Initial Setup
We are going to want Maildir folders to hold our regular mail, mail classified as spam, mail classified as unsure, missed spam, and missed ham.
If the standard Maildir directory (~/Maildir) doesn't already exist lets create it with the necessary sub directories to make it a mail box. Run these commands from your user account who will be receiving the mail.
$ mkdir ~/Maildir
$ mkdir ~/Maildir/{new,cur,tmp}
And now the other 4 directories that we are going to use for dealing with spam (Making the directories hidden is going to be done for integration with Squirrelmail):
$ mkdir ~/Maildir/{.MissedSpam,.MissedHam,.CaughtSpam,.Unsure}
$ mkdir ~/Maildir/.MissedSpam/{new,cur,tmp}
$ mkdir ~/Maildir/.MissedHam/{new,cur,tmp}
$ mkdir ~/Maildir/.CaughtSpam/{new,cur,tmp}
$ mkdir ~/Maildir/.Unsure/{new,cur,tmp}
The messages that we are going to be training as spam are going to be in ~/Maildir/.MissedSpam and ham will be in ~/Maildir/.MissedHam. Lets create the script and cron job that will take care of training the Spambayes db. Create a file ~/Maildir/train_spambayes and add this:
#!/bin/sh
# Script to copy mail missed spam and ham into correct folders
# and run sb_mboxtrain.py to train spambayes
# Training will be done only on missed spam and ham
# Files we saved or bounced may be in the new directory
# We want them in cur for training
mv /home/hypexr/Maildir/.MissedSpam/new/* \
/home/hypexr/Maildir/.MissedSpam/cur 2>/dev/null
mv /home/hypexr/Maildir/.MissedHam/new/* \
/home/hypexr/Maildir/.MissedHam/cur 2>/dev/null
/usr/bin/sb_mboxtrain.py -d /home/hypexr/.hammie.db \
-g /home/hypexr/Maildir/.MissedHam \
-s /home/hypexr/Maildir/.MissedSpam
Make the script executable:
$ chmod +x ~/Maildir/train_spambayes
Set up our database that will be consulted for new email and trained on new spam/ham:
$ /usr/bin/sb_filter.py -d $HOME/.hammie.db -n
You will want cron installed so that it can launch the script above, which trains Spambayes, every night at 3:21 am. Edit your cron jobs with the command crontab -e and add:
21 3 * * * ~/Maildir/train_spambayes
Procmail Configuration for Spambayes
These are basic procmail recipes that will sort the spam from ham. From Spambayes on Unix or Linux. Add these lines to your mail recipient's ~/.procmailrc:
SHELL=/bin/sh
MAILDIR=$HOME/Maildir
DEFAULT=$HOME/Maildir/
CAUGHT_SPAM=$MAILDIR/CaughtSpam/
UNSURE=$MAILDIR/Unsure/
#Spambayes process
:0fw:hamlock
| /usr/bin/sb_filter.py -d /home/hypexr/.hammie.db
:0
* ^X-Spambayes-Classification: spam
${CAUGHT_SPAM}
:0
* ^X-Spambayes-Classification: unsure
${UNSURE}
# Catches everything else.
# Anything left over goes into the DEFAULT folder
:0:
${DEFAULT}
So incoming mail is not filtered through procmail's rules where Spambayes's sb_filter decides whether it is spam or ham. The mail is now sitting in the correct directories and ready to be accessed by whatever reader/front-end you would like. Next I going to describe setting up Mutt and SquirrelMail for viewing your mail. Mutt will be the most powerful option combining speed and sleekness but SquirrelMail will be able to match functionality.
Mutt Configuration
First we need to set the MAIL variable so that Mutt will know where to look for your mail. In your ~/.bashrc (create if does not exist) add the line:
MAIL='~/Maildir'
Exit out of your shell and log back in so that the variable will be set or run export MAIL='~/Maildir'. On some systems mutt will not use $MAIL to determine the location of your mail box. When trying to start mutt if it reports a message that no mail directory exists you can try setting the MAILDIR variable or just create the directory that it wants to use and make it a valid Maildir: mkdir -p ~/.maildir/{cur,new,tmp} where .maildir is the folder that mutt wants to use. If you use a directory other than ~/Maildir remember to substitute that wherever I reference Maildir in this document.
Now for the fun part. Lets make everything look pretty and integrated in Mutt. If you start up Mutt right now you will see any mail that your system has received since you have completed the steps above. If no mail shows up in Mutt and there should be mail you need to first check if Spambayes has filtered it into your .Unsure or .CaughtSpam directories. Type 'c' to change directories and enter ~/Maildir/Unsure and check out the mail thats been classified as spam: 'c' ~/Maildir/CaughtSpam. If you still have not found the mail that you are expecting to have received make sure that you have configured Postfix, Fetchmail, and Procmail correctly. Also, try sending mail directly to your_user_accunt@example.com and see if it shows up in one of the directories above.
Now mail is flowing into your system and we can check how Spambayes has classified it by checking the different mail folders. The miss classified mail just needs to be moved into the correct folders for the training.
To manually move the misclassified spam mail:
1. From you Maildir inbox tag each message that has arrived that is spam. Do this by pressing 't' and an asterisk will appear by the messages. After all of the messages have been tagged, press '; m' and enter ~/Maildir/MissedSpam and the spam messages in your inbox will be moved to the MissedSpam mailbox for training by spambayes.
2. When looking at the mail that it has classified as spam in '~/Maildir/CaughtSpam', tag all of the messages it considered spam that were ham and press '; C' and enter ~/Maildir/MissedHam. Now tag them again and press '; m' and enter ~/Maildir to move them into your inbox, where they should have been placed in the first place.
Of course all of this manual stuff is a huge pain so lets automate all of it in .muttrc.
.muttrc Configuration
This .muttrc file is going to save you a quazillion.. thats right a quiazalliona hours in productivity in dealing with our current setup. Here it is:
# Emacs is my editor of choice. I have mostly read that people like
# using emacs-client here and using an emacs server so that the
# editing opens up in an existing emacs session but I prefer
# loading up a fresh instance. The default editor is some 'vi'
set editor="emacs"
# Setting these macros is going to save us all of that time I was
# talking about. Now instead of having to type all of those
# characters to move the mail around we will can just tag the messages
# and, for the instance of spam in your inbox, hit 'S'.
# Move mail to correct directories
macro index S ";s~/Maildir/.MissedSpam\r\r$\r"
macro pager S ";s~/Maildir/.MissedSpam\r\r$\r"
macro index H ";C~/Maildir/.MissedHam\r\r;s~/Maildir\r\r$\r"
macro pager H ";C~/Maildir/.MissedHam\r\r;s~/Maildir\r\r$\r"
macro index M ";s~/Maildir/\r"
macro pager M ";s~/Maildir/\r"
# This one lets me quickly move job search related emails to the
# correct directory. Want to give me a job?
macro index B ";s~/Maildir/.Job\r\r$\r"
macro pager B ";s~/Maildir/.Job\r\r$\r
# This is how we are going to move around to the different
# mailboxes. Hitting 'alt-1' will take us to our inbox
# 'alt-2' and we are looking at the mail Spambayes classified
# as spam. etc.
# The extra mailboxes I have in here are for mailing lists I am on.
# I will show the procmail recipe that automatically puts
# the mailing list mail into its correct mailbox further down.
