Orange is my favorite color
May 4, 2011

Linux - Research/HOWTO - Web/Internet | Trackback | del.icio.us | digg

March was a busy month – we were upgrading and consolidating infrastructure around here which meant taking a 2U VA Linux 2230 server that had – quite literally – been serving mail for more than a decade, and upgrading it to a more contemporary machine for which spare parts exist. Knock on wood the new machine can have the same reliable track record.

I had been considering outsourcing everything – move incoming email to Google Apps and use someone like SMTP.com or SendGrid for outbound but there’s a few wrinkles like mailing lists and customer-relayed email. We ultimately decided to roll a new machine but given how quickly email deliverability evolves, we needed to get up to date with the various authentication and verification technologies that help get emails into the inbox. That meant getting waist-deep in SPF, Sender-ID, DomainKeys and DKIM. (See current DKIM and SPF deployment rates – 23% and 51% globally as of this writing)

This isn’t a step-by-step guide; writing one would simply repeat what other people have already done a good job of documenting. What this is, is an overview that ties together missing pieces and endless Google searches to move from concept to execution with one full set of examples.

The Moving Pieces

If you’re new to all this, here’s how these technologies work briefly: Sender ID is basically a Microsoft version of Sender Policy Framework (SPF). If you get SPF to work, Sender ID is backwards compatible and will verify successfully. DKIM is almost the same thing to DomainKeys but it adds Author Domain Signing Practices (ADSP) as an optional component that specifies how strict you are about signing messages.

SPF and Sender ID rely on specially crafted DNS records that identify which mail servers are allowed to send mail for your domain. By contrast, DomainKeys/DKIM take pieces of your message, cryptographically hash them with a key (part of which only your server knows) and embed the signature as a header which is verified using the other half of your key (which is public) that is stored in a specially crafted DNS record for your domain or mail server. You must have the ability to create DNS TXT records or none of these will work for you. Here’s a good article with a more details on how it all works.

DomainKeys verified mail on Yahoo gets a special lock icon showing it's verified

The idea is to eliminate any joe spammer from using an arbitrary From address in spam, phishing and other email attacks. None of these approaches are perfect but they are used by large ISPs like Hotmail, Gmail and Yahoo so if you want your email to reach those inboxes, you need to play along.

QmailToaster

Step one, set up a QmailToaster. Our current server is (older) Qmail + Vpopmail + Courier based and it has worked well so we stuck with what we know. Not to mention, Jake Vickers and the QmailToaster team have removed all of the pain of setting it up so now you just speed along to the part where it works well. Other than getting the needed dependencies on my CentOS/RHEL box, the installation was a piece of cake using the qtp-newmodel approach.

After you install the DKIM package from above on RHEL/CentOS 5, you would see this error in your qmail logs:

@400000004d7d3bc00da74aec delivery 21: success: ould_not_find_ParserDetails.ini_in_/usr/lib/perl5/vendor_perl/5.8.8/XML/SAX/rUser_and_password_not_set,_continuing_without_authentication./<[email protected]>_96.244.219.19_accepted_message./Remote_host_said:_250_2.6.0_message_received/

Basically, something is borked in the CPAN-installed SAX package. You can fix it by pre-emptively running (thanks to centosfiles for the fix):

perl -MXML::SAX -e "XML::SAX->add_parser(q(XML::SAX::PurePerl))->save_parsers()"

At this point you have a mail server capable of handling SPF, Sender ID, DomainKeys and DKIM. Now you need to enable it in your configuration and your DNS.

SPF/Sender ID

Unlike DomainKeys and DKIM which require public/private keys and some computational effort when delivering email, SPF and Sender ID are exclusively a DNS configuration option.

  • http://wiki.qmailtoaster.com/index.php/SPF – there technically isn’t anything to set up with the QmailToaster since your outbound configuration is all DNS. However, you can configure how to handle incoming email that doesn’t pass using /var/qmail/control/spfbehavior.
  • Qmail SPF – The underlying implementation of SPF for qmail
  • OpenSPF.org’s Record Wizard – generate an SPF record for your domain to be placed in a DNS TXT record

Once generated, take the TXT record and put it into your DNS files. If you’re using djbdns, a simple record with one mail server would look something like:

'sample.com:v=spf1 ip4:your.ip.add.ress ~all:3600

The ~all says to softfail messages that don’t verify. If you are sure you have listed all of your possible outbound MTAs, then you can set it to -all which will tell remote servers to kill emails that don’t pass.

DomainKeys/DKIM

Once installed, you enable incoming verification and outgoing signing through the tcp.smtp file stored in /etc/tcprules.d:

For DKVERIFY, the default is “DEGIJKfh”. If you want to permanently reject incoming mail that has a DomainKey signature but fails to verify, add a “B” to the list: “BDEGIJKfh”.

Outbound

To sign an outbound message for DomainKeys/DKIM, you need to generate a public/private key pair. Part of it you keep secret and sign your outgoing messages and the other half you publish in your DNS so other mail servers can verify it’s authentic. Here’s what the docs say to do:

# cd /var/qmail/control/domainkeys
# mkdir yourdomain.com
# cd yourdomain.com
# dknewkey private > public.txt
# chmod 440 private
# cd ..
# chown -R root:vchkpw yourdomain.com

The word “private” above is going to be our selector. It’s 100% arbitrary but must be used consistently. QmailToaster defaults to the word “private”. In your DNS, you need a TXT record that looks something like:

private._domainkey.yourdomain.com:k=rsa; p=MEwwDQYJKoZIhvcNAQEBBQ . . .

The rest of the record comes out of the file we just created with dknewkey in /var/qmail/control/domainkeys/yourdomain.com/public.txt. In my last Qmail install, I used “default” for my selector, so my DNS entries look like:

default._domainkey.yourdomain.com:k=rsa; p=MEwwDQYJKoZIhvcNAQEBBQ . . .

This selector, be it “private”, “default” or something else, MUST match what you use to generate your public/private key using dknewkey and it must also match what you use in your /etc/tcprules.d/tcp.smtp rules where you specify when to sign outgoing messages:

# example record using QmailToaster defaults
127.:allow,RELAYCLIENT="",DKSIGN="/var/qmail/control/domainkeys/%/private"

# since I used 'default' as my selector, this is what my file looks like:
127.:allow,RELAYCLIENT="",DKSIGN="/var/qmail/control/domainkeys/%/default"

Crypto Warning

dknewkey creates a (relatively) short 384-bit public/private keypair. There are people who don’t think that’s good enough and they’re probably right. You can edit /usr/bin/dknewkey to change it to something more secure like 512 or (better yet) 1024 bits.

A quick recap with how this selector is used throughout your Qmail and DNS configuration:

  1. Generate the public/private key: dknewkey my-selector-name > public.txt; this leaves behind a file like /var/qmail/control/domainkeys/yourdomain.com/my-selector-name.
  2. Create the DNS TXT record: my-selector-name._domainkey.yourdomain.com:k=rsa; p=MEwwDQYJKoZIhvcNAQEBBQ… (using the contents of the public.txt file dknewkey creates)
  3. In tcp.smtp, use a DKSIGN environment variable: DKSIGN=”/var/qmail/control/domainkeys/%/my-selector-name”

Testing

Once you have it set up and your DNS has propagated, there are a number of online services and email reflectors that will verify your configuration and report back to you:

Web Tools

Email Reflectors

Unlike the above services which try to see if things look OK, these are email addresses you can send a test message to that will actually tell you how your email performed:

  • [email protected] – the port25 reflector sends you an email report with a summary and breakdown by type. This is the best one in my opinion.
  • [email protected] – this message will be rejected but the rejection will say what happened to the SPF test. You must tail the logs (like /var/log/qmail/smtp/current) to see the results.
  • [email protected] – Eland Systems offers a DKIM reflector along with some notes
  • MyIpTest.com Tester – tests DK, DKIM and SPF. Web page gives you a random email address to send to and then you can load your results after sending a test message.
  • http://www.brandonchecketts.com/emailtest.php – similar to above, send an email to a random address provided on the web page and then you can check results.

Port25’s reflector report should look like this when everything is working:

Summary of Results
==========================================================
SPF check: pass
DomainKeys check: pass
DKIM check: pass
Sender-ID check: pass
SpamAssassin check: ham

Microsoft will let you “register” your Sender ID settings with them. Not sure how important this is, but it can’t hurt to let them know at https://support.msn.com/eform.aspx?productKey=senderid&page=support_senderid_options_form_byemail.

Email Services

Better than a reflector is to see how a real mail service will treat your message. Once you think you’re configured correctly, try sending an email to Gmail and Yahoo. Then you can view the headers for each to see how your SPF and DK records were interpreted.

Gmail:
Verify SPF, DomainKeys and DKIM headers on Gmail

Yahoo:
View SPF, DomainKeys and DKIM headers on Yahoo mail

Final Thoughts

Putting it all together, here’s how my setup looks (using the selector ‘default’):

Public/private key pairs on file system

[root@mail domainkeys]# ls -al /var/qmail/control/domainkeys/sample.com/
total 32
drwxr-xr-x 2 qmailq root  4096 Mar 13 13:46 .
drwxr-xr-x 7 root   qmail 4096 Mar 13 13:46 ..
-r--r----- 1 qmailq root   396 Mar 13 13:46 default
-rw-r--r-- 1 qmailq root   142 Mar 13 13:46 public.txt

DKIM signconf.xml

On the file system in /var/qmail/control/dkim/signconf.xml:

<dkimsign>
  <global algorithm="rsa-sha1" domain="/var/qmail/control/me" keyfile="/var/qmail/control/dkim/global.key" method="simple" selector="dkim1">
    <types id="dkim" />
  </global>
</dkimsign>

You could override the use of /var/qmail/control/me or change the selector from ‘dkim1′ if needed.

Alternative note: It is also possible to sign your DomainKeys using the DKIM package. You can change signconf.xml to have a per-domain record with DK and then abandon DKSIGN altogether (this is the route I’ve gone):

<sample.com domain="sample.com" selector="default">
  <types id="dkim" />
  <types id="domainkey" method="nofws" />
</sample.com>

tcp.smtp

# localhost relaying
127.:allow,RELAYCLIENT="",QMAILQUEUE="/var/qmail/bin/simscan",DKVERIFY="DEGIJKfh",DKQUEUE="/var/qmail/bin/qmail-queue.orig",DKSIGN="/var/qmail/control/domainkeys/%/default"
# external web box
69.36.226.12:allow,RELAYCLIENT="",DKSIGN="/var/qmail/control/domainkeys/%/default"
# everyone else that connects gets the full cavity check
:allow,BADMIMETYPE="",BADLOADERTYPE="M",CHKUSER_RCPTLIMIT="50",CHKUSER_WRONGRCPTLIMIT="3",QMAILQUEUE="/var/qmail/bin/simscan",DKVERIFY="DEGIJKfh",DKQUEUE="/var/qmail/bin/qmail-queue.orig",DKSIGN="/var/qmail/control/domainkeys/%/default"

DNS TXT Records

Using djbdns: 

'sample.com:v=spf1 ip4\07269.36.226.12 include\072emailcenterpro.com ~all:3600
'_domainkey.sample.com.:o=-;:3600
'default._domainkey.sample.com.:k=rsa; p=MHwwDQYJKoZIhvcNAQEBBQADawAwaAJhAPBdKl9mx3KIjLwGfq83LNzUN4aDWc4t3PVTnc0d+duI9HVD+UG+i1suifV6obDzo4ugHji6EH+zei39Cch9vTXcpVWkaFCEmVu0AXjY/98WIjYb5Hh5+SYQFyQkz0Mq0wIDAQAB;:3600
'dkim1._domainkey.mail.sample.com.:v=DKIM1; k=rsa; p=MEwwDQYJKoZIhvcNAQEBBQADOwAwOAIxAK+vhWHoPv+3DhM1u3MaHk7AayBa+CdNnjPHhg3/3Nv7hEIYUbOfKSUqNxDtjJkgcQIDAQAB;:3600
'_adsp._domainkey.mail.sample.com:dkim=all:3600
'_adsp._domainkey.sample.com:dkim=all:3600

First line is my SPF/Sender ID record. It also uses an include to allow a third party (in this case, EmailCenterPro) to send mail on our behalf.

The second line says that we sign all outgoing messages with DomainKeys/DKIM (o=-).

Third line is for DomainKeys using our selector ‘default’ (which correspondingly shows up in tcp.smtp in DKSIGN and on the filesystem as /var/qmail/control/domainkeys/sample.com/default).

The fourth line is the DKIM record which uses the global.key and public.txt stored in /var/qmail/control/dkim. Why is it dkim1._domainkey? Because that’s the arbitrary selector specified in the signconf.xml file as listed above. Again, totally arbitrary, could be ‘private’ or ‘default’ or something else, but best to have it different from your DomainKeys. You might also notice that it is mail.sample.com instead of sample.com? By default it must match /var/qmail/control/me file because that’s how the message will be signed by the QmailToaster DKIM package.

The fifth and sixth line are ADSP records that say we sign all outbound messages with DKIM. Some hosts appear to be checking for it without the mail subdomain so I’m currently running both. I don’t think it should be strictly required but at this time seems to cover the bases.

The 3600 on every line is the TTL, meaning these records expire after 1 hour.

Quarantining Spam in IMAP Folder

Before upgrading, we used DSPAM as our spam filter of choice. In a word, it’s awesome. It quarantined mail so effectively that I eventually stopped checking the quarantine until it became so large I could no longer view or delete it from the web interface. The secret sauce in my setup, which involves Mozilla Thunderbird was to use Thunderbird’s built-in “Junk” folder as a feedback loop for DSPAM. For some reason, Thunderbird is really good at catching certain kinds of spam that DSPAM was not and, if a spam did get through, I can select it and press “J” which trains Thunderbird and moves the message to my Junk folder. The server processed those Junk folders nightly, feeding them to DSPAM to further increase its accuracy.

I like not downloading all of the assumed junk mail to my phone or, worse yet, on a slow net connection while traveling somewhere around the world. Unfortunately, QmailToaster’s SpamAssassin approach is to flag but leave it in your inbox. No bueno. Luckily we can use some of the software already installed and a very short script to replicate the previous behavior.

Create generic mailfilter

If you accept users will use a “Junk” folder (or “Spam” or whatever you want to call it), then you can create a single maildrop mailfilter script in ~vpopmail/etc/mailfilter-to-spam:

SHELL="/bin/bash"
import EXT
import HOST

VUSER=`echo ${EXT%-*}`
export VUSER
USERHOME=`/mail/bin/vuserinfo -d $VUSER@$HOST`
export USERHOME

logfile "/var/log/maildrop/mailfilter.log"

# Test for the existance of the Junk directory.  Create it if it doesn't exist
#VERBOSE=9
#log "testing for $VUSER@$HOST in $USERHOME"

if ( /^X-Spam-Status: Yes/ )
{
        `test -d $USERHOME/Maildir/.Junk`

        if ( $RETURNCODE == 1 )
        {
            `/usr/bin/maildirmake -f Junk $USERHOME/Maildir`
            `echo INBOX.Junk >> $USERHOME/Maildir/courierimapsubscribed`
        }

        to "$USERHOME/Maildir/.Junk"
}
else
{
        to "$USERHOME/Maildir/"
}

Now create a ~vpopmail/domains/sample.com/username/.qmail file for those users who want their spam quarantined in a “Junk” folder rather than just tagged:

|preline /usr/bin/maildrop ~vpopmail/etc/mailfilter-to-spam

Done! All messages that meet your SpamAssassin minimum score (default of 5.0) will be quarantined in the “Junk” folder for review from SquirrelMail or your IMAP client. You can process these folders using a script like the following (warning, not tested in production yet!):

#!/bin/bash
cd ~vpopmail/domains

# find all Junk folders
for ii in `find . -type d -name '.Junk'`; do

  # find all mail 5 days or older
  for jj in `find $ii/cur/ -type f -mtime +4`; do

    # learn from it and delete it
    echo "processing $jj"
    /usr/bin/sa-learn --spam --no-sync $jj
    rm -f $jj

  done;

done;

# now synchronize the overall SA database
/usr/bin/sa-learn --sync

The argument “-mtime +4″ says look for files that are 5 days or older. You could adjust that to +1 for 48 hours or “-mmin +60″ for 60 minutes or older depending on your policies. Schedule this to run as a cronjob nightly and you’ll be provided an ongoing set of spam to learn from.

Hope these links and notes help someone else in the future!

1 Comment

  1. October Qmail Follow-up » ghidinelli.com said:

    on October 24, 2011 at 6:11 pm

    [...] notes that I don’t want to forget to follow-up on my March (Q)mail Server Madness post earlier this year. Things are running great with the exception of SpamAssassin being pretty [...]

{ RSS feed for comments on this post}