A huge proportion of the spam I receive is in non-English languages. I've often wondered why SpamAssassin has allowed waves of incoming messages to spill into my inbox consisting entirely of unreadable (at least for me) hieroglyphics when at other times it marks legitimate messages from my friends as spam. Almost all of my legitimate correspondence is in either English or Spanish. Very, very occasionally I get a customer inquiry written in French or German.
So I am prepared to make SpamAssassin get tough on non-English (and non-Spanish) messages at the expensive of risking a few very rare false positives; and such false positives are easily distinguished from spam messages with subjects like "-6月23-24日（深-圳）开-讲-" and "Автомобильные шины и диски! Низкие цены!", thanks to familiar words like "Synergy" in their subject lines.
So the first thing I did was add this to my ~/.spamassassin/user_prefs file:
# Mail using languages used in these country codes will not be marked # as being possibly spam in a foreign language. # - english spanish ok_languages en es
# Mail using locales used in these country codes will not be marked # as being possibly spam in a foreign language. # - en Western character sets in general ok_locales en
This change immediately made a big difference; notice how many points were derived from the non-English aspects of the first incoming spam message that I received after making the changes:
Content analysis details: (14.10 points, 5 required) UNDESIRED_LANGUAGE_BODY (4.0 points) BODY: Written in an undesired language CHARSET_FARAWAY (3.2 points) BODY: Character set indicates a foreign language BODY_8BITS (1.5 points) BODY: Body includes 8 consecutive 8-bit characters MSG_ID_ADDED_BY_MTA_3 (0.7 points) 'Message-Id' was added by a relay (3) CHARSET_FARAWAY_HEADERS (2.1 points) A foreign language charset used in headers DATE_IN_FUTURE_24_48 (2.6 points) Date: is 24 to 48 hours after Received: date
Compare this with a similar, obviously spam message received prior to making the changes and which easily slipped past SpamAssassin without being tagged:
X-Spam-Status: No, hits=1.6 required=5.0 tests=HTML_60_70,HTML_FONT_BIG,HTML_FONT_COLOR_BLUE, HTML_FONT_COLOR_RED,HTML_MESSAGE,MANY_EXCLAMATIONS, MIME_HTML_ONLY
These changes mean that foreign language spam will be caught much more often by SpamAssassin from here on, and being a user-level preference setting I can make the change without affecting other users on the same mail server.
The next thing I did was automatically divert messages that are very likely to be spam (those that score 15 or over) into a separate mailbox on the server. Messages that score between 5 and 15 continue to be tagged by SpamAssassin and I filter them on my local machine using SpamSieve.
This is done by creating a ~/.procmailrc file with the following contents:
# mail very likely to be spam (>= 15) quarantined in "spam" folder :0 * ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\* spam #/dev/null
Ideally, messages in the less-than-5 range will be legitimate and contain very few false negatives. Messages in the 15-and-greater range will hopefully be spam 100% of the time and if an extended trial supports this then I'll mark them for automatic deletion. The remaining messages (in the 5 to 15 range) should be mostly spam with very few false positives which I can manually check with a quick visual scan in Mail.app. The truth is that SpamSieve does such an excellent job (much better than SpamAssassin) that I don't use SpamAssassin's score as a basis for sorting my mail; I just let SpamSieve do its job and SpamAssassin's role is reduced to a simple message tagger.
Once I am convinced that the set-up works I'll change my .procmailrc file to this:
# mail very likely to be spam (>= 15) quarantined in "spam" folder :0 * ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\* #spam /dev/null
Now the only remaining issue for me with SpamAssassin is this bug, long ago fixed but probably never to be backported to the older version that comes with Red Hat Enterprise Linux ES 3. I find that I can't override the weighting of the defective FORGED_MUA_OUTLOOK rule because it would allow too much spam to slip under the radar; so the workaround is to instead use whitelist_from directives in my user preferences file to handle the few cases in which this bug actually causes legitimate email to be misclassified as spam.
So that auto-deleted messages don't just disappear into the ether without a trace you could add logging for them like this:
LOGFILE=$HOME/procmail.log VERBOSE=off LOGABSTRACT=all
# mail very likely to be spam (>= 15) piped directly to /dev/null :0 * ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\* /dev/null
# clean the environment before continuing LOGFILE= LOGABSTRACT= VERBOSE=
This will produce log messages like this in the procmail.log file for each auto-deleted spam:
From yuumi_basel0903@grate.half-time.info Tue Jun 27 10:24:03 2006 Subject: [SPAM] Folder: /dev/null 14890
Needless to say it is probably a good idea to check this log file frequently to be sure that legitimate messages and getting blown away.
The old version of SpamAssassin that comes with Red Hat Enterprise Linux ES 3 doesn't have the auto-whitelisting feature enabled by default. I've decided to turn it on to see if improves SpamAsssasin's abysmal accuracy. Overnight I received 213 spam messages; of these:
So I changed my procmail recipe from:
# Filter messages through SpamAssassin: # - use a lock file to reduce load # - only filter files less than 256KB in size (save CPU/memory) :0fw: spamassassin.lock * < 256000 | /usr/bin/spamassassin
To:
# Filter messages through SpamAssassin: # - use a lock file to reduce load # - only filter files less than 256KB in size (save CPU/memory) # - the "-a" switch enables the auto-whitelisting feature of SpamAssassin :0fw: spamassassin.lock * < 256000 | /usr/bin/spamassassin -a
In an effort to improve on SpamAssassin's poor accuracy (only 43% as accurate as SpamSieve even with auto-whitelisting turned on) I decided to do some Bayesian training. It is necessary to train the filter using both spam and ham (non-spam) messages.
I had 1558 recent spams on hand that had been sent to my business account, and 547 sent to my personal account. I moved these into a new IMAP mailbox called "train-spam" in each account.
I then grabbed the 1558 most recent ham messages for my business account using Mail.app's powerful search facility (which allows you to select multiple mailboxes and sort search results by date). I did the same to locate 547 ham messages sent to my personal account. I copied (not moved) these messages to a new IMAP mailbox called "train-ham" in each account. Copying can be achieved in Mail.app just like in the Finder by holding down the Option key while dragging.
My first attempts at using sa-learn on the server were problematic because the training has to be run as the user to whom the account belongs, but on my server such accounts do not have shell access. I tried using sudo but this didn't work:
$ sudo -u john%example.com sa-learn --showdots --spam --mbox /home/john%example.com/mail/train-spam Password: [enter password of admin account] Failed to create default user preference file /home/xxxxxx/.spamassassin/user_prefs lock: 12345 cannot create tmp lockfile /home/xxxxxx/.spamassassin/bayes.lock.x.12345 for /home/xxxxxx/.spamassassin/bayes.lock: Permission denied
As you can see this allowed the command to run as the correct user but it still tried writing to files in another home directory. I then tried using su but that didn't work because the email accounts do not have login shells.
$ su john%example.com ls Password: [enter password of email account] This account is currently not available.
My first workaround was to temporarily enable login shells for those accounts and su to them before training, but I then went in search of a better solution. It turns out that the -H switch to sudo was what was needed; this sets the HOME environment variable before executing the command:
$ sudo -u john%example.com -H sa-learn --showdots --spam --mbox /home/john%example.com/mail/train-spam
There was still one problem to solve. Training using this command produced lots of messages about problems untainting paths:
security: cannot untaint path: "/home/john%example.com/.spamassassin" security: cannot untaint path: "/home/john%example.com/.spamassassin/user_prefs" security: cannot untaint path: "/home/john%example.com/.spamassassin" security: cannot untaint path: "/home/john%example.com/.spamassassin/bayes" security: cannot untaint path: "/home/john%example.com/.spamassassin" security: cannot untaint path: "/home/john%example.com/.spamassassin/bayes"
I filed a bug report with the SpamAssassin team and it looks like the problem will be fixed for version 3.2.0 of SpamAssassin. I also made the necessary changes on my local install.
My plan is to use the initial training run as a base and from here on do mistake based training. This means feeding only false positives and false negatives back to SpamAssassin back for training. My reading indicates that mistake based training is the best in the long run. Here is a short script I threw together to automate future runs of the training process:
#!/bin/sh
USERS="john%example.com barry%example.com"
echo "Starting SpamAssassin training run: will use sudo to run sa-learn as appropriate user(s)." sudo -v
for USER in ${USERS} do echo "Spam training for user ${USER}:" sudo -u "${USER}" -H sa-learn --showdots --spam --mbox "/home/${USER}/mail/train-spam" echo "Ham training for user ${USER}:" sudo -u "${USER}" -H sa-learn --showdots --ham --mbox "/home/${USER}/mail/train-ham" done
echo "Training run complete: you should now empty the train-spam and train-ham folders for these users:" echo " ${USERS}"
So after training for a few days accuracy is greatly improved but there is still a long way to go before SpamAssassin catches up with SpamSieve. It seems that without Bayesian training, SpamAssassin is next to useless. We'll see how accurate it can become after more training.
Last night in an approximately 12 hour period I obtained the following results:
So, I will continue to train SpamAssassin on its errors, but it seems unlikely that it will ever catch up with SpamSieve. Seems like I'm approaching the limit of possible accuracy using the SpamSieve/SpamAssassin pair, so I have a few more steps in mind that I can use to reduce the amount of spam that gets to the filters in the first place... watch this space for more details.
]]>/etc/cron.hourly/freshclam:
ERROR: Problem with internal logger.
I found the explanation here: that there was a hung freshclam process lingering on the system.
]]> The solution? Kill the process and run freshclam manually (slightly edited to shorten long lines):$ ps auxww | grep fresh root 13372 0.0 0.0 2132 880 ? S Jun09 0:00 /bin/sh /etc/cron.hourly/freshclam clamav 13373 0.0 0.0 2424 896 ? S Jun09 0:00 /usr/local/bin/freshclam --quiet root 13374 0.0 0.0 1948 596 ? S Jun09 0:00 awk -v progname=/etc/cron.hourly/freshclam obfuscat 23287 0.0 0.0 4772 676 pts/0 S 08:14 0:00 grep fresh $ sudo kill 13372 $ ps auxww | grep fresh clamav 13373 0.0 0.0 2424 896 ? S Jun09 0:00 /usr/local/bin/freshclam --quiet root 13374 0.0 0.0 1948 596 ? S Jun09 0:00 awk -v progname=/etc/cron.hourly/freshclam obfuscat 23299 0.0 0.0 4752 672 pts/0 S 08:15 0:00 grep fresh $ sudo kill 13373 $ ps auxww | grep fresh obfuscat 23306 0.0 0.0 4752 672 pts/0 S 08:15 0:00 grep fresh $ sudo /usr/local/bin/freshclam ClamAV update process started at Sat Jun 10 08:15:34 2006 Downloading main.cvd [*] main.cvd updated (version: 39, sigs: 58116, f-level: 8, builder: tkojm) Downloading daily.cvd [*] daily.cvd updated (version: 1524, sigs: 903, f-level: 8, builder: tkojm) Database updated (59019 signatures) from db.us.clamav.net (IP: 63.236.138.5) Clamd successfully notified about the update.
This is what I saw in the /var/log/freshclam.log file; firstly, the last successful update:
-------------------------------------- ClamAV update process started at Fri Jun 9 15:01:01 2006 main.cvd is up to date (version: 38, sigs: 51206, f-level: 7, builder: tkojm) daily.cvd is up to date (version: 1523, sigs: 7866, f-level: 8, builder: sven)
Then problems accessing the mirrors:
-------------------------------------- ClamAV update process started at Fri Jun 9 16:01:01 2006 ERROR: Mirrors are not fully synchronized. Please try again later. Trying again in 5 secs... ClamAV update process started at Fri Jun 9 16:02:12 2006 ERROR: Mirrors are not fully synchronized. Please try again later. Trying again in 5 secs... ClamAV update process started at Fri Jun 9 16:03:08 2006 ERROR: Error while reading database from db.us.clamav.net ERROR: Can't download main.cvd from db.us.clamav.net (IP: 216.24.174.245) Giving up on db.us.clamav.net... ClamAV update process started at Fri Jun 9 16:12:41 2006 ERROR: Mirrors are not fully synchronized. Please try again later. Trying again in 5 secs... ClamAV update process started at Fri Jun 9 16:13:36 2006 ERROR: Verification: Broken or not a CVD file Trying again in 5 secs... ClamAV update process started at Fri Jun 9 16:13:44 2006 ERROR: Error while reading database from database.clamav.net ERROR: Can't download main.cvd from database.clamav.net (IP: 63.166.28.8) Giving up on database.clamav.net... ERROR: Update failed. Your network may be down or none of the mirrors listed in freshclam.conf is working. ERROR: Update failed. Your network may be down or none of the mirrors listed in freshclam.conf is working.
And on the next hourly run, the hung process:
-------------------------------------- ClamAV update process started at Fri Jun 9 17:01:01 2006
Finally, the manual update performed after killing the hung process:
-------------------------------------- ClamAV update process started at Sat Jun 10 08:15:34 2006 main.cvd updated (version: 39, sigs: 58116, f-level: 8, builder: tkojm) daily.cvd updated (version: 1524, sigs: 903, f-level: 8, builder: tkojm) Database updated (59019 signatures) from db.us.clamav.net (IP: 63.236.138.5) Clamd successfully notified about the update.]]>
<FilesMatch ^wp-config.php$> deny from all </FilesMatch>
This will make it harder for your database username and password to fall into the wrong hands in the event of a server problem. Better still would be to move the username and password outside of the document root entirely and include or require it but that would require you to turn off PHP Safe Mode and/or the open_basedir restriction, which I don't recommend with a product like WordPress which is both popular and has a less-than-perfect security record.
]]>sudo mkdir wp cd wp sudo svn co http://svn.automattic.com/wordpress/branches/2.0/ .
Then it's time to do a diff to see the differences between the freshly-checked-out copy of WordPress 2.0.3 and the installed, customized copy of 2.0.2 already on the server. It's necessary to manually merge in these changes, but only on this initial switch to Subversion. Once the first merge is done, future upgrades are as easy as running svn up.
diff -r path-to-checked-out-copy path-to-installed-copy
Concretely, in my case this meant that I had to use the following paths:
diff -r wp public_html/wp
Given that most of my existing WordPress install was not customized I found that it was more useful to do a non-recursive diff and then examine the appropriate subdirectories manually:
diff wp public_html/wp
The diff output immediately indicated to me a couple of missing files that I would need to copy over from my old install to the new copy:
sudo cp public_html/wp/.htaccess wp/ sudo cp public_html/wp/wp-config.php wp/
There were a couple of common subdirectories that I could skip over and exclude from consideration because I've applied no customizations to the WordPress code itself:
Common subdirectories: wp/wp-admin and public_html/wp/wp-admin Common subdirectories: wp/wp-includes and public_html/wp/wp-includes
Likewise there was a wp-images folder in the old install that is evidently not present in the latest version of WordPress, so I didn't bother to copy it over:
Only in public_html/wp: wp-images
The key directory that I needed to examine carefully was the wp-content directory:
Common subdirectories: wp/wp-content and public_html/wp/wp-content
I did a non-recursive diff on that too:
diff wp/wp-content public_html/wp/wp-content
There were a few directories that needed to be copied over:
Only in public_html/wp/wp-content: backup-xxxxx Only in public_html/wp/wp-content: cache Only in public_html/wp/wp-content: uploads
These were copied as follows:
sudo cp -R public_html/wp/wp-content/backup-xxxxx wp/wp-content/ sudo cp -R public_html/wp/wp-content/cache wp/wp-content/ sudo cp -R public_html/wp/wp-content/uploads wp/wp-content/
There were a couple of common subdirectories that needed closer attention:
Common subdirectories: wp/wp-content/plugins and public_html/wp/wp-content/plugins Common subdirectories: wp/wp-content/themes and public_html/wp/wp-content/themes
So, another non-recursive diff:
diff wp/wp-content/plugins public_html/wp/wp-content/plugins
This was really the only subdirectory that I cared about:
Only in public_html/wp/wp-content/plugins: text-control
And I copied it thusly:
sudo cp -R public_html/wp/wp-content/plugins/text-control wp/wp-content/plugins/
Likewise I did a non-recursive diff of the themes subdirectory:
diff wp/wp-content/themes public_html/wp/wp-content/themes
Once again only one subdirectory to worry about:
Only in public_html/wp/wp-content/themes: hemingway
Copied like this:
sudo cp -R public_html/wp/wp-content/themes/hemingway wp/wp-content/themes/
It was then time to set the permissions and ownership on the new install (especially important seeing as I am running under PHP safe mode which is much more sensitive to permissions):
cd wp sudo chown -R user:user . cd wp-content sudo chmod 777 backup-xxxxx cache sudo chown apache:apache backup-xxxxx/index.php sudo chmod -R 777 uploads/* cd cache sudo chown -R apache:apache *
It was then time to make the actual switch from 2.0.2 to 2.0.3 on the live site. This consisted of five steps:
And that's all there is to it. To see differences between the installed version of WordPress and the last version that was checked out it's a simple matter of doing an svn status from inside the installation directory. On my installation I see the following output:
? wp-config.php ? .htaccess ? wp-content/cache ? wp-content/backup-xxxxx ? wp-content/uploads ? wp-content/plugins/text-control ? wp-content/plugins/markdown.php X wp-content/plugins/akismet ? wp-content/themes/hemingway
Performing status on external item at 'wp-content/plugins/akismet'
As you can see, the changes correspond to those that I made during my manual merge. For more information on the svn status command type svn help status. For (much) more detail on the differences you can do an svn diff.
Now, when the next version of WordPress comes out, all you have to do is run an sudo svn update. Given the sensitivity of PHP Safe Mode you should also make sure no new root-owned files have been created during the update:
sudo svn update find . -user root
Any root-owned files will need to be appropriately modified using commands like the following.
sudo chown -R user:user directory sudo chown user:user file]]>
There were a number of command-line tools that I depend on that did not get transferred by the migration assistant. These are the notes that I made while compiling native Intel versions of these tools and installing them on the iMac.
]]> wgetI prefer wget to curl. The --limit-rate switch below is so that downloading the source didn't flood my pathetic dial-up connection:
# get the source curl --limit-rate 1K -O ftp://ftp.rediris.es/pub/gnu/gnu/wget/wget-1.10.tar.gz
# extract tar xzvf wget-1.10.tar.gz cd wget-1.10/
# configure and build ./configure --with-ssl make
# test src/wget --version
# install sudo make install
When I went looking for the latest official version of pstree it seems that the website has packed up and gone forever. Luckily I still had a copy of a source archive for a recent version (2.27), possibly the latest. Seeing as it's redistributable under the GPL I have placed a copy here. To build it, you would follow a procedure like this one:
# get the source wget http://wincent.com/gpl/pstree-2.27.tar.gz
# extract tar zxvf pstree-2.27.tar.gz cd pstree
# build gcc -O -o pstree pstree.c
# install sudo install pstree /usr/local/bin/pstree
Or a slight variation of this; building a Universal Binary by combining a previously built PowerPC executable, pstree, with a newly built Intel executable:
mv pstree pstree.ppc
# build gcc -O -o pstree.i386 pstree.c
# make a Universal Binary lipo -arch ppc pstree.ppc -arch i386 pstree.i386 -output pstree -create file pstree
# install sudo install pstree /usr/local/bin/pstree
# get the source wget "http://ovh.dl.sourceforge.net/sourceforge/pwgen/pwgen-2.05.tar.gz"
# extract tar xzvf pwgen-2.05.tar.gz cd pwgen-2.05
# configure and build ./configure make
# test ./pwgen
# install sudo make install
# get the source wget http://www.users.qwest.net/~hmahon/sources/ee-1.4.2.src.tgz
# extract tar xzvf ee-1.4.2.src.tgz cd easyedit/
# build make
# install sudo make install
# get the source wget http://www.users.qwest.net/~hmahon/sources/aee-2.2.3.tar.gz
# extract tar zxvf aee-2.2.3.tar.gz cd aee-2.2.3/
# make both aee (normal) and xae (for use in X11) make both
# install sudo make install
# get the source wget http://apache.gva.es/httpd/httpd-2.0.58.tar.bz2
# extract tar xjvf httpd-2.0.58.tar.bz2 cd httpd-2.0.58/
# workaround APR bug (see Subversion ac_cv_func_poll=no; export ac_cv_func_poll
# configure and build ./configure --prefix=/usr/local/apache2 --enable-dav --enable-so --enable-ssl --enable-deflate make
# install sudo make install
# get the source wget http://subversion.tigris.org/downloads/subversion-1.3.2.tar.bz2
# extract tar xjvf subversion-1.3.2.tar.bz2 cd subversion-1.3.2/
# set-up for access via Apache sudo mkdir -p /usr/local/apache2/var/dav echo 'svn::200:200::0:0:Subversion:/var/empty:/usr/bin/false' | sudo niload -v passwd / echo 'svn:*:200:svn' | sudo niload -v group / sudo chown svn:svn /usr/local/apache2/var/dav
# workaround APR bug (see Subversion FAQ) ac_cv_func_poll=no; export ac_cv_func_poll
# configure and build sh ./autogen.sh ./configure --with-apxs=/usr/local/apache2/bin/apxs make
# test make check
# install sudo make install
It's then necessary to make some changes to /usr/local/apache2/conf/httpd.conf so it will work with Subversion:
219c219 < Listen 80 --- > Listen 8080 233a234,236 > LoadModule dav_svn_module modules/mod_dav_svn.so > LoadModule authz_svn_module modules/mod_authz_svn.so > 267,268c270,271 < User nobody < Group #-1 --- > User svn > Group svn 358,359c361,362 < Order allow,deny < Allow from all --- > Order deny,allow > Deny from all 994a998,1007 > <Location /svnrep> > DAV svn > SVNParentPath /Users/wincent/Developer/svnrep > AuthType Basic > AuthName "Subversion repository" > AuthUserFile /Users/wincent/Developer/svnrep-auth > Require valid-user > Order deny,allow > Allow from all > </Location>
Finally we can start up Apache:
sudo /usr/local/apache2/bin/apachectl start
I found it was unnecessary to set up a start-up item to launch Apache 2 at boot time because it was correctly transferred across by the migration assistant. I did, however, need to add this line back into /etc/hostconfig:
WEBSERVER2=-YES-]]>
And it would be even easier if it could be handled by up2date like the rest of the system. Human intervention is only required in this case because RHEL ES 3 only ships with MySQL 3.23 by default. If you want the features of the 4.1 series you have to manually upgrade.
]]>Turns out that there was one which was occasionally available, but with a very weak signal and I could only ever get a pitiful few bytes per second over it.
When I first found this network I foolishly decided to check my email despite the fact that it uses inherently insecure protocols which transmit usernames and passwords in plaintext. I realized that the person running this open network could easily snoop them, so I decided to log into my server securely via SSH, immediately change the email passwords that had been transmitted as plaintext, and not try checking my email again until I had worked out a way to secure the protocols.
]]> It turns out that the open network never gave me enough performance to actually make checking my email possible, but at least it prompted me to tighten things up. There are three principal ways to secure email traffic: SSH tunneling; using secure protocols like IMAPS, POP3S and SMTP with SSL; and using the stunnel tool. None of these provide end-to-end encryption for the content of your emails — you'll need to encrypt them before sending them in order to get that — but they do secure the connection between your machine and your local mailserver, thus protecting your passwords and preventing them from being snooped by a casual attacker.This is the easiest of the three methods, but it requires you to have an SSH account on your mailserver. Basically all you have to do is use the -L switch to ssh to tunnel the appropriate ports. You would execute a command like this on your local machine:
sudo ssh -L 143:wincent.com:143 \
-L 110:wincent.com:110 \
-L 25:wincent.com:25 \
robinson@wincent.com
The sudo is necessary because root privileges are required to open any ports below 1024. The port numbers above are the standard port numbers for IMAP, POP3 and SMTP respectively. If you don't want to run the ssh process with root privileges you'll need to choose local port numbers that are higher than 1024; for example, you could just add 10000 to each of the numbers:
sudo ssh -L 10143:wincent.com:143 \
-L 10110:wincent.com:110 \
-L 10025:wincent.com:25 \
robinson@wincent.com
These commands basically say, "Log in via SSH to my account, robinson on the host wincent.com; forward any network traffic that's sent to the local ports on this machine to the remote ports on the host". The traffic is encrypted over the SSH connection. (Needless to say, the account "robinson@wincent.com" doesn't actually exist; I'm just using it for illustration purposes. I guess it'll only be a question of a few weeks before I start seeing spam directed to that non-existent address...)
You then need to change the settings in your local email client. Instead of connecting to wincent.com port 143, it will need to connect to localhost port 143 (or port 10143 if you followed the second example above). The forwarding is transparent and the email client doesn't need to know anything more; to the email client it looks just like a normal, plaintext IMAP, POP3 or SMTP connection.
The advantages of this method are that it is easy to set up and requires no modifications to the server. The downside is that it requires each email user to have an SSH account on the server and you have to make sure SSH is up and running with port forwarding activated before firing up your email client.
See the ssh man page for more information about SSH tunneling.
I didn't actually try using this method but it seems like a viable option. Basically, the stunnel process serves as a wrapper for the insecure protocol. For example, you fire up stunnel on the server and get it to listen to port 993 (the standard port for IMAPS), and it forwards incoming, encrypted connections to the real IMAP process running on the server (not encrypted). You then set up your local mail client to communicate with SSL to the appropriate ports.
I chose not to use this method because I didn't want to have to meddle too much with the server configuration (for example, configuring the stunnel-wrapped servers to automatically start up at the correct runlevels). Furthermore, it is not clear to me whether it is possible to offer both secure and insecure access at the same time (which would be useful, for example, if you wanted to allow users to access via either IMAP or IMAPS during the transition period).
See the stunnel man page on your server (if it has one) for more information.
I later went ahead to explore the use of Stunnel. It co-exists peacefully with the STARTTLS solution.
In the end this is the way I chose to go. It doesn't require users to have SSH accounts on the server, and it allows you to run both IMAPS and IMAP, POP3S and POP3, at the same time during a period of transition.
Basically, you set up an IMAPS server listening on port 993, a POP3S server running on port 995, and an SMTP server on port 25 which is capable of providing encrypted transport using TLS.
It is the most difficult to set-up, but also the most architecturally elegant. On the whole, I think it's worth the extra effort. The following summary applies to Red Hat Enterprise Linux ES 3.
I already had an SSL certificate for use with Apache and my server running at secure.wincent.com (actually the same machine as the mail server). I wasn't interested in using a self-signed certificate (self-signed certificates are free but not trusted by connecting clients; they do serve to encrypt your mail transport but they cause unsightly warning messages that don't exactly inspire trust). I was curious to know whether the same certificate could be used for email transport. The answer is yes.
I needed to assemble three components for this to work:
The order of these components is important. All three must be pasted, in order, into a file at /usr/share/ssl/certs/imapd.pem. The file should be owned by root and readable only by root.
The need to include the signing certificate authority's certificate is explained here:
Now for a small caveat - this all assumes your certificate was signed by a root certificate authority. In some cases, the CA which signed your certificate is not a root CA, but is a CA signed by a CA (or signed by a CA who was signed by a CA who is a root CA, etc.) This is often known as a chained certificate, or a ca-bundle.
What makes things tricky is that the remote client will look at your certificate, and try to verify it against the root CAs it knows about. If there is an intermediate CA between you and the CA the client knows about, it will need this certificate to sucessfully verify your certificate. As such, the server needs to not only provide clients with its own certificate, but also those of the intermediate CAs.
In the UW-IMAP server, this is achieved by appending all intermediate certificates to the file containing your own certificate, with the highest-level certificate last. (The root certificate is not required, as the client already has it.)
You can confirm that the certificate is valid and verifiable by running a command like this:
sudo openssl verify imapd.pem
If it is valid and the certificate chain can be successfully resolved you'll see a message like this:
imapd.pem: OK
Otherwise you'll see something like this:
imapd.pem: /C=GB/O=Comodo Limited/OU=Comodo Trust Network/OU=Terms and Conditions of use: http://www.comodo.net/repository/OU=(c)2002 Comodo Limited/CN=Comodo Class 3 Security Services CA error 2 at 1 depth lookup:unable to get issuer certificate
If you get the order of the certificates in the file wrong then you'll see messages like this in your /var/log/maillog when you try to connect to your IMAPS server:
Unable to load private key from /usr/share/ssl/certs/imapd.pem
These errors are evidently caused because the ordering confuses the server and it is no longer sure to which certificate the private key applies to, and it ends up guessing incorrectly. As stated in this page:
If you get the error messages above, chances are the key and certificate do not match. Make sure you aren't using the default server.key file. You should also check the httpd.conf file to make sure that the directives are pointing to the correct private key and certificate.
You can check to make sure that you your private key and certificate are in the correct format and match each other. To do this, give the commands below to decrypt the private key in one terminal window and decrypt the certificate in the other. What you will be comparing are the Modulus and the Exponent of each key. If the modulus and exponent from the key matches the set from the certificate, you have just confirmed that your certificate and key are correctly paired.
After checking the modulus and exponent of my key and certificate I was sure that I was using the right components, so I tried changing the order of elements in the file and it started working.
You can use the same file for POP3S:
touch ipop3d.pem chmod 400 ipop3d.pem cp imapd.pem ipop3d.pem
For secure SMTP you can use the same file as a base, but you'll need to make some alterations:
touch sendmail.pem chmod 400 sendmail.pem cp imapd.pem sendmail.pem
Specifically, you'll need to prepare a separate file (call it ca.crt) containing only the certificate authority's certificate. You can then remove that portion from the sendmail.pem file. Sendmail will look for the certificate authority certificate in a separate file.
Red Hat Enterprise Linux already has built-in IMAPS support. Now that your certificate is in place all you have to do is turn it on:
sudo chkconfig imaps on
You can verify that both IMAP and IMAPS are turned on (via xinetd) using:
sudo chkconfig --list
The same is true for POP3S. Support is already in place; you just have to turn it on:
sudo chkconfig pop3s on
You can verify that both the pop3s and ipop3 services are turned on (via xinetd) using:
sudo chkconfig --list
Unlike IMAPS and POP3S, adding TLS support to SMTP requires a bit more work. First of all, you must make sure that your domain name MX records are updated to point to the correct host name (in this case, secure.wincent.com). This is because third parties connecting to your mail server may issue a STARTTLS command and it is important that the host name matches the name in the certificate. And being an MX record, you have to make sure that your host name is a proper A record (pointing to an IP address) and not just a CNAME alias (pointing to a domain name).
The basic sequence of events looks like this:
The items you'll need to add to your Sendmail configuration are the following:
define(`confCACERT_PATH',`/usr/share/ssl/certs')dnl define(`confCACERT',`/usr/share/ssl/certs/ca.crt')dnl define(`confSERVER_CERT',`/usr/share/ssl/certs/sendmail.pem')dnl define(`confSERVER_KEY',`/usr/share/ssl/certs/sendmail.pem')dnl
To test that STARTTLS is correctly offered by the mail server you can do a telnet secure.wincent.com 25 and see if it lists STARTTLS in response to ehlo secure.wincent.com. When testing mail delivery it is a good idea to watch the mail log with sudo tail -f /var/log/maillog to make sure that SSL is being applied when you expect.
For more information about STARTTLS see these documents from the Sendmail website, as well as the text for RFC 2487.
Now you just need to point your email client to the new host name (in my case, secure.wincent.com), the new ports (993 and 995), and turn on SSL support for SMTP (no change to the port). It is important to change the host name as well as the ports because the certificate is tied to the host name.
]]>$ wget http://subversion.tigris.org/downloads/subversion-1.3.1.tar.bz2 $ tar xjf subversion-1.3.1.tar.bz2 $ cd subversion-1.3.1 $ sudo /usr/local/apache2/bin/apachectl stop $ sudo rm -f /usr/local/lib/libsvn* $ sudo rm -f /usr/local/lib/libapr* $ sudo rm -f /usr/local/lib/libexpat* $ sudo rm -f /usr/local/lib/libneon* $ ac_cv_func_poll=no; export ac_cv_func_poll $ sh ./autogen.sh && ./configure && make && make check $ sudo make install $ sudo /usr/local/apache2/bin/apachectl start $ svn --version
$ wget http://subversion.tigris.org/downloads/subversion-1.3.1.tar.bz2 $ tar xjf subversion-1.3.1.tar.bz2 $ cd subversion-1.3.1 $ sh ./autogen.sh && ./configure && make && make check $ sudo chkconfig svn off $ sudo make install $ sudo chkconfig svn on $ sudo netstat -anp | grep LISTEN | grep 3690 $ telnet localhost 3690]]>
Basically, headers like the following were being added:
X-Authentication-Warning: s69819.wincent.com: apache set sender to user@example.com using -f
In order to overcome this you need to have the "trusted users" feature enabled in Sendmail (I believe it is by default in Red Hat Enterprise Linux):
FEATURE(use_ct_file)dnl
Then all that has to be done is to add "apache" to /etc/mail/trusted-users and rebuild the Sendmail configuration (if you changed it). I also stopped and restarted Sendmail for good measure; not sure if that's required.
]]>@protocol MyProtocol
- method;
@end
@interface MySuperclass
- method;
@end
@interface MySubclass : MySuperclass <MyProtocol>
@end
GCC will generate a warning at compile time that MyClass seems to be missing a method implementation required by MyProtocol. You can suppress these warnings using the -Wno-protocol switch; from the GCC man page:
-Wno-protocol If a class is declared to implement a protocol, a warning is issued for every method in the protocol that is not implemented by the class. The default behavior is to issue a warning for every method not explicitly implemented in the class, even if a method implemen- tation is inherited from the superclass. If you use the -Wno-pro- tocol option, then methods inherited from the superclass are con- sidered to be implemented, and no warning is issued for them.
There are reasons why you might actually want to see these warnings which I won't go into here (for more info check the objc-language mailing list archives), but if you're sure that you want to be rid of them, then the -Wno-protocol switch is for you. It certainly beats declaring a bunch of methods like this in your subclass:
@implementation MySubclass
- method { return [super method]; }
@end
And in many cases it is better than the alternative of dumping the formal protocol and opting for an informal one (defined as a category in a parent or root class) instead.
]]>If you have a low-level method for adding numbers which is used by a high-level method for calculating window geometry then you should write tests for the low-level method and tests for the high-level method. In this way it will be easier to isolate faults. If a flaw in the low-level method causes the high-level method to fail you'll see failures in the tests for both methods. If the low-level method passes all the tests but the high-level method is broken you'll know where to direct your attention (to the high-level method).
In practice this means writing tests for every method of every class. It may sound like a lot of work but you'll appreciate the thoroughness of your tests if you later have to find a bug in a huge framework. Don't skip writing tests for simple methods that don't seem to need it; even the simplest-looking code can have bugs lurking in it.
One side consequence of the decision to write units tests for every method of every class is that it makes sense to group your unit tests using the same class/method hierarchy: that is, all tests for a given class will be grouped together into a test class, and all tests for a given method will be grouped into a method in that test class. You can of course add other, higher-level tests to the test class: these higher-level tests don't just test that a single method works as expected; rather they test more complex interactions between methods, classes (and sometimes mocks). I tend not to worry too much about keeping my unit tests totally isolated from one another (ideally you would test each class in total isolation from others, using mock objects as stand-ins); in practice keeping the tests "fairly isolated" is enough.
The highest level tests of all are sometimes called "acceptance tests". This basically means that you test that the program does what the user expects it to. Unit tests are lower-level, "atomic" tests that test the components (units) of the program. Acceptance tests are used to check the behavior of the program at the level at which the user interacts with it. They're called "acceptance" tests because they indicate whether the program will be acceptable to the user or not; that is, if it does what the user wants it to do. Unit tests are for programmers; acceptance tests are for users. In practice you write both kinds of tests using the same kinds of tools.
Sometimes you write code that takes advantage of some undocumented aspect of the Cocoa frameworks or the Objective-C runtime. I'm not talking here about using private or undocumented APIs; I'm referring to those places where the official documentation sometimes leaves things unsaid and taken for granted.
In those cases you end up making assumptions about the operation and characteristics of the context in which your program runs: assumptions about the operating system, the APIs, the environment.
Don't let these assumptions go untested. Write unit tests whenever your code rests upon an assumption you've made about something that isn't affirmed with 100% clarity in the official documentation. Use unit tests to eliminate potential ambiguity or uncertainty about the context in which your code is running.
For me writing a working program consists of three simultaneous activities:
Note that writing code is only one of the three parts of making a working program. You write the code so that the machine has a set of instructions to follow. You write the unit tests to ensure that the code works as you think it does and to give you added confidence when it comes to working quickly and making big changes. You write the documentation — even for closed-source code that nobody else will ever see — for two reasons: firstly, because it will be helpful to you if you ever have to revisit that same code months or years later; secondly, because it forces you to think about coming up with better designs that are more programmer-friendly (if you have trouble writing documentation for something it is a sure sign that you could have implemented things in a better, simpler way).
So you write unit tests based on that documentation. For example, if your documentation says, "raises an exception when passed nil", you write a test that confirms that an exception is raised when you pass nil. If you documentation says, "returns NO if the receiver does not recognize the passed selector", you write a test that passes an unrecognized selector and checks to make sure that NO is returned. These unit tests confirm that your code conforms to the documentation.
You'll then want to go though you code and look for every instance where you make an assertion, where an error can occur, or an exception can be thrown. You write unit tests for each of these places where the code can go off the rails. Basically you want to make sure that it goes off the rails when you expect it to. Use tests to ensure that assertions are correctly raised, that errors are reported, and that exceptions are thrown when appropriate.
This is probably the most obvious basis for writing unit tests. You write tests that confirm that your methods return the expected results. Let's say you have a method that adds two numbers. You'll want to write tests that confirm that your method correctly returns that "2 plus 2 equals 4". There are three types of test to write here:
In the first type of test you expect success. You check that "2 plus 2" does indeed equal "4".
In the second type of test you intentionally provoke a failure. For example, you provoke an overflow and check that the result is invalid.
In the third type of test you pick the borderline between success and failure — that is, conditions that approach as close as possible to "failure" without actually getting there — and make sure that the actual behavior is as expected. For example, in code that loops through items in an array you will want to ensure that the code works properly with empty arrays (0 items) or single-item arrays. With code that works with strings you'll want to check that nil strings and empty strings are handled. With code that operates with buffers you'll want to check that things work correctly when the buffer is one-byte away from full, totally full, or not big enough (the failure case).
What this generally means is that I almost never write a WO_TEST_TRUE test without pairing it with an inverse WO_TEST_FALSE case. Basically the pair of tests says, "I expect this to work but this to fail". Just because a method works when I expect it to work doesn't mean that it's correct; it must also not work when I expect it to not work. And by testing for boundary conditions as well as straightforward success and failure cases I can have more confidence that the method will work in all cases where I believe it should work and fail in all cases where I believe it should fail.
Each time you find a bug you should write a test that exposes it. In other words, you write a test that fails because of the bug, but which would pass if the bug didn't exist. You then fix the bug and the test passes. If the bug ever comes back your test will catch it straight away.
]]>// TODO: add user preference for opacity
If I later want to find all of the TODO items I can use Xcode's "Find In Project..." functionality or I can grep from the command-line. For this purpose I have the following function defined in my ~/.bash_profile:
]]> # grep for "TODO" string todo() { if [ $# -lt 1 ]; then grep -R -n "TODO: " . | grep -v ".svn" else # loop through the args while [ -n "$1" ] do grep -R -n "TODO: " "$1" | grep -v ".svn" shift done fi }I can then use the function to grep for TODO items in the current directory, in a specific file (or files), or in a specific directory (or directories):
$ todo $ todo . $ todo WOSynergyAdvanceController.m $ todo SynergyAdvanceFramework synergyd
I'll then see a list of matching file names, line numbers, and the TODO items themselves. Items in .svn directories are filtered from the output. If no arguments are supplied then the function starts the grep in the current directory. The output will look like this:
./WOITunesController.m:1683: // TODO: add caching for this value
So that's lightweight issue tracking. You can do the same with bugs using a function like this:
# grep for "BUG" string
bugs()
{
if [ $# -lt 1 ]; then
grep -R -n "BUG: " . | grep -v ".svn"
else
# loop through the args
while [ -n "$1" ]
do
grep -R -n "BUG: " "$1" | grep -v ".svn"
shift
done
fi
}]]>