Click here for the tutorial.

craigslist has implemented Haraka SMTP Server, based on its compelling combination of scalability and configurability.

craigslist classifieds are used by over 50 million users each month in the US alone, generating massive demand for its email relay services – demand that Haraka is designed to accomodate with ease.

Haraka plugins written in Javascript can access any part of the SMTP conversation, for email filtering, sender and recipient filtering and modification, authentication, content modification, and custom delivery. By leveraging the Node.js framework, the Haraka project provides extremely high mail server performance along with ease of use and flexibility.

craigslist has contributed significantly to Haraka development, and hopes to continue providing new plugins and features to the open source project.

Haraka can be found here: http://haraka.github.com/

Here are the tools I use for this:

• EC2 running Linux (Ubuntu)
• Ruby and sinatra (but you can use something else if it floats your boat)
• Nginx frontend
• runit for keeping services running, and logging

I’m not going to go into details on how to setup EC2 – just get a Linux host of some sort (we’re using Ubuntu) up and running there, and assign an easy to remember CNAME to point to it (below I call this “yournewhostname.yourdomain.com”).

Firstly create a user account for your deployment on your EC2 instance, and create ssh keys for this user – we’ll call this user “deployment”. Then create a github account for that user, and add your ssh public key into that new account’s github configuration. [This is necessary only for private repositories - if you're using a public repo then it's not required]. You then need to make a “Team” for this user, so that account can get “pull” request rights on your project. We call our team “EC2 Deployment”. Set the team up so it gets pull rights only. Add this new user to that team.

Then in your Github repository, go into Admin, Teams, and add the “EC2 Deployment” team to your project. Now while you’re in Admin, go into Service Hooks, and add a Post-Receive URL hook pointing to http://yournewhostname.yourdomain.com/.

Now configure nginx to forward requests to yournewhostname.yourdomain.com to a server on localhost on a unique port. I use 10001, but pick whatever works.

Now you need a Ruby sinatra app running on 10001 which receives the POST data, performs a pull request, and restarts the service. This has to run as root (to restart the service) so make sure you audit this code!

Finally, here’s the sinatra code that processes the request:

post '/' do
 if (!params[:payload])
   return "Stop sending me rubbish"
 end
 push = JSON.parse(params[:payload]);

 if (push['ref'] != 'refs/heads/master')
   return "Not a push on master - ignorning"
 end

 repo = push['repository']['name'];

 Dir.chdir("/var/deployments/" + repo)
 if (system("/bin/su", "deployment", "-c", "/usr/bin/git pull"))
   puts "git pull of #{repo} successful"
   if (system("/usr/bin/sv", "kill", "."))
    return "Restarted!"
   end
 end
 return "Thanks";
end

I removed a lot of the logging here, but you get the idea. It’s pretty simple, and doesn’t do any testing or anything like that, but I’ve provided this here for you as an example, not as the final solution.

PS: We’re hiring. Come work for us!

This is a minor version bump because of some significant logging changes:

• Fix for helo_checks plugin for dynamic looking HELOs
• All logging should now include the log level, UUID, the plugin name (if available) and the logging details

For those happy with the current logging there is no particular reason to update.

Firstly why did I bother upgrading? Well it was mostly for Airplay – the ATV1 doesn’t do Airplay (it does sound only), and so I wanted to be able to load up a youtube video or things like that on the iPad (which I also bought), and have them play on the TV. I also just wanted to see what the latest version was like, and for $110 (CAD) it is a great price.

My biggest gripe is that despite the software version being the same as what I had on the old ATV, the user interface has changed for the worse. The ATV2 no longer syncs from my iTunes, it only does “Home Sharing”. Previously it would sync the playlist (I didn’t have it copy things over – just the playlist) and have a Movies menu containing my movies, a TV Shows menu showing my TV Shows, and a Music menu showing my music. It would also give me direct access to the most recently added items in there.

The new version however has the “Computers” menu, which shows my iMac iTunes share, so you go in there, click on Movies, and it syncs the list of movies right then (it’s not too slow, but it takes a couple of seconds), and then just shows them all. No access to the most recently added movies, though technically I could make a playlist for this I guess.

Furthermore, the TV Shows are no longer grouped by TV Show, but by TV Show Series. So for something like Dexter and How I Met Your Mother I now have 6+ entries in the list to scroll through. This makes the list of TV Shows way too long.

Since watching movies that I have on my iMac iTunes share is my primary use of this device, this has been a disappointing experience.

Of course that’s not how Apple wants you to use the device any more. They want you to rent movies and TV Shows on it from the iTunes Store, or use Netflix. I haven’t signed up for Netflix yet, so I can’t describe how good that interface is yet.

Pros:

• Airplay – seems to work well for youtube/vimeo videos on the iPad.
• Smaller, runs cooler, no hard drive to go wrong.
• Rewinding and ffwding movies seems to work better.
• Netflix access.

Cons:

• Airplay doesn’t seem to work with music videos on the iPad. WTF? I mean I can access them directly on the ATV itself (via the horrible menu system I described above), but why not Just Work? It transfers the sound only. This may be an iPad issue though.
• The lack of Sync and interface to Home Sharing is crap
• Not much of an update really – the UI should be much fancier
• No ability to sign up for Netflix or any of the other services on there via the ATV itself. Not a huge deal, but goes against the grain

Overall I’m going to keep with it for a while, and see if subscribing to Netflix helps things out.

You can almost never log enough. When we’ve come across problems that we don’t understand, the first thing we end up doing is adding more logging.

One of my goals in Haraka has been to ensure that issues are easy to track. Deny (5xx and 4xx to the SMTP-savvy people) messages contain a UUID which can be tracked back directly into the logs. Having sat across the developer/sysadmin divide for a number of years (ok in truth, I just had fantastically bitchy sysadmins, and I love them for that) this is HUGELY important.

A lot of developers think that writing defect free software is the most important thing. They are entirely wrong – even defect free software gets a boat load of bug reports. Most problems and expenses with software will occur after you ship it. The only way you are going to be able to track what that problem was is through extensive logging.

So my message to you, dear coder friends, is to log early, log often, and log in a way you can track it back to an individual transaction. And save logs, lots of them, for as long as you possibly can.

And for Haraka users – we are working on improving this – it will be improved in the next major release (even though it’s pretty darn good already).

What I’ve been thinking is something that takes a sender, recipients, a subject, a text body, and potentially a HTML body, and produces the correct contents for outbound, and sends the mail.

It might return a UUID which you could query for a status (but that’s yet to be determined – and not something Haraka can do yet).

Is that something the senders who follow this blog would be interested in?

I’m here to tell you that jslint (and the fork: jshint) is absolutely bloody useless, and hence I don’t use it.

Let’s have a quick look shall we. First let’s install jshint (which is better than jslint, and more modern):

$ sudo npm -g install jshint
Password:
/usr/local/bin/jshint -> /usr/local/lib/node_modules/jshint/bin/hint
jshint@0.5.2 /usr/local/lib/node_modules/jshint
├── argsparser@0.0.6
└── minimatch@0.0.4

That all went well… So lets try it on haraka.js:

$ jshint haraka.js
haraka.js: line 3, col 1, Use the function form of "use strict".
haraka.js: line 5, col 14, 'require' is not defined.
haraka.js: line 8, col 1, 'process' is not defined.
haraka.js: line 8, col 22, 'process' is not defined.
haraka.js: line 10, col 5, 'require' is not defined.
haraka.js: line 10, col 34, 'process' is not defined.
haraka.js: line 13, col 5, 'process' is not defined.
haraka.js: line 13, col 46, 'process' is not defined.
haraka.js: line 16, col 14, 'require' is not defined.
haraka.js: line 17, col 14, 'require' is not defined.
haraka.js: line 18, col 14, 'require' is not defined.
haraka.js: line 20, col 1, 'exports' is not defined.
haraka.js: line 21, col 35, '__dirname' is not defined.
haraka.js: line 24, col 1, 'process' is not defined.
haraka.js: line 34, col 5, 'process' is not defined.
haraka.js: line 37, col 52, 'exports' is not defined.

16 errors

WTF? This script is pretty battle tested. Everything it has complained about are node.js variables, but everything I have read told me that jshint was node.js compatible.

So maybe jshint just shouldn’t be used. Let’s do a search on NPM for alternatives:

lint                  This package provide lint validation library + node-lint command line tool allows you to che

Looks promising. Installed it, tried it:

$ node-lint haraka.js 

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
Error: Cannot find module 'vows'
    at Function._resolveFilename (module.js:320:11)
    at Function._load (module.js:266:25)
    at require (module.js:348:19)
    at Object.<anonymous> (/usr/local/lib/node_modules/lint/lib/lint/vows.js:5:12)
    at Module._compile (module.js:404:26)
    at Object..js (module.js:410:10)
    at Module.load (module.js:336:31)
    at Function._load (module.js:297:12)
    at require (module.js:348:19)
    at Object.<anonymous> (/usr/local/lib/node_modules/lint/lib/lint/index.js:15:16)

Fucking useless. And dependencies really aren’t that hard in npm. Let’s press on:

nodelint              The nodelint command line tool allows you to check for problems using JSLint. You can specif

What do we get:

$ nodelint haraka.js
0 errors

Praise Thor! OK so now we’re getting somewhere. Let’s move on to another file:

$ nodelint utils.js
utils.js, line 5, character 78: Use the array literal notation [].
var CHARS = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'.split('');
utils.js, line 9, character 10: Move 'var' declarations to the top of the function.
for (var i = 0; i < 36; i++) {
utils.js, line 9, character 10: Stopping.  (10% scanned).

3 errors

These kinds of errors are completely fucking unreasonable, and laid down by Crockford who clearly needs to revise his viewpoint – putting the var in the for() statement is a reasonable thing to do, and yes I KNOW it doesn’t do what it looks like it does. I don’t care. And the problem is that these aren’t configurable checks for jslint – the “var at the top of the function” is not a parameter in the jslint config, and if it finds too many of those it just stops until you fix them!

So does nodelint actually find anything worth worrying about?

The truth is, yes it does occasionally find some issues, but getting it to accept your style is hard. Getting it to find real bugs is really hard. I’d love to be proved otherwise.

A while ago someone (somewhere – maybe the Node.js mailing list, I forget) mentioned that they had ported PEAR’s language detection module to Node. This struck me as a great thing to have in Haraka, as I’m continually bombarded with foreign spam which slips through the filters (mostly mainsleaze, and the SBL clearly misses it). There’s only me on my domain, so blocking anything not in English is a reasonable thing to do.

The first step to this though had to be decoding mail body text properly to UTF-8. There’s been a TODO line in Haraka forever to do this. I just never got around to it.

So after work today I made that work. It took a fair bit of hackery as previously I was decoding base64 line by line, but that can easily break encoding routines because you get partial characters. So I modified this to decode the entire thing at once into a single buffer, then it was just a matter of using node-iconv to convert (and cope with errors).

Then language detection was just a matter of finding the right body part – in emails you can have both a text/plain part and a HTML part saying the same thing (or sometimes different things), so in that case I prefer the HTML part as it’s more likely to contain relevant text. Then I strip HTML and pass it to node-languagedetect (available on npm). This returns a list of possibilities and confidences. Of which I just choose the top one. Bingo.

66 lines of code. It’s pretty horrible so I won’t put it into the distro. But come the next release (when the iconv stuff is there) I’ll be putting it to the test live on my server.

Haraka is a scalable event based SMTP server, designed to be extended with plugins to allow you to process your email however you require. It ships with a number of unique plugins to fight spam and malware, and provide you with many ways to deliver email either inbound to your users, or outbound to the wider internet.

v1.0 marks a major milestone in the development. While the core remains stable and there are no major API changes in this release, it was a significant step up from 0.9.0.

Major changes in this release:

• New clamd plugin for email anti-virus
• Syslog plugin allowing logs to be sent to syslog
• Support for JSON and “data” config files
• Support for the XCLIENT protocol when delivering onward to servers that support it
• Connection pool for smtp_proxy backend for large scale installations
• Plugin inheritance – allows plugins to share code by subclassing
• New quarantine plugin
• New rdns plugin which allows large amounts of control over hosts with bad/broken reverse-DNS setups
• TLS everywhere – full support for TLS on outbound, and in smtp forward and proxy plugins
• Added delivered hook to outbound mailer allowing you to gather stats on outbound mail
• The usual assortment of bug fixes and performance improvements

Thanks to the contributors (see the Commits page on github) for their contributions – this release is all down to their help.

This release is also faster than 0.9.0 – my previous benchmark showed it capable of receiving 50,000 messages in 12 seconds. This release brings that down to 8.5 seconds, and with node version 0.6 (which is net yet released), even faster still: 7.8 seconds! That’s nearly 30 million emails an hour!

Haraka also has a new web site. Contributions to improve that would be most welcome.

As usual, the main Haraka github page for download and installation instructions is in the sidebar.