Ultimate procmail Recipe

The procmail utility filters e-mail as it’s received (generally on a UNIX-like server), rather than when you retrieve it with your e-mail client (such as OSX Mail, Mozilla Thunderbird, Microsoft Outlook, etc).  As with most UNIX-based utilities, procmail is a very powerful tool for those that take the time to learn how it can be used.  I’ve used procmail for a number of years, and have become pretty comfortable with basic-to-intermediate features.  However there were a few elusive functions I’d wanted for ages but just recently found the time to investigate.  In the interest of openness, here’s what I’ve found.

First, the requirements:

1. Maildir storage behind an IMAP server (dovecot) on a CentOS 5.5 system
2. An IMAP folder may or may not contain space characters in its name
3. Some e-mail messages need to be filed, some marked as “read”, some both – all based on varying conditions
4. The primary mail client is OSX Mail.app (which uses the IMAP IDLE feature)
5. I prefer to use function calls whenever possible, instead of re-writing recipes

Marking a Maildir-based message as “read” is widely documented, and generally points to a 2005 thread in the procmail mailing list for a somewhat cryptic but functional recipe, below.

:0
* conditions
{
  foldername=.whatever
  :0c
  $DEFAULT/$foldername/ # stores in .$foldername/new/

  :0
  * LASTFOLDER ?? /\/[^/]+$
  { tail=$MATCH }
  TRAP="mv $LASTFOLDER $DEFAULT/$foldername/cur/$tail:2,S"

  HOST
}

De-facto reference implementation to mark an Maildir message as “read” with procmail

I’ve used this recipe for a long time, but found that naming folders with spaces caused all kinds of problems because of how variables are handled within the .procmailrc file.  An article at raamdev.com provides an excellent dissection of this recipe, and also yields a quick and easy solution that just combines the use of single and double-quotes to make space-laden folder names work perfectly.  (See link in the “Update” section at the bottom of this article for details on procmail’s unusual quoting syntax.)

:0
* conditions
{
  foldername=”.whatever folder”

  :0c
  “$DEFAULT/$foldername/” # stores in $foldername/new/

  :0
  * LASTFOLDER ?? /\/[^/]+$
  { tail=$MATCH }
  TRAP=”mv ‘$LASTFOLDER’ ‘$DEFAULT/$foldername/cur/$tail:2,S’”

  HOST
}

Modified recipe to handle space characters in folder name

That thread also shows how to use the INCLUDERC directive, which tells procmail to “import” content from another configuration file.  (I don’t remember why, but I started using the SWITCHRC directive instead, which completely transitions parsing to another file and does not return control to calling configuration file.)  By using INCLUDERC or SWITCHRC, the logic needed to “mark as read” is now contained in a separate file, and can be invoked from any recipe in the main .procmailrc file.  This makes maintenance much easier and the ~/.procmailrc file much cleaner in general.

:0
{
  :0c # store as new
  "$DEFAULT/$foldername/"

  :0 # move to cur/ and append a :2,S to mark as read ("seen")
  * LASTFOLDER ?? /\/[^/]+$
  { tail=$MATCH }
  TRAP="mv ‘$LASTFOLDER’ ‘$DEFAULT/$foldername/cur/$tail:2,S’"

  HOST
}

Contents of ~/.procmailrc_markread

:0
* ^X-Spam-Status: Yes
{
  foldername='.Spam Tagged'
  SWITCHRC=$HOME/.procmailrc_markread
}

Excerpt from ~/.procmailrc invoking ~/.procmailrc_markread

We’ve now addressed both the first requirement and the nice-to-have function-like invocation.  However, since OSX’s Mail.app uses the IMAP IDLE feature to quickly announce that new mail has arrived, I found that the client would “receive” e-mail before procmail could finish marking it as read.  This resulted in lots of items I didn’t want to see showing up in a smart folder for “Unread items”.  It was an annoyance – mostly because I knew there had to be a better way.

After some more research, I found a post on the “Free The Mallocs” blog, which has a slight modification to the reference recipe, designed specifically to handle the race condition between an IMAP IDLE client and procmail’s “mark-as-read” hackery.  I made a few changes to be a little more compliant with the desire for Maildir messages moved into the “cur” folder should also have the “flags” portion added to the filename, even if none are set.

:0
{
  :0c # store as new
  "$DEFAULT/$foldername/"

  :0
  * LASTFOLDER ?? /\/[^/]+$
  { tail=$MATCH }
  # first, move from "new/" to "cur/", appending the ":2,"
  # empty flag designator. Then add the "S" flag by
  # renaming the file accordingly.
  # If an idling client already moved the message to "cur",
  # the first move will fail silently, the second will
  # perform the "mark as read" function
  TRAP="mv -v '$LASTFOLDER' \
    '$DEFAULT/$foldername/cur/$tail:2,' 2> /dev/null ; \
  mv -v '$DEFAULT/$foldername/cur/${tail}:2,' \
    '$DEFAULT/$foldername/cur/$tail:2,S'"
  HOST
}

Updated ~/.procmailrc_markread, which bypasses IMAP IDLE race condition

Based on these settings, I was able to create a recipe that files a message, then leaves it “unread” if certain conditions are met or marks as read if not.

:0HBD
* ^From.*backupreport@example.com
{
        # assign the folder name to a varaible
        foldername='.SysAdmin Messages'
        # conditions to leave the message unread
        :0HBD
        * FAILED|ERRORS
        "$DEFAULT/$foldername/"

        # otherwise, procmail 'falls through' to this section,
        # changes to another RC file, and does not return.
        # this RC file marks LASTFOLDER as "read" via IMAP flags
        SWITCHRC=$HOME/.procmailrc_markread
}

Recipe from ~/.procmailrc to file message, then selectively mark as read based on body content

After all of this, I finally have a recipe set that works like I want, and can easily build on to create more elaborate filtering rules.  I’m considering creating a more robust “flag-setting” function that will enable adding and removing the various flags from messages – specifically the “flag” or “mark” flag.  I hope this is helpful in your mail filtering endeavors – please share any notes in the comments.

Update: It was recommended that I more completely explain procmail’s quoting rules, which caused the hassle related to Maildirs with space characters in the names.  I don’t want to regurgitate the procmail documentation here, but it’s an incredibly different beast than anywhere else in UNIX-land.  For a more complete explanation (and possibly a headache), take a look at the very complete  literature at the procmail documentation project.