Getting your web server mail through spam filters
Most of the websites we build have some need to send email out – even if it’s just a contact form to pass on enquiries to the business. For the more complex web applications we may need to send out email in bulk. This can be made pretty painful thanks to the ubiquitous spam filters that mailboxes will put in our way.
Here’s what we’ve learnt:
The Basics
The no-frills way of sending email from a server is the rough-and-ready PHP mail() function. Depending on how your server is set up this may or may not work for you. On a Linux box this will use the sendmail binary. On a Windows box it will normally use the local SMTP server (if one exists).
The first gotcha is the sendmail_from parameter. This is different from the ‘From:’ header, and a lot of mail servers will reject your mail without it.
On a Windows server you have to do an ini_set call to set it:
ini_set("sendmail_from", "sending_email@domain.com");On a Linux server you just can pass a final parameter to the mail() function:
mail($to, $subject, $body, $headers, '-fsending_email@domain.com');So with your email getting sent “properly”, the spam filter has five pieces of information with which to decide if it’s going to let the message through or not:
- The sending email address
- The email in the ‘From’ header
- The IP address the message was received from
- The textual content of the subject and body of the email
I’m going to leave aside the textual content, except to say that there are plenty of tools out there which will tell you how spam-like they reckon your email content looks (e.g. emailspamtest.com) – and of course there’s your own common sense. Assuming you’re not trying to flog pharmaceuticals or exotic international financial opportunities, you’re unlikely to come unstuck here.
Who really sent that?
A big part of the work of spam filters is to answer that question. What is the relationship between the server and the email address that it came from?
Identifying the correct email server for a domain is easily done from the MX record stored on the domain name system (DNS) – after all, you need this to send email to it. But most websites are hosted on servers dedicated to serving up websites. Your web server is actually just ‘spoofing’ the email address. A spam filter can tell that it didn’t come from the domain’s email server, so it will be suspicious.
The way around this is to “license” your web server to send email from the domain with a Sender Policy Framework (SPF) record in the DNS. In general you can easily add this for any domain you own through a control panel. Rather than struggle with the syntax you can use an online wizard to create the record, of which there are several (e.g. from Mail Radar, Cloudfloor DNS or Roy’s Tech Blog).
The server’s reputation is also key. If you’re on shared hosting you may come unstuck here – the server’s IP address will host hundreds if not thousands of websites and any one of them may be a long-neglected malware-infected site spewing adverts for investments in Nigerian viagra to all four figurative corners of the globe. You can find out whether you’re on a blacklist of known spammers again using any one of several online tools (such as those from MXToolbox, WhatsMyIPaddress.com or Blacklistmaster). Reputable hosts will generally try to get shared servers off blacklists pretty quickly, so if you find your site’s on one, let them know.
If you have your own dedicated server (as we do) then part of your routine maintenance should be checking that you’re not blacklisted and if you are, working out why and getting off the blacklist ASAP!
Some more gotchas
Doing the above should get you sending email pretty fluidly. But there’s a few more pitfalls.
One crucial lesson: don’t send to the system you’re spoofing! This is particularly an issue when sending server mail to Gmail – because Google pretty much knows everything that’s going on out there! If the email address you’re spoofing is also a gmail (or google-app hosted) domain, Google is going to know that it didn’t send the email. If you have delegated mail for your domain to Google apps, and your server is sending mail pretending to be from an address at your domain, you can pretty much kiss goodbye to getting anything through to a Gmail address.
So here’s the solution: send your mail using Google’s SMTP server. A typical method would be to create a noreply@ address on your domain, setting it up as a forwarder and then using it to log on to Google’s SMTP server. To do this you either need to do some tinkering with your PHP config, or (far easier) just use the PEAR mail class – this is a free download as a PHP file or grabbable through Linux. You’ll also need the Net SMTP package too.
Assuming you’ve got PEAR available to you, here’s your code:
require_once "/path/to/Mail.php";
$host = "ssl://smtp.gmail.com";
$port = "465";
$headers = array (
'To' => $send_to,
'From' => $send_from,
'Subject' => $subject,
'Reply-To' => $real_email_address
);
$smtp = Mail::factory('smtp',
array (
'host' => $host,
'port' => $port,
'auth' => true,
'username' => "noreply@yourdomain.com",
'password' => "Google account password for the noreply account";
)
);
$mail = $smtp->send($send_to, $headers, $email_body);
if (PEAR::isError($mail)) {
// Mail sending failed!
}(A sub-gotcha with this: Bcc doesn’t work as you’d expect – add the Bccs to the $send_to string, not the $headers array. Doing it the mail() way results in the Bcc appearing visibly in the email header in many mail clients, but not actually receiving the email!)
Of course since Gmail has a brilliant crowd-sourced spam filter, you can never be 100% sure – if people have started claiming your emails are spam then Google will assume they’re right and start spam-binning them across the board. That’s the genius and weakness of Google: we all work for them.
Finally, be aware that if you’re using someone else’s server there may well be a limit to the number of emails you can send out in any time period. That goes for Google’s SMTP servers and a lot of shared web hosting we’ve seen, even using the mail() command. We built a mailQueue class which can queue up and send mail at a rate which can be adjusted to the needs of different servers. But if you’re sending out bulk mail you should really have either your own server or use something like Campaign Monitor, which has an excellent API for sending out bulk mail.
