Except that the chance this is actually true for you, statistically, is very small. If we look at the statistics, then what we see is that people are storing increasing amounts of things, important things even, online because disks keep getting bigger and cheaper (though sadly not much faster, but that’s a different article).

But despite this availability of cheap storage, people for the most part still don’t back things up. This isn’t universally true of course, there are some folks who have implemented really brilliant backup systems at their companies.

Of course, it helps when you have an amazing systems guy on your team who is made out of lighting bolts and awesomeness, taking care of the nitty gritty details for you. If, however, you are one of the many many people who doesn’t have a good backup strategy in place, this article is here to help you out.

I’ve written a shell script (Ed: please remove .txt extension before use) that will serve as a good starting point, and we’ll talk through it in a little bit. For now though, let’s discuss backups in a more general sense so we can decide what our end goals are.

Types of things to back up

There are lots of different kinds of data that you may need to back up. Three of the most common are:

1. Source Code
2. Static Files (images, PDFs, etc)
3. Data in a Database

These days, there’s a good chance you have your source code files in an RCS system somewhere, such as GitHub. If you do (and you should), and your RCS system is on a different server or servers than your site, you already have your source code backed up. If your server catches on fire, you can just do a fresh checkout onto a new server. This is one of the many advantages of using an RCS system.

Static files, such as images or music or what have you, are easy to back up. A quick search will reveal dozens of different ways to ship this sort of data to some secondary place in case of emergency.

Database data, however, can be a little more tricky. You can’t just copy your binary data files safely while the database is running because if a write happens during the process, then your “backup” will be in an inconsistent state, which will probably render it useless.

Moreover, it’s not generally the best way to do a backup because restoring data from a binary copy can be tricky. What my shell script does is allow you to automate correctly backing up a standard MySQL database, or databases, with full end-to-end encryption to an offsite location.

Is my data backed Up?

This seems like an easy question to answer. In fact, it is a rather easy question to answer, but I’ve found that many people have never bothered to answer it, and it gets them in trouble. Are you doing backups of some sort currently? If so, does that mean you have backups? Actually, no, it doesn’t. If there is one golden rule of having backups that I hope you remember after reading this, it’s the following:

If your backup procedure doesn’t include instructions for doing a restore of the data, or if your restoration procedure has not been tested, then you don’t have backups.

Simply put, unless you know that you can reliably do a restoration of your data, you don’t have backups. If you haven’t tried, you don’t know. As an example, I once encountered a setup where an administrator was “backing up” several databases by using rsync to copy the binary data store to a remote machine once a day. He thought he had backups, and he was wrong.

As I noted before, if a write happens during the rsync process, those binary files aren’t guaranteed to be in a consistent state. If he’d ever tried to do a restore of those files (he hadn’t), he would have discovered this problem. So I say again, until you have tested a restore and it’s worked, and you’ve documented how to do that restore, you don’t have backups.

Another thing that I always like to stress is that you really, really want backups that are under as much personal control as possible. This typically means using a third party or offsite service distinct from your ISP to store your backups. If you ever need to restore some data, there’s a good chance it’s because something has gone wrong with your systems.

In that case, there’s also a fair chance that something has gone wrong at your ISP itself, so you’d rather not have your backups tied to your ISP. What you want is backups that are stored by some other company, preferably in an entirely different physical data center.

Basic process for setting up database backups

The steps I use to back up the databases for Virb are as follows:

• Get a dump of the database
• Compress the data
• Encrypt the data
• Send the data over an encrypted connection to an offsite service
• Automate the entire procedure with a cron job

It’s nothing fancy, and it shouldn’t be. Fancy, intricate things tend to break, and this is the last thing you want to worry about breaking.

Prerequisites

You’ll need to have a few things in place for this to all work. I’m assuming you’re using a reasonably new Debian style Linux distribution (which includes Ubuntu). You need to be able to use the mysqldump program to get a dump of the desired database data. This comes standard with the MySQL installations on every Linux distribution I know of.

I’m going to assume that you have a .my.cnf file set up in your home directory with a username and password that allow you to dump out the data you want. If you’ve never used a .my.cnf file before, it’s very easy. Create a file called .my.cnf in your home directory, and put the following in it:

[client]
user = dbusername
password = dbpassword

of course substituting the correct username and password for your systems. Then run chmod 600 $HOME/.my.cnf on the file so other people can’t read it. You can now use all the mysql* command line utilities without having to pass the -u and -p flags all the time.

Another assumption I’m making here is that you can safely run mysqldump at some point during the day, and this won’t mess anything on your site up, which practically means that you don’t have that much data. When you dump the database out, a write lock is placed on all the tables in that database (assuming you’re dumping out all the tables). So while the data is getting written out, reads can still happen, but no INSERTS, UPDATEs or DELETEs can.

If you have say a few hundred megabytes of binary data, usually located in /var/lib/mysql on Linux systems, you are probably okay just dumping the data out from whatever machine the database is on. On reasonably modern hardware, 300M of binary data takes about 15 seconds to dump out, for example. Having a write lock for 15 seconds in the middle of the night is not a big deal for most people.

If you have enough data that it really does take a long time, or your requirements are such that you can’t have a write lock happen even briefly, then the next option is to set up a mysql slave server that exists just for backups. The idea is that the slave is continuously replicating from the master.

When you do the backup on the slave, the write lock will happen there so that the data can get dumped out, and then once it’s finished the replication will catch back up to the master if it’s missed anything.

Explaining how to set up MySQL replication is beyond the scope of this article, but this “back up a slave” strategy is exactly what we do for Virb.

We’re going to be backing things up using the online back up service provided by IBackup. There are plenty of these sorts of services available, so if you want to use a different one feel free. Not much of what I’m going to describe is specific to IBackup. I have no particular affiliation to them, other than having used them for several projects and being pleased with their products and pricing. I’m going to assume you have an Economy plan with enough storage to hold your backups.

The only other assumptions I’l make are that you have stunnel4, rsync, and gpg installed. These are very standard, freely available programs. On reasonably recent Debian style systems (which include Ubuntu) you can install all of them by running this command as root:

apt-get update && apt-get install stunnel4 rsync gnupg

Setting up encryption

We are going to encrypt our database backups, and we’re going to transmit the encrypted files over a secure connection. This is technically overkill, but encryption is free and easy so we’ll do it anyway. The first thing you’ll need to do is generate GPG keys to use.

I’m not going to explain it here because this tutorial will take you all of five minutes to get through, and covers everything you’ll need. Just be sure to remember your passphrase, and put copies of your key files somewhere safe just in case.

You should now have GPG keys set up, and an account with IBackup, so create a file in your home directory called .ibackup with the following contents:

IBACKUP_USERNAME=username
IBACKUP_PASSWORD=password
GPG_EMAIL=your@email.com

The reason for this file will become apparent shortly. Just substitute the username and password for your IBackup account there, and put in the email address that was used to generate your GPG keys. Be sure to run chmod 600 $HOME/.ibackup so that nobody else can see your password.

Now we need to do a little configuration for stunnel4. We don’t need it to be running all the time, so shut it down and make sure it doesn’t start on boot with the following commands:

/etc/init.d/stunnel4 stop
update-rc.d -f stunnel4 remove

Now let’s configure stunnel4 for IBackup. First get the default configuration file out of the way.

cd /etc/stunnel
mv stunnel.conf stunnel.conf-orig

Next, with your text editor of choice, create a file called ibackup.conf in the /etc/stunnel directory with the following contents:

client = yes
[ibackup]
accept = 8455
connect = rsync4.ibackup.com:5000

These options are fairly specific to IBackup’s service, but other backup services have similar options. The accept option defines a local port for stunnel4 to use, so you don’t have to pick that particular one if, for some reason, it’s already in use.

Performing the backup

Since we have our security issues handled, we can now perform the backup. Open up that shell script (Ed: please remove .txt extension before use) I mentioned at the beginning in your text editor of choice, so we can go through it. The first 14 lines just get the names of databases we’re going to back up and identify the programs we’re going to use.

#!/bin/bash

# ibackup.com destination
IBACKUP_DEST=$1

# database names
DB_NAMES=$2

# binaries we need
MYSQLDUMP=/usr/bin/mysqldump
GZIP=/bin/gzip
GPG=/usr/bin/gpg
RSYNC=/usr/bin/rsync
STUNNEL=/usr/bin/stunnel4

If you’re not familiar with bash shell scripting, it’s not hard. Things like $1 and $2 indicate command line arguments that get passed to the script. So here, $1 is something we’re going to use in the name of the eventual backup file we’ll get, and $2 will hold the name(es) of the database(es) we will back up.

Things get more interesting from lines 23 to 39.

DOTIBACKUP="$HOME/.ibackup"

if [ ! -r $DOTIBACKUP ]; then
    echo "Need $DOTIBACKUP file for username and password!"
    exit 1
else
    PERMS=`/bin/ls -l $DOTIBACKUP | /usr/bin/awk '{print $1}'`
fi

if [ $PERMS != "-rw-------" ]; then
    echo "Permissions on $DOTIBACKUP are wrong!"
    echo "Should be -rw-------, actually is $PERMS"
    exit 1
fi

. $DOTIBACKUP
export RSYNC_PASSWORD=$IBACKUP_PASSWORD

These commands check that the .ibackup file exists and has the correct permissions. If so, they source the file, which is to say they absorb the things that file defines into the running shell script. Then it sets the environment variable RSYNC_PASSWORD to be the value of the IBACKUP_PASSWORD that you use to access the service.

Lines 41 and 42 grab the local port that stunnel4 needs in order to work.

# local stunnel4 port
STUNNEL_PORT=`/bin/grep 'accept' /etc/stunnel/ibackup.conf | awk '{print $3}'`

Next, between lines 44 and 49, we make make sure there’s a directory where we can dump out all of our data.

# place to dump everything
DUMPDIR="$HOME/.ibackup_data"

if [ ! -d $DUMPDIR ]; then
    mkdir $DUMPDIR
fi

We’re eventually going to set this up to run daily, and if we’re doing a once-per-day backup, it’s nice to keep a few days’ worth around just in case. In lines 51 through 56 we use the very convenient *NIX date program to get date stamps for the current day, the previous day, and the day before that.

# today, for tagging the dumpfile
DATE=`/bin/date +%Y-%m-%d`

# yesterday, and the day before that
ONEDAYAGO=`/bin/date -d '1 day ago' +%Y-%m-%d`
TWODAYSAGO=`/bin/date -d '2 days ago' +%Y-%m-%d`

We’re almost ready to actually dump out our data now. Lines 58 to 86 are really boring and just make sure we have a destination name and the programs we need, so I’ll gloss over those. The important part picks between lines 88 and 92, where we go into our backup directory and actually run mysqldump.

cd $DUMPDIR

echo -n "dumping $DB_NAMES to $DUMPDIR/$IBACKUP_DEST-$DATE.mysql... "
$MYSQLDUMP --opt --databases $DB_NAMES > $IBACKUP_DEST-$DATE.mysql
echo "done!"

Let’s take a look at that dump command to make sure we understand it. The $DB_NAMES variable was set earler in the script. It’s the second command line argument passwed to the script, and it tells mysqldump which databases we care about. The $IBACKUP_DEST variable, similarly, was the first command line argument passed to the script, and it just sets the part of the name of our dump file. Finally, the $DATE variable we just saw getting set a moment ago, and we use it as a date stamp for this dump.

Next, between lines 94 and 105, we compress and encrypt our dump file. We also check to make sure the encryption step actually worked and we exit out if not.

echo -n "gzipping $DUMPDIR/$IBACKUP_DEST-$DATE.mysql... "
$GZIP $DUMPDIR/$IBACKUP_DEST-$DATE.mysql
echo "done!"

echo -n "encrypting $DUMPDIR/$IBACKUP_DEST-$DATE.mysql... "
$GPG -e -r $GPG_EMAIL $DUMPDIR/$IBACKUP_DEST-$DATE.mysql.gz
echo "done!"

if [ $? -ne 0 ]; then
    echo "encryption failed!"
    exit 7
fi

We do a little cleanup and start stunnel4 in lines 107 to 112, which are pretty self explanatory.

echo -n "removing $DUMPDIR/$IBACKUP_DEST-$DATE.mysql.gz... "
/bin/rm -fv $DUMPDIR/$IBACKUP_DEST-$DATE.mysql.gz
echo "done!"

echo "Starting stunnel for rsync..."
/etc/init.d/stunnel4 start

And now on line 114, we finally send our freshly made backup off to the remote servers managed by IBackup for safe keeping.

$RSYNC -r -v -z -t --delete-after --exclude $IBACKUP_DEST-$ONEDAYAGO* --exclude $IBACKUP_DEST-$TWODAYSAGO* $DUMPDIR/ $IBACKUP_USERNAME@localhost::ibackup/$IBACKUP_DEST --port $STUNNEL_PORT

This is using several different features of rsync so let’s talk about them so we know what is going on. The command line flags -r -v -z -t are what the folks at IBackup tell us to use when syncing things over to them. The --delete-after flag tells rsync to delete anything in the target directory that’s not in the local directory being sync’d. But we also have the two --exclude options there.

So what we’re really saying is “Keep the file we’re sending over, and the one from yesterday, and the day before that, but then delete everything else”. That way we always have the last three days’ worth of backups. The last point of interest is that because we tell rsync to use $STUNNEL_PORT, all the communication is encapsulated by stunnel4 and is therefore encrypted across the wire as it gets transferred. Whew!

The last few lines from 116 to 121 simply stop stunnel4 and clean up things by removing the directory we made to put the dump file into.

echo "Stopping stunnel for rsync..."
/etc/init.d/stunnel4 stop

echo -n "Cleaning up... "
/bin/rm -rfv $DUMPDIR
echo "done!"

So that should about do it. To find out, put this script somewhere and run chmod 755 db-backup.sh on it so it’s executable. Then test it to make sure it actually works. If you want the target to be called ‘mybackup’, and you have to back up the databases named ‘db1′, ‘db2′, and ‘db3′, you would invoke the script like this.

/path/to/db-backup.sh mybackup "db1 db2 db3"

If that works, then you are almost done! We just need to automate the process, which we can do trivially with a cron job. In your shell where you tested the script from, type the following:

crontab -e

This will put you into edit mode for your crontab. Assuming you want the backup to happen starting at 12:40am each day, put the following into your crontab file:

0 40 * * * /path/to/db-backup.sh mybackup "db1 db2 db3" >/dev/null 2>&1

Save the file, and then type crontab -l to list the crontab and make sure that line is in there. If it is, then there’s just one last thing to do.

Testing a restore

I said at the start of this article that if you haven’t done a restore, you don’t have backups. And seriously, I meant that. So if you’ve made it this far, then you must do one last step. The day after you set this up, you need to go to your IBackup account, download your dump file, decrypt it with the GPG key you made, and do a restore to a running MySQL instance.

Make sure you do this last step, as the consequences of missing it can be dire.

In conculsion

I covered quite a bit in this article, but if you make it through everything, you should be able to rest easier knowing that you have good, safe backups of your database. Although we used the IBackup service in these examples, very little of what I covered is specific to them.

If you wanted to use a different company, or maybe just push the backup files to a different server you have, you’ll probably just need to change some of the commands that you pass to rsync. The basic strategy though should’t have to change.

I hope you’re found this helpful, and don’t hesitate to get in touch if you have any questions.

DNS is a big topic, and I’m certainly not going to try to cover all of it here. However, I think that by the end of this article, we should have covered the parts that developers should be aware of and understand properly. We’ll start by talking about the sequence of servers that may get consulted whenever a DNS lookup is performed.

Nameservers

The DNS system is in place to translate the representations of internet destinations from things that we humans care about (domain names) to things that computers can use (IP addresses). The system is naturally distributed as we’ll see, which keeps the servers that handle DNS requests from getting too overloaded. There are two distinct sorts of nameservers that we’ll discuss, and each is responsible for a different sort of task. They are:

• Authoritative nameservers
• Resolving nameservers

If you’ve ever moved a domain name between hosting providers, you’ve probably had to do something to “update the nameservers” for the domain. What you are doing there is changing the authoritative nameservers for that domain. Similarly, if you perform a whois query on a domain, it will tell you what the authoritative nameservers are for it. As the name suggests, the job of an authoritative nameserver is to know all the DNS information for the domains assigned to it.

What’s not understood as well is that whatever computer you are using to read this article is almost certainly not talking to an authoritative nameserver. It’s talking to a resolving nameserver, often referred to as just a “resolver”. The resolvers are typically provided to you by your ISP. Their job is to answer the requests of client computers for arbitrary DNS lookups.

When you surf to a website, say amazon.com, your computer asks the resolver what the corresponding IP address is (assuming it’s not stored already in a local cache somewhere on your computer). If the resolver knows, it just answers immediately. If it doesn’t know, the following sequence of things happen very, very quickly:

1. The resolver determines what nameservers are authoritative for the domain.
2. The resolver asks one of the authoritative nameservers for the needed info.
3. Upon getting the info, the resolver stores the information, and answers your computer.

So in this way, DNS info gets distributed around the internet. Typically lots of people will be using any particular group of resolvers from an ISP, and once the resolvers know the answers for any given lookup, they remember the answers for future use. This is why Google’s nameservers can stay up. It’s not as if they get queried every single time anybody goes to google.com, since the relevant resolvers have that information cached close to all the people surfing there.

Of course, eventually, a resolving nameserver will need to check back in with an authoritative one to see if anything has changed. Otherwise, no DNS information could ever update on the Internet. I’ll explain how this works next.

TTL

So in the last section, we noted how a resolver will ask an authoritative nameserver for information that it doesn’t already have. It will always get back a little more than just the information it asked for though. Specifically, it will get something called a TTL value for the lookup, which stands for “Time To Live”. The TTL is measured in seconds, and it tells the resolver how long it should assume the information it got back should be considered valid. After the amount of time specified in the TTL has passed, the resolver should check back in with the authoritative server even if it still has the information for the lookup stored.

This concept is important enough that it’s probably worth talking through an example. Let’s say that I have the TTL for chrislea.com set at four hours, and that a resolver asks my authoritative nameservers for the IP address at exactly 12:00pm. Let’s then assume that I change the IP address in the authoritative servers at exactly 12:01pm. In this scenario, any client computer talking to the resolver that checked in at noon won’t pick up the IP address change until 4:00pm. This is why it can take some time for an IP address change to propagate across the Internet. Had I been smarter, at 8:00am that morning, I would have lowered the TTL so some low value, say 300 seconds. That way, by the time I wanted to make the change at noon, all the resolvers that cared to check would have gotten the five minute TTL value. Therefore, in this case, the IP address change would only take five minutes to propagate across the Internet. After the propagation was finished, I would have updated the TTL again and put it back to a higher value to save strain on my authoritative nameservers.

There’s really no standard for what TTL values are supposed to be. Typical values might correspond to 12 or 24 hours, though they certainly can be longer in some cases. One important note though relates to the “lowering the TTL” trick I just described. I recommend never using a TTL lower than 300 seconds. This is because some resolvers are set up such that if they see a TTL value that’s considered “too low”, it’s assumed to be wrong, and the resolver will just use an internal default value. Said internal default will generally be something like 12 or 24 hours, so you would effectively have induced the exact opposite behavior of what you were shooting for.

We’ve now covered how the relevant servers talk to each other, and how DNS distributes information around. Now, we can dive into exactly what that information is and how it’s used. When a DNS query happens, it’s for a specific type of record. There are lots of different types of DNS records, and we’re not going to talk about all of them here, but we’re going to cover the most common ones that you should know about.

A Records

A lookup for an A record is what people are generally talking about when they generically say “DNS Lookup”. Your computer gives the resolver a domain name, gets an IP address back in response. It’s the sort of lookup that occurs when you are surfing around in a web browser, pinging something, or telnetting to a port.

An important point here is that every time you add a something ending in a dot character “.” to the left of a domain, then as far as DNS is concerned, it’s a differnet domain. For example, the following two domains are distinct:

• chrislea.com
• www.chrislea.com

In practice, basically everybody points the www. to the same IP address as just the base domain, but you don’t have to. They could point to completely different places. Conveniently, you can create a “star” record for subdomains if you want a catchall. For example, I have *.chrislea.com set up as a catchall for my personal domain, and the catchall is pointed to the same IP address as just chrislea.com. Therefore, if you try and ping the domain vogons.chrislea.com, which doesn’t have an explicit entry, it will match the *.chrislea.com entry and ping the same IP as if you had just pinged chrislea.com. However, if you ping debian01.chrislea.com, which does have an explicit entry, you will see the IP assigned to that domain directly. If a catchall doesn’t exist, and you query a subdomain with no entry, the nameserver will respond that there is no entry, and the behavior of the program you’re using in that case is dependent on the program itself.

Understanding how lookups for A records happen is quite straightforward as we’ve seen. There are more complex queries though that require multi stage lookups to complete. Let’s talk about one of these next.

MX Records

The MX part of the name stands for “Mail Exchanger”. As it turns out, email is so special that it has its own class of DNS entries. They aren’t too complex, though it is a bigger topic than the A records we just covered. To understand it, we first have to cover just the basics of how an email gets from your computer to wherever it’s supposed to go.

Many people use desktop email clients such as Outlook, Thunderbird, Mail.app, or (if you’re lucky) Evolution. When you send an email out using one of these programs, it’s not this client software that’s actually responsible for getting that email to its final destination. What happens is that your client program forwards the mail on to a server running Mail Transport Agent (MTA) software such as postfix. This MTA program is what actually sends the mail for you. When it gets an email message, the MTA looks up the domain to the right of the @ symbol in the email address, and then checks DNS for the MX record(s) for that domain. The response should be one or more domain names. Again, just to be clear, it gets domain names in the form of A records in response. It then looks up the IP address for one of the A records, just as discussed in the previous section, and sends the mail to that IP address.

As an example, let’s assume you’re sending an email to my GMail account. When the MTA you’re using gets the email message, it will check the MX records for gmail.com. The response it gets back will (currently) include:

• gmail.com mail is handled by 40 alt4.gmail-smtp-in.l.google.com.
• gmail.com mail is handled by 10 alt1.gmail-smtp-in.l.google.com.
• gmail.com mail is handled by 20 alt2.gmail-smtp-in.l.google.com.
• gmail.com mail is handled by 5 gmail-smtp-in.l.google.com.
• gmail.com mail is handled by 30 alt3.gmail-smtp-in.l.google.com.

Don’t worry about the syntax here. The important points to note are that

1. There are multiple domains returned in the response.
2. These domains are A records.
3. There is a number, used for priority, attached to each domain.

The priority number tells the MTA in what order to try the different domains to send the message to. It will try from lowest to highest. So in this case, the MTA will first try to deliver the message to gmail-smtp-in.l.google.com since that has a priority of 5. If it can’t deliver it to the IP that corresponds to that domain, it will try alt1.gmail-smtp-in.l.google.com next, with a priority of 10, and so on.

There are a variety of reasons for this infrastructure to be in place. First, it allows for some resiliency in email sending, since the receiving entity can have more than one place that mail can be delivered to. If a server goes down, then there may be another one still up that can handle getting the message. Second, it means that the place where your mail goes can be a completely different server, or group of servers, than where your website’s IP address is. And, it can do this without having to have really annoying domains to the right of the @ symbol in your email address. This is frequently done to use third party mail services for your mail, such as a hosted Exchange service or GMail.

Next, let’s cover another two stage lookup type of record. This one is actually a bit simpler.

CNAME Records

A CNAME, which stands for Canonical Name, is best thought of as an alias. The “Canonical Name” name is very poorly worded, and I recommend just always referring to it as a CNAME. The way a CNAME works is that you have a domain you control, and you assign it to be an alias of some other domain name. For example, I currently have the domain thechris.org CNAME’d to the domain virb.com. There is no A record set up for thechris.org, so if you surf there in your browser, the CNAME value is what’s returned by DNS. Much like with MX records, the domain virb.com is returned, and then a second lookup happens to determine what IP that is. Then, your browser knows to go to that IP address. However, it will still show thechris.org in the URL bar.

Now, I could have simply pointed the domain thechris.org to the virb.com IP address using an A record. Virb would not know any difference if I did. The problem with that approach is that if, in the future, we ever change the IP address for virb.com, I would have to go and explicitly update my A record for thechris.org to whatever the new IP address was in order for things to keep working. Using a CNAME alias like this, things just keep working automagically as we’d like them to.

Okay, enough with these fancy double lookup records. Let’s move on to our next topic which will bring us back to a straightforward lookup mechanism, albeit a good one to understand.

TXT Records

The TXT record is really boring, at least in terms of how it’s defined in the RFC. Essentially, this field is supposed to be a “comments” area, where you can put whatever you want, but it’s not supposed to contain important or machine readable information. So why am I mentioning it? Well, as it turns out, even though you’re not supposed to put imporant stuff in there, people do it all the time. The most common reason currently is to provide SPF data. Really covering SPF is outside of the scope of this article, but in a nutshell, it’s a way to specify via DNS what IP address are allowed to send mail out for a certain domain. For example, here is the TXT record currently in place for chrislea.com:


v=spf1 a mx ip4:70.32.89.79 ip4:76.87.248.247 ip4:64.207.133.104 ip4:72.10.46.46 ~all


This says that the four IPs listed there are “blessed” by me as being allowed to send email from the domain chrislea.com. The information is used to help in determining if a message is SPAM or not. For example, if I send a message to my GMail account from my chrislea.com domain, and I look at the headers in GMail by selecting “show original” from the actions, one of the things I see is this.


Received-SPF: pass (google.com: domain of chl@chrislea.com designates 64.207.133.104 as permitted sender) client-ip=64.207.133.104;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of chl@chrislea.com designates 64.207.133.104 as permitted sender) smtp.mail=chl@chrislea.com


So Google checked that I had an SPF record set up, and that the sending IP address from my MTA was in fact permitted. This contributed to a “not spammy” score and the message made it to my Inbox.

It’s important to note again that this technique is really sort of a hack. The DNS specification clearly states that useful information such as this isn’t supposed to go into TXT records. It’s done though because it led to people being able to adopt SPF very quickly, since all the nameserver software has supported TXT records forever. In 2005, IANA introduced a new record, specifically called an SPF record, to get around this. The syntax is identical to what people are putting into the TXT records. However, due to how recently it was introduced, not all nameserver software supports this new record type yet. Therefore most people are still putting SPF information into TXT records.

There is one more record type that I’m going to go over, and as it turns out, it’s also relevant to you largely because of the implications with SPAM.

PTR Records

A PTR record, or “pointer record”, is generally used to set up what’s called a reverse DNS lookup. Basically, the idea is that if there is a server associated with a given IP address, you should be able to ask DNS about that IP address and get some indication that there’s a server there, in the form of a domain name. This is important because many MTA programs that recieve mail are set up to flatly reject any message from an IP address that does not have reverse DNS set up. The idea here is that bad people doing spammy things aren’t going to properly set up reverse DNS, and thus any message coming from such an IP address is considered untrustworthy.

As an example, my personal mail server at mail.chrislea.com has the IP address 64.207.133.104 currently. If I check reverse DNS, for this, I will see that the reverse entry is for debian01.chrislea.com, as this is the name of the server that I handle my mail on. Note that it’s not critical for the domain names to match here. The forward lookup is for the “mail” subdomain and the reverse lookup returns the “debian01″ subdomain. Some SPAM calssifying software might use the fact that the base domains match in part of their scoring, but it’s generally not that critical. What is critical is the fact that the reverse DNS entry exists and is a valid domain name that itself has a forward DNS entry. Because of this (and the fact I don’t send spammy looking emails), my mail typically doesn’t get rejected when I send out to other people. It is a good idea to make sure that whoever you are using to send mail through has reverse DNS set up properly for their sending servers. That said, if you haven’t experienced massive problems with your mail going into SPAM folders, they almost certainly already have.

Using host to check DNS Entries

There is a command line program available on every *NIX type system I know of called host which can be used to check DNS entries directly from nameservers. If you are using Microsoft Windows, you won’t have the host program. I’d recommend scrapping Windows, installing Ubuntu or Fedora on your computer and using that. But since that may not be practical for you, you might want to just skip to the next section. :)

Using host is not hard. First you’ll need to open up a Terminal. If you are using OS X, the Terminal program is in the Applications -> Utilities folder. If you are using Linux or some other *NIX, I’ll assume you already know how to open up a Terminal. :)

It’s easiest to learn by example, so now that you have the terminal open, type in the following:


host chrislea.com


The output you get should look like this:


chrislea.com has address 70.32.89.79
chrislea.com mail is handled by 10 mail.chrislea.com.


You just looked up the A record for chrislea.com, and found that the IP is currently 70.32.89.79. By default, host also looks up the MX records for a domain when you do this sort of lookup. That’s what you see in the second line there. I only have one MX record for my personal email. If I had more, as discussed before, they would all show up here.

A very important thing to be aware of here is that with the above command, you just queried the resolving nameservers that your computer is currently using. If you want to query a different nameserver, you put that in as a final argument like so.


host chrislea.com ns2.mediatemple.net


This should return essentially the same information since the (mt) Media Temple nameservers are authoritative for chrislea.com. If it doesn’t, then it probably means that the authoritative nameserver has been updated with new information, like a new IP address, but that this hasn’t propagated to the resolver you are using yet.

If you want to look at a record that’s not an A record, you can accomplish this with the -t flag using host. Let’s say you want to check on the TXT record for my domain. The command and results on my Linux machine look like this.


chl@melian:~$ host -t txt chrislea.com
chrislea.com descriptive text "v=spf1 a mx ip4:70.32.89.79 ip4:76.87.248.247 ip4:64.207.133.104 ip4:72.10.46.46 ~all"


Here, you can see that it returned the same SPF information that we talked about before, becuase I have that set up in the TXT record for chrislea.com. As another example, let’s look at the MX records for carsonified.com.


chl@melian:~$ host -t mx carsonified.com
carsonified.com mail is handled by 0 ASPMX4.GOOGLEMAIL.com.
carsonified.com mail is handled by 0 ASPMX5.GOOGLEMAIL.com.
carsonified.com mail is handled by 5 ALT1.ASPMX.L.GOOGLE.com.
carsonified.com mail is handled by 5 ALT2.ASPMX.L.GOOGLE.com.
carsonified.com mail is handled by 10 ASPMX.L.GOOGLE.com.
carsonified.com mail is handled by 0 ASPMX2.GOOGLEMAIL.com.
carsonified.com mail is handled by 0 ASPMX3.GOOGLEMAIL.com.


As you can probably guess, Ryan and crew are using GMail to handle their mail needs.

Finally, if you want to look up a PTR record to see the reverse DNS for an IP, just feed that IP address to host like you’d expect to.


chl@melian:~$ host 64.207.133.104
104.133.207.64.in-addr.arpa domain name pointer debian01.chrislea.com.


You can put a different nameserver in as the final argument for any of those last examples if you want to query something other than your resolvers. And, of course, if you’d like to learn about all the other fun options the host command has, you should consult the man page.

Using nslookup to check DNS Entries

If you are using Microsoft Windows, I’m sorry (seriously… Ubuntu will almost certainly install painlessly on your hardware, and it’s much prettier). But, if that’s the boat you are in, the tool they provide to query nameservers directly is called nslookup. I should point out that I’ve basically never used Windows Vista, so I am assuming it works the same on there as it does in Windows XP. If I’m wrong and there’s an astute reader out there who uses Vista and can correct me, please do so.

You’ll first need to open up a DOS shell, which is sort of the same as opening up a Terminal on a *NIX type system. In Windows XP, this can be accomplished by clicking Start, clicking Run, typing “cmd” with no quotes into the dialog and pressing Enter. I’m told that in Vista, you simply click Start, type “cmd” with no quotes, and press Enter.

The nslookup command has a non-interactive mode, and a shell mode. I find the shell mode much more useful so that’s what I’m going to explain. In the DOS prompt, simply type nslookup in order to enter the shell. Once you’re in the shell, type the domain name you are interested in followed by a period to do a lookup of the A record for that domain. If you don’t remember the trailing period, you won’t get the expected results. As an example:


> chrislea.com.
Server: mjolnir.chrislea.com
Address: 192.168.0.1

Non-authoritative answer:
Name: chrislea.com
Address: 70.32.89.79

If you’d like to do a different type of query, you use the syntax set query=<type>. So, if I’d like to check on the TXT record for chrislea.com, the commands would look like this.


> set query=txt
> chrislea.com.
Server: mjolnir.chrislea.com
Address: 192.168.0.1

Non-authoritative answer:
chrislea.com text =

“v=spf1 a mx ip4:70.32.89.79 ip4:76.87.248.247 ip4:64.207.133.104 ip4:72.10.46.46 ~all”

This should look familiar to you from the examples that used the host command (assuming you read those). Now, you’ll note in these results that the first lines indicate that the Server referenced is called mjolnir.chrislea.com at address 192.168.0.1. This device is my local router which happens to be running a resolver. As you might expect, by default, nslookup uses whatever resolving nameservers your Windows installation is itself using by default. If you want to change this, you use the syntax server <IP address of nameserver> to tell nslookup to use something different. Unfortunately, you can’t just use the hostname of the nameserver as far as I know. You can see what the IP address is for a nameserver by just pinging whatever that server’s hostname is from the DOS shell. For example:


C:\Documents and Settings\Chris Lea>ping ns1.mediatemple.net

Pinging ns1.mediatemple.net [64.207.129.18] with 32 bytes of data:

Control-C
^C
C:\Documents and Settings\Chris Lea>nslookup
Default Server: mjolnir.chrislea.com
Address: 192.168.0.1

> server 64.207.129.18
Default Server: ns1.mediatemple.net
Address: 64.207.129.18

> chrislea.com.
Server: ns1.mediatemple.net
Address: 64.207.129.18

Name: chrislea.com
Address: 70.32.89.79

Once you’ve used the server directive to change which nameserver nslookup is querying, it stays with that nameserver until you change it back or you exit the program.

I think this basically covers the same functionality as what I went over with the host command. You should now be able to query arbitrary nameservers for all the sorts of DNS records that we’ve talked about here.

Concluding Information

There are just a few last things that I feel I should mention that didn’t fit in so well elsewhere.

The first is that many of the routers commonly in use today run their own, local, resolving nameservers. As I pointed out in my nslookup examples above, the standard “Default Server” read mjolnir.chrislea.com with IP address 192.180.0.1. That is the name and IP address of my local Linksys router. The Tomato firmware I’m using runs a local resolver. If the resolver on the router doesn’t know some answer that is requested, it will then ask my ISPs resolvers. If they don’t know, they find authoritative nameservers to ask. This is relevant, because sometimes it may be necessary to reboot your local router to flush DNS entries in order to get the most accurate data. I’ve found this to be particularly true with Apple’s Airport routers.

Another thing I’d like to mention is that authoritative nameservers are, generally, “open”. By this I mean that they will answer a DNS request from any IP address that queries it. They basically have to, since they by definition have authoritative information that other random systems may need to know. However, resolving nameservers do not share this concern. Therefore, typically, ISPs will limit connectivity to their resolving nameservers to their own IP space. For example, the resolving nameservers we run at (mt) Media Temple will only respond to queries from the (mt) IP space. This is something that is wise to keep in mind, but that practically will not cause much trouble for you in my experince.

Lastly, as I said at the beginning, DNS is a big topic. This should have given you a pretty solid grounding, but there is certainly a lot more to know about it if you’re trying to be comprehensive. If this applies to you, the standard first place to start reading is the book DNS and BIND from O’Reilly press. It’s commonly just known as “the bugs book” among systems administrators due to its ubiquity. A great deal of the book focuses on administering the BIND software package, which you will hopefully never have to deal with. That said it’s still the standard reference for learning DNS.

With those final notes, I will conclude. I hope you’ve found this article informative. If you have any questions or (cough) corrections, please comment below or contact me.

Like this article?

If you enjoyed, this article, feel free to re-tweet it to let others know. Thanks, we appreciate it! :)

Photo credit: flickr.com/photos/seanosh