The Linux Blog UNIX, LINUX, BSD, OSX

URL Rewriting for Beginners

Introduction

URL rewriting can be one of the best and quickest ways to improve the usability and search friendliness of your site. It can also be the source of near-unending misery and suffering. Definitely worth playing carefully with it - lots of testing is recommended. With great power comes great responsibility, and all that.

There are several other guides on the web already, that may suit your needs better than this one.

• Apache URL Rewriting Guide - The best guide around

Before reading on, you may find it helpful to have the mod_rewrite cheat sheet and/or the regular expressions cheat sheet handy. A basic grasp of the concept of regular expressions would also be very helpful.

What is "URL Rewriting"?

Most dynamic sites include variables in their URLs that tell the site what information to show the user. Typically, this gives URLs like the following, telling the relevant script on a site to load product number 7.

1. http://www.pets.com/show_a_product.php?product_id=7

The problems with this kind of URL structure are that the URL is not at all memorable. It's difficult to read out over the phone (you'd be surprised how many people pass URLs this way). Search engines and users alike get no useful information about the content of a page from that URL. You can't tell from that URL that that page allows you to buy a Norwegian Blue Parrot (lovely plumage). It's a fairly standard URL - the sort you'd get by default from most CMSes. Compare that to this URL:

1. http://www.pets.com/products/7/

Clearly a much cleaner and shorter URL. It's much easier to remember, and vastly easier to read out. That said, it doesn't exactly tell anyone what it refers to. But we can do more:

1. http://www.pets.com/parrots/norwegian-blue/

Now we're getting somewhere. You can tell from the URL, even when it's taken out of context, what you're likely to find on that page. Search engines can split that URL into words (hyphens in URLs are treated as spaces by search engines, whereas underscores are not), and they can use that information to better determine the content of the page. It's an easy URL to remember and to pass to another person.

Unfortunately, the last URL cannot be easily understood by a server without some work on our part. When a request is made for that URL, the server needs to work out how to process that URL so that it knows what to send back to the user. URL rewriting is the technique used to "translate" a URL like the last one into something the server can understand.

Platforms and Tools

Depending on the software your server is running, you may already have access to URL rewriting modules. If not, most hosts will enable or install the relevant modules for you if you ask them very nicely.

Apache is the easiest system to get URL rewriting running on. It usually comes with its own built-in URL rewriting module, mod_rewrite, enabled, and working with mod_rewrite is as simple as uploading correctly formatted and named text files.

IIS, Microsoft's server software, doesn't include URL rewriting capability as standard, but there are add-ons out there that can provide this functionality. ISAPI_Rewrite is the one I recommend working with, as I've so far found it to be the closest to mod_rewrite's functionality. Instructions for installing and configuring ISAPI_Rewrite can be found at the end of this article.

The code that follows is based on URL rewriting using mod_rewrite.

Basic URL Rewriting

To begin with, let's consider a simple example. We have a website, and we have a single PHP script that serves a single page. Its URL is:

1. http://www.pets.com/pet_care_info_07_07_2008.php

We want to clean up the URL, and our ideal URL would be:

1. http://www.pets.com/pet-care/

In order for this to work, we need to tell the server to internally redirect all requests for the URL "pet-care" to "pet_care_info_07_07_2008.php". We want this to happen internally, because we don't want the URL in the browser's address bar to change.

To accomplish this, we need to first create a text document called ".htaccess" to contain our rules. It must be named exactly that (not ".htaccess.txt" or "rules.htaccess"). This would be placed in the root directory of the server (the same folder as "pet_care_info_07_07_2008.php" in our example). There may already be an .htaccess file there, in which case we should edit that rather than overwrite it.

The .htaccess file is a configuration file for the server. If there are errors in the file, the server will display an error message (usually with an error code of "500"). If you are transferring the file to the server using FTP, you must make sure it is transferred using the ASCII mode, rather than BINARY. We use this file to perform 2 simple tasks in this instance - first, to tell Apache to turn on the rewrite engine, and second, to tell apache what rewriting rule we want it to use. We need to add the following to the file:

1. RewriteEngine On # Turn on the rewriting engine
2. RewriteRule ^pet-care/?$ pet_care_info_01_02_2003.php [NC,L] # Handle requests for "pet-care"

A couple of quick items to note - everything following a hash symbol in an .htaccess file is ignored as a comment, and I'd recommend you use comments liberally; and the "RewriteEngine" line should only be used once per .htaccess file (please note that I've not included this line from here onwards in code example).

The "RewriteRule" line is where the magic happens. The line can be broken down into 5 parts:

• RewriteRule - Tells Apache that this like refers to a single RewriteRule.
• ^/pet-care/?$ - The "pattern". The server will check the URL of every request to the site to see if this pattern matches. If it does, then Apache will swap the URL of the request for the "substitution" section that follows.
• pet_care_info_01_02_2003.php - The "substitution". If the pattern above matches the request, Apache uses this URL instead of the requested URL.
• [NC,L] - "Flags", that tell Apache how to apply the rule. In this case, we're using two flags. "NC", tells Apache that this rule should be case-insensitive, and "L" tells Apache not to process any more rules if this one is used.
• # Handle requests for "pet-care" - Comment explaining what the rule does (optional but recommended)

The rule above is a simple method for rewriting a single URL, and is the basis for almost all URL rewriting rules.

Patterns and Replacements

The rule above allows you to redirect requests for a single URL, but the real power of mod_rewrite comes when you start to identify and rewrite groups of URLs based on patterns they contain.

Let's say you want to change all of your site URLs as described in the first pair of examples above. Your existing URLs look like this:

1. http://www.pets.com/show_a_product.php?product_id=7

And you want to change them to look like this:

1. http://www.pets.com/products/7/

Rather than write a rule for every single product ID, you of course would rather write one rule to manage all product IDs. Effectively you want to change URLs of this format:

1. http://www.pets.com/show_a_product.php?product_id={a number}

And you want to change them to look like this:

1. http://www.pets.com/products/{a number}/

In order to do so, you will need to use "regular expressions". These are patterns, defined in a specific format that the server can understand and handle appropriately. A typical pattern to identify a number would look like this:

1. [0-9]+

The square brackets contain a range of characters, and "0-9" indicates all the digits. The plus symbol indicates that the pattern will idenfiy one or more of whatever precedes the plus - so this pattern effectively means "one or more digits" - exactly what we're looking to find in our URL.

The entire "pattern" part of the rule is treated as a regular expression by default - you don't need to turn this on or activate it at all.

1. RewriteRule ^products/([0-9]+)/?$ show_a_product.php?product_id=$1 [NC,L] # Handle product requests

The first thing I hope you'll notice is that we've wrapped our pattern in brackets. This allows us to "back-reference" (refer back to) that section of the URL in the following "substitution" section. The "$1" in the substitution tells Apache to put whatever matched the earlier bracketed pattern into the URL at this point. You can have lots of backreferences, and they are numbered in the order they appear.

And so, this RewriteRule will now mean that Apache redirects all requests for domain.com/products/{number}/ to show_a_product.php?product_id={same number}.

Regular Expressions

A complete guide to regular expressions is rather beyond the scope of this article. However, important points to remember are that the entire pattern is treated as a regular expression, so always be careful of characters that are "special" characters in regular expressions.

The most instance of this is when people use a period in their pattern. In a pattern, this actually means "any character" rather than a literal period, and so if you want to match a period (and only a period) you will need to "escape" the character - precede it with another special character, a backslash, that tells Apache to take the next character to be literal.

For example, this RewriteRule will not just match the URL "rss.xml" as intended - it will also match "rss1xml", "rss-xml" and so on.

1. RewriteRule ^rss.xml$ rss.php [NC,L] # Change feed URL

This does not usually present a serious problem, but escaping characters properly is a very good habit to get into early. Here's how it should look:

1. RewriteRule ^rss\.xml$ rss.php [NC,L] # Change feed URL

This only applies to the pattern, not to the substitution. Other characters that require escaping (referred to as "metacharacters") follow, with their meaning in brackets afterwards:

• . (any character)
• * (zero of more of the preceding)
• + (one or more of the preceding)
• {} (minimum to maximum quantifier)
• ? (ungreedy modifier)
• ! (at start of string means "negative pattern")
• ^ (start of string, or "negative" if at the start of a range)
• $ (end of string)
• [] (match any of contents)
• - (range if used between square brackets)
• () (group, backreferenced group)
• | (alternative, or)
• \ (the escape character itself)

Using regular expressions, it is possible to search for all sorts of patterns in URLs and rewrite them when they match. Time for another example - we wanted earlier to be able to indentify this URL and rewrite it:

 

1. http://www.pets.com/parrots/norwegian-blue/

And we want to be able to tell the server to interpret this as the following, but for all products:

1. http://www.pets.com/get_product_by_name.php?product_name=norwegian-blue

And we can do that relatively simply, with the following rule:

1. RewriteRule ^parrots/([A-Za-z0-9-]+)/?$ get_product_by_name.php?product_name=$1 [NC,L] # Process parrots

With this rule, any URL that starts with "parrots" followed by a slash (parrots/), then one or more (+) of any combination of letters, numbers and hyphens ([A-Za-z0-9-]) (note the hyphen at the end of the selection of characters within square brackets - it must be added there to be treated literally rather than as a range separator). We reference the product name in brackets with $1 in the substitution.

We can make it even more generic, if we want, so that it doesn't matter what directory a product appears to be in, it is still sent to the same script, like so:

1. RewriteRule ^[A-Za-z-]+/([A-Za-z0-9-]+)/?$ get_product_by_name.php?product_name=$1 [NC,L] # Process all products

As you can see, we've replaced "parrots" with a pattern that matches letter and hyphens. That rule will now match anything in the parrots directory or any other directory whose name is comprised of at least one or more letters and hyphens.

Flags

Flags are added to the end of a rewrite rule to tell Apache how to interpret and handle the rule. They can be used to tell apache to treat the rule as case-insensitive, to stop processing rules if the current one matches, or a variety of other options. They are comma-separated, and contained in square brackets. Here's a list of the flags, with their meanings (this information is included on the cheat sheet, so no need to try to learn them all).

• C (chained with next rule)
• CO=cookie (set specified cookie)
• E=var:value (set environment variable var to value)
• F (forbidden - sends a 403 header to the user)
• G (gone - no longer exists)
• H=handler (set handler)
• L (last - stop processing rules)
• N (next - continue processing rules)
• NC (case insensitive)
• NE (do not escape special URL characters in output)
• NS (ignore this rule if the request is a subrequest)
• P (proxy - i.e., apache should grab the remote content specified in the substitution section and return it)
• PT (pass through - use when processing URLs with additional handlers, e.g., mod_alias)
• R (temporary redirect to new URL)
• R=301 (permanent redirect to new URL)
• QSA (append query string from request to substituted URL)
• S=x (skip next x rules)
• T=mime-type (force specified mime type)

Moving Content

1. RewriteRule ^article/?$ http://www.new-domain.com/article/ [R,NC,L] # Temporary Move

Adding an "R" flag to the flags section changes how a RewriteRule works. Instead of rewriting the URL internally, Apache will send a message back to the browser (an HTTP header) to tell it that the document has moved temporarily to the URL given in the "substitution" section. Either an absolute or a relative URL can be given in the substitution section. The header sent back includea a code - 302 - that indicates the move is temporary.

1. RewriteRule ^article/?$ http://www.new-domain.com/article/ [R=301,NC,L] # Permanent Move

If the move is permanent, append "=301" to the "R" flag to have Apache tell the browser the move is considered permanent. Unlike the default "R", "R=301" will also tell the browser to display the new address in the address bar.

This is one of the most common methods of rewriting URLs of items that have moved to a new URL (for example, it is in use extensively on this site to forward users to new post URLs whenever they are changed).

Conditions

Rewrite rules can be preceded by one or more rewrite conditions, and these can be strung together. This can allow you to only apply certain rules to a subset of requests. Personally, I use this most often when applying rules to a subdomain or alternative domain as rewrite conditions can be run against a variety of criteria, not just the URL. Here's an example:

1. RewriteCond %{HTTP_HOST} ^addedbytes\.com [NC]
2. RewriteRule ^(.*)$ http://www.addedbytes.com/$1 [L,R=301]

The rewrite rule above redirects all requests, no matter what for, to the same URL at "www.addedbytes.com". Without the condition, this rule would create a loop, with every request matching that rule and being sent back to itself. The rule is intended to only redirect requests missing the "www" URL portion, though, and the condition preceding the rule ensures that this happens.

The condition operates in a similar way to the rule. It starts with "RewriteCond" to tell mod_rewrite this line refers to a condition. Following that is what should actually be tested, and then the pattern to test. Finally, the flags in square brackets, the same as with a RewriteRule.

The string to test (the second part of the condition) can be a variety of different things. You can test the domain being requested, as with the above example, or you could test the browser being used, the referring URL (commonly used to prevent hotlinking), the user's IP address, or a variety of other things (see the "server variables" section for an outline of how these work).

The pattern is almost exactly the same as that used in a RewriteRule, with a couple of small exceptions. The pattern may not be interpreted as a pattern if it starts with specific characters as described in the following "exceptions" section. This means that if you wish to use a regular expression pattern starting with <, >, or a hyphen, you should escape them with the backslash.

Rewrite conditions can, like rewrite rules, be followed by flags, and there are only two. "NC", as with rules, tells Apache to treat the condition as case-insensitive. The other available flag is "OR". If you only want to apply a rule if one of two conditions match, rather than repeat the rule, add the "OR" flag to the first condition, and if either match then the following rule will be applied. The default behaviour, if a rule is preceded by multiple conditions, is that it is only applied if all rules match.

Exceptions and Special Cases

Rewrite conditions can be tested in a few different ways - they do not need to be treated as regular expression patterns, although this is the most common way they are used. Here are the various ways rewrite conditons can be processed:

• <Pattern (is test string lower than pattern)
• >Pattern (is test string greater than pattern)
• =Pattern (is test string equal to pattern)
• -d (is test string a valid directory)
• -f (is test string a valid file)
• -s (is test string a valid file with size greater than zero)
• -l (is test string a symbolic link)
• -F (is test string a valid file, and accessible (via subrequest))
• -U (is test string a valid URL, and accessible (via subrequest))

Server Variables

Server variables are a selection of items you can test when writing rewrite conditions. This allows you to apply rules based on all sorts of request parameters, including browser identifiers, referring URL or a multitude of other strings. Variables are of the following format:

1. %{VARIABLE_NAME}

And "VARIABLE_NAME" can be replaced with any one of the following items:

• HTTP Headers
  • HTTP_USER_AGENT
  • HTTP_REFERER
  • HTTP_COOKIE
  • HTTP_FORWARDED
  • HTTP_HOST
  • HTTP_PROXY_CONNECTION
  • HTTP_ACCEPT
• Connection Variables
  • REMOTE_ADDR
  • REMOTE_HOST
  • REMOTE_USER
  • REMOTE_IDENT
  • REQUEST_METHOD
  • SCRIPT_FILENAME
  • PATH_INFO
  • QUERY_STRING
  • AUTH_TYPE
• Server Variables
  • DOCUMENT_ROOT
  • SERVER_ADMIN
  • SERVER_NAME
  • SERVER_ADDR
  • SERVER_PORT
  • SERVER_PROTOCOL
  • SERVER_SOFTWARE
• Dates and Times
  • TIME_YEAR
  • TIME_MON
  • TIME_DAY
  • TIME_HOUR
  • TIME_MIN
  • TIME_SEC
  • TIME_WDAY
  • TIME
• Special Items
  • API_VERSION
  • THE_REQUEST
  • REQUEST_URI
  • REQUEST_FILENAME
  • IS_SUBREQ

Working With Multiple Rules

The more complicated a site, the more complicated the set of rules governing it can be. This can be problematic when it comes to resolving conflicts between rules. You will find this issue rears its ugly head most often when you add a new rule to a file, and it doesn't work. What you may find, if the rule itself is not at fault, is that an earlier rule in the file is matching the URL and so the URL is not being tested against the new rule you've just added.

1. RewriteRule ^([A-Za-z0-9-]+)/([A-Za-z0-9-]+)/?$ get_product_by_name.php?category_name=$1&product_name=$2 [NC,L] # Process product requests
2. RewriteRule ^([A-Za-z0-9-]+)/([A-Za-z0-9-]+)/?$ get_blog_post_by_title.php?category_name=$1&post_title=$2 [NC,L] # Process blog posts

In the example above, the product pages of a site and the blog post pages have identical patterns. The second rule will never match a URL, because anything that would match that pattern will have already been matched by the first rule.

There are a few ways to work around this. Several CMSes (including wordpress) handle this by adding an extra portion to the URL to denote the type of request, like so:

1. RewriteRule ^products/([A-Za-z0-9-]+)/([A-Za-z0-9-]+)/?$ get_product_by_name.php?category_name=$1&product_name=$2 [NC,L] # Process product requests
2. RewriteRule ^blog/([A-Za-z0-9-]+)/([A-Za-z0-9-]+)/?$ get_blog_post_by_title.php?category_name=$1&post_title=$2 [NC,L] # Process blog posts

You could also write a single PHP script to process all requests, which checked to see if the second part of the URL matched a blog post or a product. I usually go for this option, as while it may increase the load on the server slightly, it gives much cleaner URLs.

1. RewriteRule ^([A-Za-z0-9-]+)/([A-Za-z0-9-]+)/?$ get_product_or_blog_post.php?category_name=$1&item_name=$2 [NC,L] # Process product and blog requests

There are certain situations where you can work around this issue by writing more precise rules and ordering your rules intelligently. Imagine a blog where there were two archives - one by topic and one by year.

1. RewriteRule ^([A-Za-z0-9-]+)/?$ get_archives_by_topic.php?topic_name=$1 [NC,L] # Get archive by topic
2. RewriteRule ^([A-Za-z0-9-]+)/?$ get_archives_by_year.php?year=$1 [NC,L] # Get archive by year

The above rules will conflict. Of course, years are numeric and only 4 digits, so you can make that rule more precise, and by running it first the only type of conflict you cound encounter would be if you had a topic with a 4-digit number for a name.

1. RewriteRule ^([0-9]{4})/?$ get_archives_by_year.php?year=$1 [NC,L] # Get archive by year
2. RewriteRule ^([A-Za-z0-9-]+)/?$ get_archives_by_topic.php?topic_name=$1 [NC,L] # Get archive by topic

mod_rewrite

Apache's mod_rewrite comes as standard with most Apache hosting accounts, so if you're on shared hosting, you are unlikely to have to do anything. If you're managing your own box, then you most likely just have to turn on mod_rewrite. If you are using Apache1, you will need to edit your httpd.conf file and remove the leading '#' from the following lines:

1. #LoadModule rewrite_module modules/mod_rewrite.so
2. #AddModule mod_rewrite.c

If you are using Apache2 on a Debian-based distribution, you need to run the following command and then restart Apache:

1. sudo a2enmod rewrite

Other distubutions and platforms differ. If the above instructions are not suitable for your system, then Google is your friend. You may need to edit your apache2 configuration file and add "rewrite" to the "APACHE_MODULES" list, or edit httpd.conf, or even download and compile mod_rewrite yourself. For the majority, however, installation should be simple.

ISAPI_Rewrite

ISAPI_Rewrite is a URL rewriting plugin for IIS based on mod_rewrite and is not free. It performs most of the same functionality as mod_rewrite, and there is a good quality ISAPI_Rewrite forum where most common questions are answered. As ISAPI_Rewrite works with IIS, installation is relatively simple - there are installation instructions available.

ISAPI_Rewrite rules go into a file named httpd.ini. Errors will go into a file named httpd.parse.errors by default.

Leading Slashes

I have found myself tripped up numerous times by leading slashes in URL rewriting systems. Whether they should be used in the pattern or in the substitution section of a RewriteRule or used in a RewriteCond statement is a constant source of frustration to me. This may be in part because I work with different URL rewriting engines, but I would advise being careful of leading slashes - if a rule is not working, that's often a good place to start looking. I never include leading slashes in mod_rewrite rules and always include them in ISAPI_Rewrite.

Sample Rules

To redirect an old domain to a new domain:

1. RewriteCond %{HTTP_HOST} old_domain\.com [NC]
2. RewriteRule ^(.*)$ http://www.new_domain.com/$1 [L,R=301]

To redirect all requests missing "www" (yes www):

1. RewriteCond %{HTTP_HOST} ^domain\.com [NC]
2. RewriteRule ^(.*)$ http://www.domain.com/$1 [L,R=301]

To redirect all requests with "www" (no www):

1. RewriteCond %{HTTP_HOST} ^www\.domain\.com [NC]
2. RewriteRule ^(.*)$ http://domain.com/$1 [L,R=301]

Redirect old page to new page:

1. RewriteRule ^old-url\.htm$ http://www.domain.com/new-url.htm [NC,R=301,L]

Useful Links

• Regular Expression Testing Tool
• mod_rewrite Forum
• Webapplication Firewall
• .htaccess Tricks
• mod_rewrite Cheat Sheet
• Regular Expressions Cheat Sheet

Summary

Hopefully if you've made it this far you now have a clear understanding of what URL rewriting is and how to add it to your site. It is worth taking the time to become familiar with - it can benefit your SEO efforts immediately, and increase the usability of your site.

Thanks To www.addedbytes.com