Better Understand your Ad Traffic

A good understanding of the traffic coming to your site from AdWords is key to having a successful campaign. This page will provide you with an overview to help you improve your understanding of the traffic, especially your unwanted traffic and clicks. Proper use of the auto-tagging feature in Google Analytics (or other website analytics software) can help you to accurately identify the segment of your site visits which correspond to paid clicks and thoroughly track the performance of your campaigns.

Introduction into Web Server Logs

The web server logs are where a web server records and maintains a history of the activities it performs. Commonly, a web server keeps track of:

  • The IP address that issued the request
  • The date and time of the request
  • The resource (for example, the page, image, etc.) which was requested
  • The user agent (basically, the detailed information about operating system and browser) that issued the request
  • The success of the request
  • The referrer of the request

Please note, the information your web server records depends on the configuration of the server. If your logs do not contain all the above mentioned information, your web server might be set to record its activities according to the common log format. If that's the case, change your web server's configuration to track its activities according to the combined log format. The log formats might be named differently depending on the vendor of your web server.

A snippet of an Apache web server log in the combined log format:

172.26.136.223 - - [19/Jul/2011:15:25:57 +0100] "GET /index.html HTTP/1.1" 200 11579  "http://www.google.com/search?q=movies+shop" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30"
172.26.136.223 - - [19/Jul/2011:15:26:30 +0100] "GET /books.html HTTP/1.1" 200 21176 "http://www.mysite.com/index.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30"
172.26.136.223 - - [19/Jul/2011:15:27:21 +0100] "GET /movies.html HTTP/1.1" 200 26781 "http://www.mysite.com/contact.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30"
172.26.136.223 - - [19/Jul/2011:15:27:55 +0100] "GET /movies.html?id=12341 HTTP/1.1" 200 8404 "www.mysite.com/movies.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30"
172.26.136.223 - - [19/Jul/2011:15:27:55 +0100] "GET /movie.jpg HTTP/1.1" 200 320362 "http://www.mysite.com/movies.html?id=12341" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30"

As mentioned, this is a snippet of an Apache log. Commonly, a complete log will have millions of entries and it will grow by hundreds or thousands of new entries each day. A single entry doesn't represent a unique user or a single page impression. One page request can (and mostly will) trigger a large array of requests for images, CSS files etc. that are saved in the web server's logs. In the above shown excerpt, this can be seen in the last two entries. The second entry from the bottom is for the HTML file of the page. The last entry is for a picture (in this case: movie.jpg) that is embedded in the requested page (in this case: movie.html?id=12341).

The following paragraphs explain in more detail the information that is decoded in a single entry:

172.26.136.223 - - [19/Jul/2011:15:27:55 +0100] "GET /movies.html?id=12341 HTTP/1.1" 200 8404 "www.mysite.com/movies.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30"

The red number “172.26.136.223” states the IP address of the computer that requested the page. Usually, a unique IP address indicates a unique user. However, some Internet Service Providers (ISP), such as AOL, rotate IP addresses among their users. In these instances, the IP address assigned to a particular user when they connected to the Internet could have been assigned to another user who disconnected just moments earlier. In addition, organizations like companies and universities use proxy servers to handle all of their Internet traffic. This means that, for instance, that all of the students of a university would appear to share the same IP address in your web server's logs even though they are in reality many different unique users.

The yellow string ”[19/Jul/2011:15:27:55 +0100]” refers to the date and time of the request. The blue string “/movies.html?id=12341” denotes the page that was requested.

The green string “www.mysite.com/movies.html" states the referrer of the request. Usually it denotes the webpage the user loaded before she loaded the page on your web server. Please note that without URL tracking, referrers generated from clicks on Google's organic search results and AdWords ads look the same. In some cases, there may be no referrer information at all. This can happen for a variety of reasons, for example when: the user's browser does not support it, the user deactivated the feature, or the user accessed the website directly by typing the URL into the addressbar of the browser.

The long purple string “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30" denotes the user agent of the visitor. It states the operating system and the browser version the user is using to access the website. However, this can easily be faked by the user.

Tracking Software

You may find reading raw web server logs too cumbersome, especially if they contain hundreds or thousands of entries per day. Fortunately, there is software available to do this task by collecting and then organizing the data into an easily readable format. One example of such software is Google Analytics. It uses JavaScript to gather information about the users of a website, aggregates the collected data, and presents it in a easily readable format.

If you want to use a third-party tracking software, select it with great care, because we have encountered erroneous data from some vendors in the past. In general, the most accurate third-party programs in terms of tracking AdWords clicks have been those which require the enabling of auto-tagging. However, please keep in mind that charges are based solely on Google's metrics. For more information see AdWords terms and conditions.

URL Tracking

URL tracking distinguishes between clicks that come from AdWords those which are from other sources like the natural search results from Google. In the absence of tracking URLs, the log of requests generated by a user who clicked on an AdWords ad and clicked an organic search result on Google.com might look like the following:

213.73.23.193 - - [19/Jul/2011:15:27:55 +0100] "GET /movies.html HTTP/1.1" 200 8404 "www.google.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30"
172.26.136.223 - - [19/Jul/2011:19:21:43 +0100] "GET /movies.html HTTP/1.1" 200 8404 "www.google.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30"

As you can see in the example, there is no way to distinguish between a click coming from Google Search and a click coming from AdWords.

The technique of URL tracking works by assigning unique destination URLs to the landing pages which are used in a site's AdWords campaigns. For instance, in addition to the usual index.html page, there should be a copy of that page with the same content, but with a filename of index_AdWords.html. In this example, any AdWords campaigns which drive traffic to the index page would use index_AdWords.html URL as their destination URL. After implementing this change, the corresponding log entries would look something like the following:

213.73.23.193 - - [19/Jul/2011:15:27:55 +0100] "GET /movies_AdWords.html HTTP/1.1" 200 8404 "www.google.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30"
172.26.136.223 - - [19/Jul/2011:19:21:43 +0100] "GET /movies.html HTTP/1.1" 200 8404 "www.google.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30"

As this simple example shows, basic URL tracking is very easy to implement. For even greater granularity it is possible to associate a unique landing page with each of the keywords in a campaign.

However, this method of URL tracking is not 100% accurate since the requests for these unique landing pages do not necessarily correspond to a click on an ad. This can happen for a variety of reasons, including when users: click on an ad and then bookmark the landing page, go past the landing page and return to the page by hitting the “back” button of the browser, etc. A more accurate tracking method is Auto-Tagging.

Auto-Tagging

Auto-tagging is a feature within Google Analytics and AdWords. The feature works by automatically appending a unique parameter to each AdWords click received by an ad. This parameter is called the “GCLID”. Each click will have its own unique GCLID. A log entry for a campaign using auto-tagging would look like the following:

172.26.136.223 - - [19/Jul/2011:19:21:43 +0100] "GET /movies.html?gclid=CLzkq4Lcz5ECFQcKGgodwjgiyA HTTP/1.1" 200 8404 "www.google.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30"

The red string is the GCLID parameter. If you have auto-tagging activated and configured correctly, you know that only the log entries that include the GCLID parameter are clicks from AdWords. Even if you find several entries in your log with the same GCLID, you know that you have only been charged for one click.

Using auto-tagging and Google Analytics allows you to thoroughly track the performance of your keywords because: you'll be able to see which keyword brought a visitor to your site, where the visitor is from, which campaign that keyword was from, and how much that click cost. You can easily calculate your return-on-investment (ROI) for a given campaign by associating this data with goals or e-commerce conversions. Based on that data you can make informed decisions on how to optimize your advertising campaign.

Common Misunderstandings and Misconceptions in Traffic Analysis

Some advertisers misunderstand their web log data, especially when they haven't enabled auto-tagging or are using third-party tracking software that lacks support for auto-tagging. The most common misunderstandings are related to:

  • Discrepancies in clicks and visits
    Misunderstanding the relationship between clicks and visits can result in apparent discrepancies between the information in your server logs and the information provided by AdWords. In general, AdWords tracks clicks, while many tracking services count visits. A click occurs when someone sees your ad and clicks on the title, leading her to your website. Visits, particularly unique visits, generally define the number of unique sessions initiated by visitors. However, there are several reasons why the number of clicks and visits may not match:
  • Short visit times and high bounce rate
    Many advertisers are worried about very short visit times (zero second visits) or high bounce rates (video explaining bounce rate) in their statistics. Neither metric necessarily means that the user left your site the instant they loaded it, merely that they left the site from the landing page without viewing any other pages. A high frequency of either of these metrics can signal that your site is difficult to navigate or that you've implemented your tracking code incorrectly, so don't simply ignore them, however.
  • Relationship between IP address and people
    A common belief is that a single IP address is unique to one person. A common question we receive from advertisers is: “I saw some entries in my logs from the same IP address. Does that mean I was charged for that many clicks from the same user?” However, there are a variety of reasons that log entries with the same IP address might belong to several different users. For example: some ISPs assign IP addresses dynamically, cycling through the same limited number of IP addresses with a much larger number of users as they connect to the Internet one after the other.

In conclusion, enabling auto-tagging is a simple, but highly effective change that you can make to improve how you track your campaign activity, because it provides insight into every single click and helps to navigate around the pitfalls associated with identifying unique users solely by IP address.

Video

To get a better understanding of your ad traffic you can also watch a summary of this article in the following video.