Riding the Rails Through a TV Ad Campaign

Over the weekend, we had one of our flagship sites, greatlittleideas.com be subjected to a TV advertising campaign laid on by the site owner Premier Foods. This consisted of three days of advertising over a weekend across a range of different ads and channels ranging from 10 to 20 seconds in length.

Now, initially this doesn’t sound too scary, except that we were seeing projections that were predicting thousands of users visiting the website as a result of these ads. What’s more, they were all going to arrive at the same time which meant that we had to think about handling massive peaks in traffic.

The results were fascinating and the site stayed solid as a rock, but there were a lot of lessons that we learnt that are all described below. But first, here are some stats:

Normal traffic for the site is in the 20-40 requests per minute range throughout the day. The site is Rails 3, running against PostgreSQL and using Memcached for some fragment caching. More importantly this site is hosted on Heroku, the EC2 based rack hosting start-up that’s just been bought by SalesForce.com. Behind all of this we were monitoring the stack with New Relic’s RPM.

During the TV advertising we saw peaks in the 5600 requests per minute ballpark (an 18000% increase in traffic) but no downtime. This, for us, was massive load over normal, and one that required a little babysitting to ensure that users were always given fast and responsive pages. At no time did we want users to be waiting for a page to load. For us the site had to maintain it’s fast, snappy pace.

So, what did we learn through this weekend? Well, quite a lot of things:

Your URL is irrelevant

One surprising thing we noticed is the proportion of people came to the site from search engines. Given that the adverts in question were advertising the website, and emblazoned a massive URL across the screen using the most obvious URL format possible (greatlittleideas.com), two thirds of the traffic came to the site via search engines such as Google having searched for the term ‘Great Little Ideas’. What’s more, this is a 93% increase in the amount of traffic that usually comes via this source, which tells us that TV advertising is good at getting a brand into peoples’ heads, but not any particular details (like a URL).

People come, and come straight away

The most fascinating thing for us was the immediacy of TV advertising. From an advert going out, we would see a traffic peak within two or three minutes. This suggests that people, when they see something interesting on the TV, will immediately hit the web looking for the site, and this will continue. At peak times, we were seeing the site go from 50-60 requests per minute to over 4000 within 120 seconds of an ad, quite simply an astounding traffic increase that lots of traditional platforms would have trouble handling.

Another thing that we found surprising, was the long tail of traffic we would get after an ad was shown. For instance, we were expecting a sharp spike of traffic that dropped off when people went back to watching whatever it is they were watching. However, we found after any ad that you could easily expect an hour or two of elevated traffic which had to be accounted for.

It just goes to show that whatever your predictions of human behaviour are, they’re most likely wrong.

TV Advertising is not predictable

One thing we did have in our arsenal prior to the advertising was a timetable of what ads were going out when, and on what channels. This meant we were roughly able to predict when the peaks would be and to what extent. For instance, a 20 second ad at prime time on a Sunday was worth five or six ads placed during a soap on a Friday early afternoon. However, one thing we found was that the TV companies don’t always stick to the timetable. Some ads were going out an hour late, some weren’t showing at all, and some were showing at such random times that we don’t know who showed what, when. One thing we do know though is that prime time is simply massive in terms of gathering engaged users.

If you need to place one ad, that’s the time to do it.

TV Regions are not one and the same

In the UK, ITV has different output for the different regions of the UK. Normally this is only an advertising difference, but sometimes the programming will change too. At one point we had an ad due to go out across six or seven regions simultaneously. On time, the ad went out and we started receiving some additional traffic. However, I didn’t see the ad in my local region which was puzzling me somewhat. An hour later we saw the ad and traffic went up massively, double the previous peak.

After thinking about it and looking at the stats it turned out ITV London put the ad out an hour late, but also generated twice the traffic of the other regions put together.

So lesson learnt, TV Regions yield very different results.

Having ‘infinite’ capacity is awesome

As mentioned earlier, we host all our our larger Rails stuff on Heroku, the EC2 based rack hosting start-up that’s just been bought by SalesForce.com. One of the reasons for this is the way that the hosting works from a scaling point of view.

Using Heroku, we are able to take our available resources from the equivalent of one physical server, to over a hundred with a simple console command, and those new servers are up and provisioned within a couple of seconds.

This is a massively handy tool to have when dealing with the sort of peaks that TV brings, and not something that is practical using traditional hosting methods.

For instance, at peak we were running on 45 dynos (roughly equivalent to 12 physical servers). Had we wanted to host this using our own physical hardware, we would have been looking at purchasing around £20,000 worth of hardware, the associated rack and installation costs, plus the bandwidth required – all to serve a 10 minute peak.

With Heroku we were able to do this with very little outlay (sub $5) and no additional costs or lead time to worry about. Having seen Heroku under load, I estimate that if needed, we could scale to around 1000 requests a second on Heroku with no application changes required.

Performance tuning for the most part isn’t worth it

A lot of developers will spend ages wringing every bit of performance out of an application, trying to shave off 10ms here, 50ms there, getting the page as fast as possible.

We found that given the scalability of Heroku, this isn’t really worth it. It’s a lot simpler to increase capacity for the short term, than spend ages tweaking the code.

However, this only applies to peaky style traffic. If you’re sustaining high load, performance improvements will certainly help you reduce your hosting bill, but not really make any difference in the eyes of the user as they won’t be able to notice such small improvements.

New Relic RPM is a must-have tool for hosting Rails applications.

New Relic RPM is a tool that we use for monitoring the performance of our Rails applications. It tells us all sorts of statistics regarding throughput, response time, and breaks the requests down to the different layers and components of the application in question. Without RPM you’re effectively flying blind, not really knowing where the bottlenecks might be or how much traffic you’re pushing out.

However, in the rapid peaks scenario we were experiencing, we found a shortfall of the product which is the latency between receiving the traffic, and it showing in RPM (around a minute or so).

Normally this wasn’t an issue, but a couple of times we had a queue of users to deal with as we hadn’t provisioned sufficient horsepower to process the requests. Only once RPM reported the load could we react. To get round this we added the Heroku queue depth to our source so we could hit the website in real time, assessing the response time, but also seeing the state of the queue.

Saying this though, RPM is a must-have tool, whether you’re hosting large or small.

In Summary

So, all in all hosting a website through such substantial peaks is an interesting experience, but also one that’s perfectly achievable if you put a little thought into the basics: What tool are you going to build in? What method of hosting are you going to use? What metrics do you have to try and predict the numbers you’ll be seeing? How accurate are these?

However, this style of hosting is very different to that which is a website with sustained high traffic. For this sort of environment you’ll need to focus more on performance, cost, and reliability, but that’s probably another post for another time.

Posted by Neil Middleton on 17 Jan 2011