The value of simple metrics closely watched
(Warning: At the end of this post I'm going to say "It's a journey, not a destination." Whenever I say this at home, my kids run out of the room. When I'm with software developers there is a pause, and then conversation continues as if I hadn't said anything. Nevertheless, I continue to say it.)
Marc Hedlund has an interesting post about some useful Web 2.0 development techniques. I agree with many of them, but the practice of creating internal metrics based on website stats is not a Web 2.0 innovation. We followed the same policy at Andover.net in the late Nineties with a statistics system we built ourselves. Other than the usual web traffic values, such as page views and unique visitors, we had a few ratios that became the basis of almost every design and development decision we made. These metrics were displayed on a continuous graph, so we could see the trend over time. One of my favorites was pages per visitor (PPV). Whenever we redesigned a site, that was the first metric I checked. Since we made our money from advertising, the more pages people viewed the more ads we sold. We convinced advertisers (this was in the stone age, when websites sold ads directly to advertisers) that a higher page per visitor count meant more committed readers. Pages per visitor was also an indirect measurement of server performance, and ease of site navigation; a slow or confusing site had a low PPV. So the PPV was a way of measuring the user's approval of the website experience.
It might seem that a simple way to raise the PPV would be to chop up articles into multiple pages, but that would backfire. Users would suffer from fatigue, and leave earlier than if articles were displayed on a single page. We could prove this by testing the design and then watching the PPV. We did, however, discover the value of previous and next links on every page, which led to additional articles or whatever content the site contained. That simple change could sometimes triple a site's PPV. We called it the "peanut principle," substitute your snack of choice. Another obvious way to boost PPV was to add an archive of past content. None of this is rocket science, but it meant that we could acquire new sites that didn't follow these principles, and be reasonably sure of doubling or tripling the traffic overnight with a few design changes.
Another great example of using a simple metric as the indicator of progress was when we acquired Slashdot. We quickly discovered that the site was failing due to server overload. Unfortunately, there was no way of monitoring the code, so we couldn't tell where it was failing. We knew that the MySQL database was rolling over, so a new database architecture was a first priority. At the same time that I assigned people to the task of rebuilding the schema, I also had one person dedicated to building a monitoring system with a continual graph of database metrics that anyone on the team could reach on a webpage. The most important metric proved to be queries per page, which were extraordinarily high when we started. There was one memorable day when we hit the switch to introduce a new version of the Slashdot code, and this graph of queries per page dropped like a rock. We all cheered, and the coder who had designed the new database schema pasted that part of the graph on his door. This simple metric had not only allowed us to see our progress, it gave the people responsible for fixing it a tremendous amount of pride.
While all of this may seem like common sense, it is funny how coders often react cynically when I tell them about it, as if this were a marketing trick. I had lunch recently with a group of young programmers, and when they asked me how to scale a database to handle more website traffic, I told them this Slashdot story. I then explained that scaling was a journey, not a destination. They had to find a way to see their progress, not just look for an endpoint. There was a pause, and then they started talking again as if I hadn't said anything.
