This is a bit of a morality tale. A lá David and Goliath. But bear with me. I promise there might be a point at the end of this.

Once upon a time, there was a mighty giant called Yahoo. It ruled the internet, being the gateway and gatekeeper for much of the online world. If anyone wanted to go anywhere, they went to the Yahoo god and prayed before its altar for knowledge. This gateway used to be called a “PORTAL”. From this web portal, the internet user was allowed to search for that which he needed, and if he were especially lucky, he would find it, albeit after much searching and swearing.

Then in early 1996, a couple of young Stanford students, Larry Paige and Sergei Brin stumbled on to a research topic. They wanted to see what the relationship of each website was with another. Halfway through their research, they had what would be called their “aha! moment”. Up to now, search engines trawled the web, looking for the search string that people had typed. The more times the search string appeared, the higher the ranking of the web page was.

But Larry and Sergei had a different idea. Their idea was this, ‘If a website has the text that is being searched for, its not enough. It should also be linked to by other sites…which means that lots of websites think that this page is good. IF those websites are also linked to by lots of other websites, then its even better! It means that the page that is found is really a good page and should be the one the user wants. This was the genisis of their holy grail…the “Page Rank Algorithm” (enter flourish of trumpets).

This was a far better way to search than the existing web portals. Yahoo immediately recognized this, and invited Larry and Sergei to take over Yahoo’s search for a fee (the exact number is not that important). Larry and Sergei registered google.com, and then history as we know it changed forever.

In just about 10 years, Google moved from being two scruffy Stanford students to a several billion dollar teddy bear that vows to “Do No Evil”. At the same time, poor Yahoo, from being the web portal of the world languished in web purgatory, first being hailed as a stalwart in the dot com killoff, and then pilloried for being a fuddy duddy in an age where wonderous phrases like web 2.0 and social networking were being bandied around. Today one and all worships at the altar of Google, which has those funky text ads and makes money by truly not doing any evil. The god that was Yahoo is now on the backfoot fighting off another ageing suitor in Microsoft.

So what does this little David vs Goliath say? To me, it seems to make a point that always seems to happen. The big companies seem to have a serious problem adapting to change. Sure, there are exceptions. But every large company had to start small. It started off with something new, what the strategists call “competitive advantage”. But over time, this “advantage” begins to erode. For Yahoo, the advantage of being the 900 pound gorilla eroded when Google came up with a radically different way of searching for things on the web.

This actually seems a bit strange. After all, Yahoo had a heck of a lot of money. It should have been able to research a new search algorithm method much easier than two (possibly) cash strapped young students. But it did not. And this seems to be the case more often than not. Large companies have the money, the talent and the drive…but they somehow do not seem to innovate as well. Why?

One reason that comes up is comfort. For Yahoo, they were comfortable with their way of working. They believed in something called incremental innovation. They tried to make their search technology work faster and faster by crawling through the web searching for words the user typed faster. So they kept investing money and talent and drive…but diminishing marginal returns was setting in. A user does not really care if the search engine returns results in 0.1 seconds or 0.01 seconds. He cares if he gets what he is looking for!

Yahoo was delivering faster results. But Google was getting better results! This is called disruptive technology. It was like cassette players versus CD’s. There was just no comparison. CD’s won every time!

Now, Google is faced with a challenge. They disrupted the entire search industry with their funky search algorithm. Are they going to rule forever? Unless they keep running, probably not. The reality is that Google is still not perfect. Someday, a better search algorithm will come out. The probability is that Google will be busy making its page ranking algorithm more and more efficient. Some disruptive type thing might appear and totally wipe Google out. In order for Google to actually come up with another disruptive technology, it needs to throw money and talent at seemingly fruitless lines of research that might have NOTHING to do with better search engines.

Its a tricky tight-rope to watch. Even google does not have enough money to spend money researching all sorts of things. But Larry and Sergei know that if they only spend their time perfecting page ranking, they will end up beaten to the gun by a more nimble competitor.

One solution that the strategy books give is to continually scan the world around us. The instant anything that looks like a promising search tech. arrives, Google has to buy it out, paying exhorbitant sums of cash for it. If they plan well, then they can still maintain their advantage…if they don’t….they go bankrupt.

Another way to go is to madly research away, and hope like hell that something you do clicks. It does not have to be related to search. Maybe a better mail client, or a chat client…or even a mobile phone! :P Again, the worry is that madly researching diverts attention from the avenue that actually makes the company money…and that is search, and text ads.

So lets watch out who the next David is going to be. Will it be the return of the Yahooligan? Or will there be a new player altogether, crowding out both grandpa and the teddy bear?