The Search

I just finished reading ‘The Search’ by John Battelle. What an amazing story to read. Learning about Bill Gross and his IdeaLab alone was worth reading the book, and he still keeps ideating, like Snap.com. Heck, even Picasa came from IdeaLab.

There are many tidbits from the book that were interesting, such as about Louis Monier:

It was Louis Monier who took AltaVista from concept to executable code … “I’ve always been interested in big, nasty problems,” Monier told me. Search provided one of the nastiest. Not only do the numbers scale to the near infinite, there was a very real need for good search in 1994. “Search engines at the time were just terrible,” Monier recalls. “Yahoo was a great catalog, but it had no search. So I set about to work on the crawl.”

About Stanford:

Stanford’s 6,200-acre patch of rolling California woodlands is the most productive incubator of technology companies the world has ever seen. Nestled between the silicon factories of Intel and Apple on one end and Sand Hill Road’s venture capitalists on the other, Stanford is a place where students have already dreamed of starting their own companies or going to work for a pre-IPO start-up. And Stanford’s computer science department, where Yang and Filo hung their hats, is perhaps the most prodigious start-up incubator of them all.

About Yahoo:

Another reason Yahoo succeeded was its sense of fun – a characteristic that would come to define not only Yahoo, but nearly every Internet company seeking the fickle approval of the Web public. Yahoo pioneered some of the Web’s earliest social mores – including, for example, links to competitors’ sites in case a searcher could not find what he or she was looking for, and listing “what’s hot” prominently on its home page, thereby driving extraordinary amounts of traffic to otherwise obscure sites.

Thanks to practices like these, the company captured the public’s imagination early and often, garnering a slew of adoring press notices familiar to anyone watching Google’s rise to prominence over the past few years.

About how a mathematical curiosity led to PageRank:

Page didn’t land on the idea of Web-based search at the outset; far from it. Despite the fact that Stanford alumni were getting rich starting Internet companies, Page found the Web interesting primarily for its mathematical characteristics. Each computer was a node, and each link on a Web page was a connection between nodes – a classic graph structure. “Computer scientists love graphs,” Page tells me, referring to the mathematical definition of the term. The World Wide Web, Page theorized, may have been the largest graph ever created, and it was growing at a breakneck pace. One could reasonably argue that many useful insights lurked in its vertices, awaiting discovery by inquiring graduate students. Winograd agreed, and Page set about pondering the link structure of the Web.

About Google’s geeky sense of humor and control:

On April 29, 2004, Google filed what certainly had to be the most unusual S1 – the formal public offering document – in recent memory. At filing, Google declared it would sell $2,718,281,828 worth of its shares – a seemingly random number, which was, in fact, the mathematical equivalent of e, a concept not unlike pi that has unique characteristics and is well known to serious math geeks. By manipulating the actual offering to provide this knowing wink to nerd humor, Google was in effect declaring: the geeks are in control.

Perhaps, the most interesting part of the book for me was the last chapter – ‘Perfect Search’. Battelle profiles what could be the future of Search.

When it comes to search, as with the Internet itself, the most interesting stuff is yet to come. As every engineer in the search field loves to tell you, search is at best 5 percent solved – we’re not even into the double digits of its potential. And search itself is changing at such a rapid pace – in the past year important innovations have rolled out once a week, if not faster – that attempts to predict the near future are almost certainly doomed.

I’ve been working on the Yahoo! Buzz Index for the past 2 years, and many a time I’ve been asked (by friends and colleagues) why I haven’t changed teams yet. But I often ponder to myself – change to what? Being a rabidly information-hungry internet user (well, I’ve calmed down off late), I always found search engines remarkable and Buzz does a lot of analysis on search, it’s quite fascinating, and the sheer volume of data is equally interesting. I’ve had my share of ups and downs (and some very steep downs), but it has been interesting.

We do a lot more than what Google Trends does, however Buzz has a more practical business model in which the interesting insights are kept for the paid customers and the interesting stories are written for the public.

Isn’t it striking on how the ability to search for what you want  (and relatedly, blogs have become an important factor in this) has made us more knowledgeable? All the articles on productivity that I came across have really helped me improve in my professional and personal life. My problem has always been converting thoughts into actions, but GTD and the ready-fire-aim approach are helping me improve, but that’s a whole different story. Search has helped me find lyrics of so many songs. Search helped me research about the places I travelled to and want to travel to. Search has helped me discover some hands-on activities that could help me teach the Parikrma kids to explore and think about science, the earth and the human body. I could go on, but just count the number of times you use a search engine in a day, and you’ll know what I mean.

All this has got me more interested in the Search-related domain as a field to work in. And all the money that Search makes is interesting as well ;-) Will I ever move on to working on something more in this direction, or will I find something more interesting tomorrow, or will I simply remain yet-another-software-coolie (which seems more likely)? I honestly don’t know, but don’t we all have a search(!) for “that one thing that drives us”?

I’m not saying that Search is the important problem out there in the world, far from it, there are more real world problems like our community, our economy,  and quality of life. I believe that knowledge is an important way of empowering people, and Search is a step in helping people to keep finding and gaining knowledge.

I’ve been thinking about all of this, especially, because of Richard Hamming’s words:

Over on the other side of the dining hall was a chemistry table. I had worked with one of the fellows, Dave McCall. I went over and said, “Do you mind if I join you?” They can’t say no, so I started eating with them for a while. And I started asking, “What are the important problems of your field?” And after a week or so, “What important problems are you working on?” And after some more time I came in one day and said, “If what you are doing is not important, and if you don’t think it is going to lead to something important, why are you at Bell Labs working on it?” I wasn’t welcomed after that; I had to find somebody else to eat with! That was in the spring.

In the fall, Dave McCall stopped me in the hall and said, “Hamming, that remark of yours got underneath my skin. I thought about it all summer, i.e. what were the important problems in my field. I haven’t changed my research,” he says, “but I think it was well worthwhile.” And I said, “Thank you Dave,” and went on. I noticed a couple of months later he was made the head of the department. I noticed the other day he was a Member of the National Academy of Engineering. I noticed he has succeeded. I have never heard the names of any of the other fellows at that table mentioned in science and scientific circles. They were unable to ask themselves, “What are the important problems in my field?”

If you do not work on an important problem, it’s unlikely you’ll do important work. It’s perfectly obvious. Great scientists have thought through, in a careful way, a number of important problems in their field, and they keep an eye on wondering how to attack them. Let me warn you, ‘important problem’ must be phrased carefully. The three outstanding problems in physics, in a certain sense, were never worked on while I was at Bell Labs. By important I mean guaranteed a Nobel Prize and any sum of money you want to mention. We didn’t work on (1) time travel, (2) teleportation, and (3) antigravity. They are not important problems because we do not have an attack. It’s not the consequence that makes a problem important, it is that you have a reasonable attack. That is what makes a problem important. When I say that most scientists don’t work on important problems, I mean it in that sense. The average scientist, so far as I can make out, spends almost all his time working on problems which he believes will not be important and he also doesn’t believe that they will lead to important problems.

I spoke earlier about planting acorns so that oaks will grow. You can’t always know exactly where to be, but you can keep active in places where something might happen. And even if you believe that great science is a matter of luck, you can stand on a mountain top where lightning strikes; you don’t have to hide in the valley where you’re safe. But the average scientist does routine safe work almost all the time and so he (or she) doesn’t produce much. It’s that simple. If you want to do great work, you clearly must work on important problems, and you should have an idea.

Back to the book … As a colleague said, this book should be made compulsory reading for anyone working in this field.

Also, special thanks to Rajiv Vyas for sending the book across the ocean to me just because he thought “I might want to read it”! … come to think about it, how did Rajiv and me become friends? Well, he searched for something and chanced upon my blog … :-)