Search engines learning to anticipate user's needs

----------------------------------------------------------------------
This article was sent to you by someone who found it on SFGate.
The original article can be found on SFGate.com here:
http://www.sfgate.com/cgi-bin/article.cgi?file=/c/a/2010/10/17/BUOI1FSPE5.DTL
---------------------------------------------------------------------
Sunday, October 17, 2010 (SF Chronicle)
Search engines learning to anticipate user's needs
James Temple, Chronicle Staff Writer

The next time you get a helpful search result from Google, you might offer
a nod of gratitude to a depressive Austrian philosopher named Ludwig
Wittgenstein.
In a book published posthumously in 1953, he argued that words are so
pliable as to lose most meaning outside of a sentence. The word "hit," for
example, doesn't conjure up a distinct mental image until neighboring
words like "movie," "baseball" or "the hay" tug your thoughts in the right
direction.
Google Inc. Fellow Amit Singhal studied Wittgenstein while working on his
doctorate in computer science at Cornell University, and integrated that
idea into a major reworking of the company's core search algorithm
unveiled in 2003.
It wasn't enough, he concluded, for Google to match up keywords in queries
to the pages on which they're found online, because those words mean
little in isolation. To provide truly relevant results, the Internet
giant's machines had to begin to understand language the way that humans
do.
"That basic, key insight was the driving force behind the last seven years
of work that we've done, which has gone way beyond anything I have ever
seen in the academic world," Singhal said.
To accomplish this, Google employed machine learning algorithms,
essentially artificial intelligence programs that study patterns in the
vast quantities of human writing across the Internet, to figure out the
signals that suggest specific meanings of words.
Other companies and research institutes, including Microsoft Corp., SRI
International, Wolfram|Alpha and Yahoo Inc., have applied similar methods
to the problem of online search. As machines have become better at
understanding the mechanics of human language, the quality of what we
think of as search has vastly improved, while the very notion of what it
means has expanded.
In fact, some believe computers will soon be capable of addressing our
needs and wants before we've articulated them in our own minds.
The term online search suggests that people are just looking for
information, but in many cases they're hoping to complete tasks.
For instance, if a user types or says "Oakland restaurant" in a search
engine, often what they want to do is make reservations. But search-engine
results typically start with a list of possible establishments. The user
can then scroll through those, hop over to Yelp for reviews, and click
onto OpenTable to see if they can make reservations.
Siri, an artificial intelligence iPhone app developed by SRI and later
bought by Apple Inc., leapfrogs some of these steps. When a user says or
types "Oakland restaurant," a list of spots pops up with ratings, prices
and buttons that say "reserve table."
Over time, as the tool learns more about you - your location, preferences
and habits - it can provide increasingly personalized results. It might
know, for instance, that you favor fancy restaurants, French bistros or
Korean BBQ.
Siri's founder declined an interview request, but SRI continues to work on
virtual personal assistants that address this shortcoming of search
engines.
"With most of the systems right now, we express it piece by piece. It's up
to us, in our head, to make sure the pieces all flow together and at the
end combine to meet our intent," said Bill Mark, vice president of
information and computing sciences at SRI. "We're moving to a world where
the technology does a better job of understanding higher level intent and
completes the entire task for us."
Similarly, a search for "flights to Boston" on Microsoft's Bing search
engine returns not just links to travel sites, but a box on the results
page that allows users to enter flight dates and the departure airport to
directly pull up times and prices. Bing also moves its local, video or map
results to more or less prominent parts of the page, depending on how a
user worded a query.
In both cases, the algorithm is making decisions based on its improving
understanding of how humans use language.
Bing "is taking things to another level in terms of anticipating what
users really want," said Jan Pedersen, chief scientist for core search at
Microsoft. "There's a lot of ambiguity in the queries, but there's usually
a very particular meaning that any particular user has." 'Autonomous
search'
But such a leap might occur even without parceling out queries, if
machines can learn to anticipate our desires based on other facts they
collect.
As smart phones become increasingly ubiquitous, more and more of us are
spending our days tethered to constantly connected gadgets that know:
where we are at any given moment, the entries in our calendars and to-do
lists, the albums we listen to the most, the headlines we read, who our
friends are and where we shop.
There's an enormous amount of predictive power in all that data, if
companies can connect even parts of it together. And it can be employed in
a lot of different ways.
Singhal said it could help effectively schedule time. For instance, a
smart phone could let users know when there's a store nearby where they
can check off a task, if their calendar shows they have the time. The
device would react to shifting circumstances - location and time, as well
as entries on a calendar or to-do list - rather than a typed search query.
Google Chief Executive Officer Eric Schmidt described a similar scenario
last month at a conference in Berlin, in which a history buff like himself
automatically receives information from his handset about exhibitions at
the museums he walked past in Germany.
He dubbed the concept "autonomous search."
"It knows who I am, it knows what I care about and it knows roughly where
I am," he said of the smart phone. "The ability to tell me things I didn't
know but I probably am very interested in is the next great step, in my
view, of search."
He and Singhal both stressed that Google would ask permission from users
before switching on such applications. Seductive systems
But others are dubious of - or frightened by - the possibility.
Jeff Chester, executive director of the Center for Digital Democracy, said
it's more akin to the next great step in targeted marketing. He believes
that new rules must be established that take into account the accelerating
collection and prediction abilities of these technologies.
At a minimum, privacy advocates say, consumers should have to opt in
before any information can be collected about them and must be made aware
of how such personal data might be used - standards that haven't been put
in place for such information to date.
"There needs to be a very serious public debate about what the proper role
and safeguards should be for the emergence of this ubiquitous, artificial
intelligence-based world," Chester said.
"These are very powerful and seductive systems," he added. "But you're
trading off your autonomy and ultimately your security for the ease of
allowing someone else to make decisions for you and set your agenda."
Ray Valdes, an analyst with Gartner Research, said autonomous search
remains a fairly far out technological possibility at this point.
He also stressed that the advances in online search realized from machine
learning have been tempered by the increasing complexity of search.
The amount of online content has exploded, and consumer expectations have
soared. They want to find pictures, video and music that often aren't
accurately tagged with language. They want to pull relevant information
from social sites like Facebook and Twitter. And they want to know what's
on a page even if it's written in French or Spanish.
"Search engines have gotten better, but at the same time their tasks have
gotten more difficult," he said.
E-mail James Temple at jtemple@sfchronicle.com. ----------------------------------------------------------------------
Copyright 2010 SF Chronicle