Cognitive Search Starts with Understanding Search
Cognitive search, and indeed, the entire new wave of cognitive applications, are the next leap forward in information access. These apps rest on a search backbone that integrates information, making it findable and usable.
Search engines were the first probabilistic software applications in widespread use. Although not widely recognized as such, they have groomed a generation of information seekers to accept ranked results as well as to enter probes, rather than precise queries. While this approach helps users flounder around on the Web as they look for “acceptable” or “approximate” answers, it is often inadequate, by itself, for the many other uses of search on which enterprise users have come to depend.
At heart, the strengths of search rest on two pillars:
- First, an index structure that enables quick updating without a schema
- Second, similarity matching rather than exact matching
Similarity matching is an unsung hero. It can be used to match terms, meaning, images, data, whatever, by identifying the raw information elements in a body of information, no matter whether they are pixels, letters, pictographs or sounds.
For any cognitive application to succeed, defining the problem is the hardest part. Cognitive search is the same – understanding the type of question and the type of search application is essential to providing good results. Search isn’t all one thing, and in fact, trying to view it as one thing gets in the way.
Different Classes of Search Applications
In broad terms, search applications may be divided by their purpose or intent:
- Find me all the relevant answers that match my query
- Find me all the relevant answers even if they are partial matches to my query
- “Where’s that thing?” – find a known item, wherever it may be located
- “What’s in here?” – help me browse (preferably visually) through large collections of information to find out what they contain
- Understand my purpose in asking a question within the context of what I am doing, what I have done already, and what I hope to accomplish within the context of my role and the goals of the organization
- Become a partner in information finding and exploration, making recommendations and helping to refine queries, problem statements, or asking pertinent questions, much as a human partner would.
Traditional search engines have addressed the first four of these, with varying success, depending on the degree to which knowledge bases, rule bases and language understanding have been embedded.
Where traditional search falls down is in the last two categories, which require a more sophisticated approach. That’s where cognitive search comes in, building on the probabilistic framework of traditional search, but adding:
- Ability to learn
- Context as a filter: user preferences, role and history; geographic background and relationships; time and history for each information project; business goals
- Automatic metadata extraction rather than simple, more brittle rule bases or dependency on manual tagging
- Ability to normalize across data sources and infer relationships no matter the source or format
- Inferencing to uncover possibly unknown but relevant patterns based on use of context
- Conversational, open dialog—give and take—with the user
Where is Cognitive Search Going?
Today, we already see elements like these embedded in advanced search engines, turning them into partners in information discovery. As they become more multifaceted, we see use of search expanding. We have identified two different types of cognitive search systems. The first delivers highly relevant information within the context of the individual’s information needs of the moment. The second finds patterns and surprises—a smart serendipity machine.
- Cognitive search is becoming a layer rather than a standalone application. It will be used in several different ways:
- As a backbone for cognitive applications—a platform that is highly integrated with modules for connecting to all kinds of sources and formats, image recognition, emotion detection, etc.
- As a gateway to existing information infrastructure to help knit together sources
- As an underlying knowledge base and storage mechanism
- As the basis for personal assistants that are domain-based and conversational
In my column “cognitive applications in real life”, I outlined the flowering in experimentation, in new products and in new services over the last year. We are still early in the journey to cognitive search, and as an industry, we are beginning the important move from hype to reality. Companies such as BA Insight are already able to not only provide better search results, but also uncover patterns and solve problems that traditional search engines can’t.