In Defense of Our Optimism | BA Insight

Jeff Fried

Steve Arnold wrote an article “BA Insight on the Resurrection Trail” yesterday,which was a comment on our Joel Oleson’s article “Artificial Intelligence Is Resurrecting Enterprise Search”.  While I’m a fan of Steve’s and enjoy his style as self-described “curmudgeon”, I strongly disagree with his take on a number of points.  I couldn’t resist picking a fight with him,so here’s my side of the argument:

On a key point, Steve’s article said we’ve proclaimed Enterprise Search dead.  On the contrary, I adamantly believe that our field is having a resurgence of innovation and is delivering substantial value. Joel’s article says in the first sentence, “The future of enterprise search is bright.”  Steve’s article talks about the ennui in the industry, the many dead or struggling ventures, and how new initiatives are doomed to fail the same way past ones have.

We Need Optimism, Not Ennui

In the big picture, the spirit of Joel’s article is noticing and recognizing some exciting trends in the industry. I definitely feel that we need new growth, new trends, and new interests in the possibilities of the technology –  if for nothing more than to muster the energy to succeed with projects. In the enterprise, frustration with finding important information is not abating, in fact it’s getting worse – as evidenced by Findwise’s annual survey (which was released today).

Steve’s article says, “What the heck does this have to do with advanced software methods for information retrieval? Not much.” Agreed, it is not about advanced algorithms. There is plenty of interesting and advanced stuff in the work we reference, but an obsession with magic new algorithms and an overhype of them has hurt our industry.  The core algorithms have not progressed significantly; search cores are commoditized and even learning algorithms are getting mainstreamed.  Many companies are stuck in the “search immaturity cycle” and need help applying this technology, and they are doing the homework they need to do.   We should put focus on successful applications and the products and practices that support this.

It is possible to field great search, if people put enough value on it to pay attention to how and where to apply it. This is true of traditional keyword search and equally true of the emerging user paradigms and applications our article discussed.  Our industry is trying to tackle an extremely hard problem, and there’s lots of innovation still to be done, for sure.  We agree with Steve that Elasticsearch is part of that innovation and is thriving – that’s why we are working with it, too.

But no technology or search core will succeed without good practices and effective applications.  People need to see that search can be good, understand that it can be different than their current experience, and be interested enough to pay attention and resource projects to succeed – not just plug things in and walk away.

Personal Assistants, AI, and Delve

Personal assistants are not a new idea; there have been many incarnations over the years.   And they will never be clairvoyant nor universal – we agree.  But that doesn’t matter.  What matters is that they are hitting the mainstream – undeniable if you watch your kids using Siri, and clear from the industry events our expert talked about.

Delve is not the first attempt to provide you with information without your explicitly asking for it; the Office Graph is not the first graph search, and there are many dead bodies by the side of the road – we agree.  But we’ve been working with Delve for months and the key is the access to content.  Steve says “this Delve stuff is sort of smart, but until the product ships and provides access to a range of content types..”.  Well – the product shipped from Microsoft on Tuesday (the 8th), and we at BA Insight are working to provide connections to it from a wide range of content outside of Office 365, from outside of the Microsoft sphere altogether, and from outside the enterprise as well.  No organization of any size has all of its content in one place or all important information inside its boundaries.

Is “AI an oxymoron”? Amusing to say, sure, and as a field it certainly suffered from an enormous overhype and consequent fallow period – even more than our text analytics/NLP/search field.   But tell that to the people building self-driving cars or semi-autonomous robots. Look at the prevalence of machine learning. Or just look at the journals of the AAAI today.  That old joke is stuck in the ’90s.

As for Steve’s assertion that “Microsoft will be in the same rubber raft” as previous failures of predictive or learning algorithms, this comes back my point that it’s more than the algorithms.   Sure the algorithms and use of machine learning are ‘cool’ – but for any of these types of algorithms the output depends on the data available as input and the features chosen, more than the algorithm itself.  Speech recognition technology has steadily improved over the years primarily because of more data, not fundamentally different algorithms. Delve’s embedding in Office 365 and its basic premise of tapping information where you use it and leveraging your behavior in Office gives it a real fighting chance.  Hence the built-in connections to your calendar, your and your teammates’ documents, your email, your OneDrive and Yammer networks.  “How will a SharePoint centric solution know I need that information?” Steve asks.  Well, for example, when I have a meeting scheduled on my calendar, the subject and the invitees are pretty strong clues to relevant stuff I might want to know. The documents the invitees have been working on that relate to the subject are good candidates to push over to me.

Let’s Debate

It would be fun to have a real debate about the future of the industry and to what degree the trends in our article are overhyped versus being leading indicators.   I’ve invited Steve to debate anytime.  I would also love to hear from any readers out there – tell us how you view the health of the search industry, and how much you think the new breed of search-driven application is overhyped.  I can be reached at

We at BA Insight are still excited about the possibilities in the industry and what can be done.

Leave a Reply

Your email address will not be published. Required fields are marked *

For security, use of Google's reCAPTCHA service is required which is subject to the Google Privacy Policy and Terms of Use.

If you agree to these terms, please click here.