Top 3 SharePoint / FAST Search Implementation Mistakes
Mistake #1 - Failure to connect to backend systems

This particular topic generated quite a bit of interest when I presented at the New York, and Washington D.C. SharePoint Saturday events that recently occurred so I thought it made sense to summarize the webinar here.

Many people were surprised to learn that users often complain about enterprise search tools despite the high marks that they garner from Analysts such as Gartner. The analysis from Gartner below was conducted back in 2008 and pegged Microsoft as the leader in Enterprise Search. Microsoft has remained there ever since.

So with Gartner declaring Microsoft's search technology state of the art, how is it that studies are released on what seems to be a weekly basis, proclaiming there is an across-the-board dissatisfaction with Enterprise Search? Take the following two quotes released in June, for example:

• More than half (52%) of respondents say they cannot find the information within what most define as an acceptable amount of time - Mind the Search Gap - June 2011
  
• "Little wonder most enterprise search systems generate dissatisfaction levels among their users of 50% to 65% and even higher." - Steve Arnold - June 2011

A 50% success rate, might be a fantastic batting average, but anyone would be hard pressed to declare a successful search project with those kind of numbers.

There are many factors that can impact the success of an I.T. Project, but having seen hundreds of deployments three factors come top of mind:

• Failure to connect into backend systems
• Poor user interfaces
• Poor relevance

As I indicated in the title, I will take a stab at the first challenge, L.O.B. Systems.

According to Gartner, the average company/organization has at least six line of business systems supporting operations. Yet rarely are these included in the scope of a search initiative. The first question that begs to be asked, is why in the world would you want to include them? I believe the answer to that is to consider what happens if you don't. If half the company values data in these systems and your search doesn't help these individuals, naturally they won't see value in your project.

A second question which is related to the first is: Why would we want to use SharePoint or FAST Search, when line of business systems already have embedded search engines? The answer to this question is that indeed they do, but generally speaking, they are quite poor. The reason for this, is that search is not core business for these companies and they are not willing to make the investment needed to bring search to an acceptable level. In contrast, Microsoft has already spent $1.2 Billion acquiring top tier search vendor, FAST Search and Transfer and continue to invest heavily. Search is strategic to Microsoft and they will continue to invest heavily in this space.

The final question which one would naturally ask is: "If connecting to line of business systems is so important, why doesn't Microsoft sell connectors?" The answer to this is two fold. For starters, Microsoft would prefer if you migrated the data, rather than connected to it. Many of these line of business systems compete with Microsoft products, after all. More significantly, however, is the technical difficulty associated with developing connectors. Each class of connector has different characteristics (ERP, ECM, CRM, Messaging, etc) Microsoft does not want to go down that road. Rather, what they have done is develop a connector framework called Business Connectivity Services (or BCS for short), and left this as a partner opportunity.

Now before you go running out the door to build a connector.  You should know that connector technology is not trivial. Microsoft's BCS can be used to easily connect and surface data in simple systems.  Enterprise Systems are much more complex. BA Insight has spent quite a bit of time and effort developing a connector framework which has several features designed to surface line of business systems while honoring the following:

• Scalabilities
• Security
• Flexibility
• Manageability

The graphic above highlights the 5 core components of a BA Insight connector. Each component is necessary to ensure performance of both the SharePoint / FAST crawl and minimize the impact of the crawl on the Enterprise System.

Reading from right to left, the components are:

• Search Optimized API - The API's provided by Enterprise Systems such as SAP are not optimized for a search engine crawl. In fact, a search engine such as FAST or SharePoint will crawl a system without any regard for the stress on it at any point in time. As a result, a different API must be developed.


• Security Mapping - Another challenge associated with connector technology is mapping security of what SharePoint and FAST understand (Active Directory) to the security of the target system which is typically non-Active Directory based. A map must be created between the two and automating both the creation of the map and maintaining it is essential from a TCO perspective.
  A point worth stressing is that single sign-on will not solve this mapping problem because the search engine pulls every record and the security protecting it. This information is replicated the search index.


• Change log - Scalability is another challenging problem. A search engine will crawl a system as fast as possible. This puts tremendous stress on the Enterprise System. For those of you familiar w/ search technology, you may ask…"What about an Incremental Crawl?". An Incremental Crawl still puts a lot of stress on the system. During an Incremental Crawl, the Crawler will check the time stamp on every record to see if it has changed. BA Insight's approach is to keep track of only what has changed since the last crawl. The search engine first checks the log and only updates records that have actually changed. The may represent 5-10% of the overall corpus.


• Metadata Enrichment - Metadata Enrichment is a pillar to Microsoft FAST's approach to Enterprise Search. The FAST indexing pipeline is complex and is designed to enrich data as it's being indexed. Any connector developed for the purpose of crawling Enterprise Content should have this capability.


• Targets - This blog is devoted to Enterprise Search, but often users aren't interested in searching, they want to browse content in a document library. After connecting and extracting data, the Admin should have the means to "target" where the data should go. This includes search index, SharePoint lists or libraries, or people profiles.

Please keep in mind that the challenges I mention above are associated with Enterprise Systems. If you're connecting into and Active Directory based system, security mapping wouldn't be necessary and certainly custom apps that contain smaller datasets wouldn't need the change log.

I recently conducted a 30 minute webcast which covers all of this in greater detail. The link to that webinar can be found here: http://www.bainsight.com/resources-for-sharepoint-fast-search/Pages/view-sharepoint-search-top-3-mistakes.aspx

I'll cover the two other challenges of deploying Enterprise Search in next months' newsletter.