Elasticsearch UI: A Platform for Enterprise Search

Elastic and SharePoint

If you haven’t already heard of Elasticsearch, or Elastic, you’re in a dwindling minority.  A year ago, there were 20 million downloads of Elasticsearch, and now there are 50 million, so the word is definitely out. Elasticsearch is an open-source search stack, started in 2010, that has since grown exponentially and provides users with the power of enterprise search and the flexibility of customizable software.

Elastic as a company was founded in 2012 and is growing like crazy. This growth can be attributed to the quality of their software, the extensive developer community (50,000 strong and growing) and impressive customer service.

At BA Insight we bet on Elasticsearch about a year ago, and now support it with all of our products. Using SharePoint as our UI, we’ve been able to create a complete enterprise search solution on top of Elasticsearch. We released our product offering this spring; you can read more about it here on our web site. I’ve since had the pleasure of working with prospects and customers solving real problems with it, which for me is what it’s all about. I was recently presenting at an Elasti{con} event – and the excitement about it prompted me to write this blog. I also did a podcast, which you can listen to on our podcast page.

We’ve built some powerful applications with Elasticsearch. The screenshots below are an example of a typical application, designed for investment managers at financial services. It’s a great example of why SharePoint and Elasticsearch are a good combination.

Elasticsearch Charts

SharePoint and Elasticsearch: Better Together

SharePoint and Elasticsearch are two products that, much like chocolate and peanut butter, are better together. They also both have vibrant ecosystems, strong communities, and incredible growth momentum. There are 155 listed Elasticsearch meetups, about 200 SharePoint Saturdays, and about 300 SharePoint User Groups. What’s surprised me, though, is how separate these communities are. Most enterprises are now using both SharePoint and Elasticsearch, but in different places. I’m often in a room with people who work on each of these, at the same company, but who have never met each other. It’s both fun and fruitful to make those connections, because the combination of products is so powerful.

Those who already know SharePoint are usually intrigued to know what’s possible with Elasticsearch. There are some things that you just couldn’t do with SharePoint search, but you can with the combination. Take, for example, real-time indexing, which is a huge value add for financial companies who need to access financial transactions in seconds. Before Elasticsearch, indexing these transactions took minutes, which when you’re dealing with money is practically forever. Another great value-add for our oil and gas customers, who need to know where things like mineral rights and so forth are, is geo-search – which is a capability that is built in to Elasticsearch. You can also have pattern searches in Elasticsearch, where you may search – for instance – for a specific part number format, as we see a lot in manufacturing, logistics, and research applications.

Using Elasticsearch for Enterprise Search

Elasticsearch serves a huge range of applications (take a look at some of these use cases). It is often applied for “data analytics” – working on machine-generated data such as logs, transaction records, etc. to provide insight into things such as application performance or cybersecurity threats. It’s also used for “embedded search” to provide the search functionality within services such as Spotify and Netflix. It’s used as well for enterprise search, for finding knowledge within organizations. Enterprise search is BA Insight’s focus. It’s also where the combination of Elasticsearch and SharePoint applies, because the users are all the employees of a company – who very commonly use SharePoint as their intranet or research portal because it’s so easy to manage and tailor the UI.

The main purpose of an intranet is unlocking the content you already have – making it findable and useful. The search core is very important, but you also need other elements to make a full search solution. That’s where BA Insight comes in; we have a whole portfolio of products for this.

A Complete Enterprise Search Solution with Elasticsearch, SharePoint, and BA Insight

One key element for enterprise-wide search is connectors. Elasticsearch has a great API but doesn’t provide any connectors; BA Insight has a portfolio of over 50 connectors. These have item-level security built in, which is very important and also quite tricky because of the variety of different security models across source systems.

Another important area is user experience. Elasticsearch has a UI template (a “head”) with it, but many of the other elements that enterprises are looking for have to be developed. BA Insight has a set of applications that are prebuilt and ready to go. These can run in SharePoint, which is often the portal of choice for the IT group.

There are other elements to a complete enterprise search solution, too. Auto-Classification is important where content has poor metadata (which is often), and Analytics is also important for ongoing administration. There is also often a need to further customize, for specialized requirements.

What does it all look like together?   Here’s a color-coded block architecture, to give you a sense.

auto-classification block architecture

BA Insight provides the connectors, autoclassification, and a content orchestration framework for setting up and managing content pipelines. SharePoint provides the UI and is the face of your intranet, using a set of our prebuilt applications as well as the native elements.

Smart Analytics is our analytics tool, which lets users get reports on exactly how their intranet is being used, identify high-volume queries, identify areas that may need adjustment or improvement, and basically keep tabs on how well your intranet does what it was designed to do.

Elasticsearch, and its sister program Kibana, which provides visualizations, ties everything together in a very powerful way. We also have a product called Federator which lets you mix-and-match your search functionality; if there are certain things that SharePoint search handles well, you use SharePoint search, and if you want to go beyond its capabilities, you use Elasticsearch.

No Time Like the Present

Whether you’re coming from using SharePoint and wanting to extend your search capabilities, or from using Elasticsearch and to expand it into a full enterprise search, now is the time to combine these two powerful platforms into one package.

One of the great benefits of Elasticsearch being open-source is that it’s not going anywhere. There have been a lot of mergers and acquisitions of search engines and some of them have gone away. As a case in point, the Google Search Appliance (GSA) was recently discontinued (see my blog post “Saying Goodbye to the Google Search Appliance”). We’ve had many folks interested in using our Elasticsearch solution as a replacement for the GSA, and I am psyched that we can provide them with a solid path that has low risk. You can be confident with an open-source product because in the worst case scenario, you can always support it yourself with community help. If you need to change things or see how they work, then you can read the code – which is exactly what we do at BA Insight. We’re also starting to contribute things back into the open source repository, as do others, which means that the product is only getting better with time and use.

Elastic as a company is great to work with.  Their support is top-notch enterprise-class support with open-source software, which really gives you the best of both worlds.   We also use and recommend their plugins, and use X-Pack with our customers as a value-added capability. For example, our customers can manage and monitor their Elasticsearch cluster, set alerts about changes in their data, get a full set of reports, and secure their clusters and their data very flexibly. Elastic’s new Graph API (which is part of X-Pack) is very powerful, and it’s a good match for our text analytics capabilities, like entity extraction and relationship extraction.  Elastic is also a good partner in many ways, including a recent feature about us on their blog.

Elasticsearch may not be the perfect solution for every company – there are some things that are better in SharePoint search. People search in SharePoint is very effective – it has things like phonetics and nicknames built in, and is really strong in that area. Microsoft’s focus has been on making SharePoint accessible to the masses, so you don’t have to be a developer to administer it. But there are other things – some of which have already been mentioned – that are better in Elasticsearch, like query expressiveness for more sophisticated queries. That’s where BA Insight’s products and integration really bring value. Also, the majority of enterprise search users are using SharePoint, but a lot of Elasticsearch users come from a Linux and Java world. BA Insight is working on removing dependencies so you can have a pure Elastic solution as well as an Elastic SharePoint solution.

From Zero to Sixty

In the year we’ve been working with Elasticsearch, we’ve incorporated the Elastic stack inside our own Smart Analytics product and are working to triangulate with Microsoft. We now have an Azure template for Elasticsearch which we recommend as a starting point for deploying our solution. If you’re a developer, you can download Elasticsearch and install it, or use the Azure template. If you’re not a developer, then you can certainly call on BA Insight for help. You can move pretty quickly with this solution because BA Insight has put a lot into making it a very plug-and-play kind of application. Elasticsearch isn’t just for developers anymore, especially when combined with BA Insight’s product catalog. If you’re in IT and don’t want to write code, but want to make use of Elasticsearch, BA Insight can help.

One thought on “Elasticsearch UI: A Platform for Enterprise Search

  1. Avatar
    David Murgatroyd says:

    Thanks for a helpful post, Jeff.

    > there are some things that are better in SharePoint search. People search in SharePoint is very effective – it has things like phonetics and nicknames built in, and is really strong in that area.

    FYI: there are easy-to-use plug-ins for Elasticsearch that give it people search that I bet make are even better than SharePoint’s people search. My favorite is described in detail at http://www.basistech.com/fuzzy-search-names-in-elasticsearch/ (disclosure: I’m the VP, Eng of the team behind it). We’d love to get feedback from anyone using Elasticsearch on it, especially if folks can confirm or deny my bet that it makes Elasticsearch’s people search better than SharePoint’s.

Leave a Reply

Your email address will not be published. Required fields are marked *

For security, use of Google's reCAPTCHA service is required which is subject to the Google Privacy Policy and Terms of Use.

If you agree to these terms, please click here.