I was inspired by an article in the Communications of the ACM about the “ Reducing the Software Value Gap”. It talks about how much “companies today depend on value creation from software solutions delivered by IT”, and the issue that “demand for software solutions and functionality often exceeds the IT budget (by up to 500%)”.
In the industry, we live with this issue every day, and meet overwhelmed IT groups and frustrated business customers regularly, so this is nothing new. But it was refreshing to see the evidence that pre-packaged software, outsourcing, subcontracting, SaaS, agile development, and requirements management all are failing to solve this problem. Consolidated infrastructure and platforms, strategic gating, value-first triage, use of patterns, and tailorable ‘kits’ or applications are proving to be effective techniques, though there’s plenty of evidence that this imbalance or bottleneck is a permanent fixture.
That’s a lot of ground to cover in one breath! I think it’s a nice perspective on how a bunch of software and methodology megatrends relate to each other.
Now what does all this have to do with enterprise search?
When IT people ask me “what’s special about search?” as a technology, it comes down to three main points:
1) Search provides a very natural, familiar, fluid, user interface.
2) Search bridges across content that resides in many places and provides a unified view without moving that stuff.
3) Search supports very rapid, fluid, and user-centric application development.
Now I could talk about any of these for a long time. But in particular the third one, the rapid construction of applications using search applications, is what I flashed on when reading about the state of the software value gap.
Quick construction of search driven applications is in some sense the heart of what BA Insight is about. It’s a great model. The schemaless nature of a search index makes it really easy to bring in new content types, map them in and change things as the system is running. Similarly, the user interface of search is data-driven and focused on where the content is coming from and is therefore dynamic. That not only lends itself to a natural user interface, but it is perfect for a rapid iteration approach.
I tell people not to wire frame search applications ahead of time. Instead, they should focus on high level user needs, then stand up a prototype, show it to people, and evolve it very quickly. It’s very easy to tailor and change, especially if you have the right tools and building blocks, and people can only really relate to it the system when they see it.
I tell people not to design their information architecture too early or too tightly. Instead, they should get a few core elements right and then stand up an exploratory search-driven application and go offroad. In real life, all data is dirty, and discovering what kind of information is out there firsthand before finalizing your architecture is a key to success. Otherwise, you design elegant and complex structures that just don’t fit the real world – so they are never fully used.
So many processes in the IT world are set up to mitigate the high cost of change and ensure against errors. But if the cost of change is not high and the likelihood of change is high, it’s best to adopt techniques – or technologies – that fit that reality. That’s not new, of course; it’s the central thesis of agile software development – but people usually think of the techniques, and tools, not the technologies. Building a data-driven portal instead of a structure-driven portal is a great example; using an index instead of a datamart is another.
Search can also reduce the cost and risk of change for other projects. For example, content migration projects are simpler if content is accessed by search, since the users don’t change how they get at the content throughout the process. This makes projects less risky and gets rid of dependencies between them. With the prevalence of more and more SaaS applications and cloud based content sources, the benefit of this fluid connectivity (what we sometimes call agile information integration) makes a huge difference.
Of course, there are patterns for portals and other search-driven applications that come up over and over – so the technique of capturing those patterns and putting them into products is effective. Similarly, tailored applications are what is often requested of IT – and tools or components that fit the ‘tailored kit’ model are a big win.
There are ways to expand this well beyond portals and use these technologies inside other applications. But I’ll save that for another article.
There is a natural match between the rapid, fluid, and user-centric application development that search supports and all of the ‘more successful’ techniques cited by the “Reducing the Software Value Gap” article. Thinking about development differently, not doing wireframes, not starting with a tight information architecture, and opening up agile to technology choice as well as project methodology are good ways to close that 500% gap.