It’s hard to admit after 25 years that the business intelligence (BI) stacks we all worked to build are failing us, but to make progress, we must. If we, as technologists, take a hard look at what we’ve learned from the current generation of big data and other analytics technology and keep the following principles in mind, we can build empowering solutions.
1. Easy access to data
Traditional BI protected users from raw data and mandated a high degree of data normalization. This was a mistake because users can also benefit from raw and intermediate forms of data. The BI machinery for data modeling and normalization is also hard to adjust to ever changing requirements and lacks much needed agility. We should create a simple way to get unstructured, structured, machine, human-generated, real-time, historical and any other type of data through a universal repository, or at least a universal search.
2. Add context
Systems should enable users to ask relevant questions in the context of their work and the objects they are familiar with. What data would they like to inspect and in what timeframe? What are the key objects?
Users should be provided with the data they need and tools to analyze it. The system should add context and enrich data using machine-learning techniques for correlating data and adding relevance. If call detail records have cryptic internal IDs, the system should substitute phone numbers. User IDs should be linked to data about the user, including transaction history.
3. Advanced analytics for everyone!
We trust a GPS to get us where we want to go, but have no idea how it works. Similarly, in the future, business will be guided by systems using machine learning and advanced analytics. We must make advanced analytics easier to use and to trust. We must understand when the system can do the job and when it needs guidance. Data scientists can be a tremendous asset for a company, but you shouldn’t have to employ an army to run advanced analytics. Analytical systems that more people can use are more valuable than feature-rich systems usable only by experts.
4. Build on search
When people want to learn, they search. Let’s build on that.
Casual users can use natural language and assisted search techniques. We must make it easy to set the stage for analysis, jump into an experience, and jump out to keep searching. Search also makes it very easy to correlate data that is isolated in siloed data marts or in disparate systems. Advanced search technologies provide mechanisms to group events or matching records together, as transactions for example, making it easy to understand otherwise hidden relationships.
5. Provide speedy interactions
Responsiveness is implied in the search paradigm. Users understand that they may have to wait for complex requests, but they are happy to get partial results. Building results in an incremental fashion and streaming data continuously to the user interface supports an iterative workstyle for users to refine their requests with each iteration without waiting for long running batch jobs to complete. It won’t be easy to make things fast, but we must keep trying to bring instant value to users.
6. Make real-time information easy to use
In most companies, real-time systems are seen as specialized and advanced, not something you use every day. Most environments weren't built to handle a real-time display and required significant efforts to reduce the latency between data being created e.g. by a transaction and the event showing in your analytical environment.
However we must realize that the cost of acquiring and using real-time information has dropped dramatically. And while there are the scenarios where real time capabilities are irrelevant, we should always consider how real-time information could help. Particularly most operational systems will greatly benefit from very low latencies.
7. Make visualization pervasive
Often visualizations are the big finish of a long train of data analysis, a fancy painting on the wall. While the results are impressive, it also means visualization is often an afterthought. Visualization should be used more often, not just at the end of the process, but in exploring data, in reviewing interim results. BI should be more like a video game -- primarily visual. Visualization tools are important to help overcome data overload when presenting information derived from big data and to help people recognize actionable insights. Information workers need to be empowered to explore data without preconceived notions.
8. Embed collaboration
Almost every analysis involves working with others. But if we want to collaborate and share an idea, we must stop what we are doing, move to email or IM, and record our thoughts. Often, it is hard to invite others to see what we’re looking at. The systems we create in the future must be collaborative and allow knowledge to be shared so that advanced practitioners can help others. Social networks are rapidly spreading into the corporate world and transforming communication and information sharing.
Collaboration enables organizations to tap into the intelligence of their teams, ecosystems, and customer communities.
9. Create a platform for applications
The smartphone industry has demonstrated how applications that run on flexible platforms such as Android or iOS can provide a scalable pathway to deliver an incredible amount of value to users. The barriers in these app economies are low not only for end users but also for developers. Any effective analytical environment that aims at broad adoption should provide similar mechanisms to enable a wide spectrum of valuable applications that are sufficiently decoupled from the underlying platform and that can be updated and shipped using an incremental low friction model.
10. Deliver instant value for everyone
The litmus test for every analytical solution is based on delivering value to many users almost instantaneously. This last principle is also the essence of all of the preceding principles. To deliver value instantaneously, a system has to avoid any wait states for users. It needs to provide results in seconds, with minimal or no setup. To be relevant for everyone, it can’t require the skillset of a data scientist but needs to be easy to use. To provide value, it needs to solve relevant problems with out of the box solutions or applications.
All of these principles should inform creation of applications that help knowledge, expertise, analytics and automation reach the widest audience. But applications must take many forms. Users should be able to create lightweight, ad hoc apps. Superusers should be able to create advanced apps to share what they have learned. Developers should be able to create rich applications with advanced functionality. In this way, an organization gets smarter as time passes.
The leading data-obsessed companies have made many of these principles part of their culture and are relentlessly pushing forward on those they haven’t mastered. If these principles can be adopted more widely, I am certain we can make BI less frustrating and more delightful, with better business results.
Title image courtesy of Sergey Nivens (Shutterstock)
Editor's Note: Want more business intelligence? Check out Randy Bean's Using Big Data to Accelerate Time-to-Answer