Business in the 21st century is being redefined by a data-driven revolution. Take the MIT Media Lab's experiment to see whether it could estimate retail sales performance on "Black Friday," the day following the US Thanksgiving holiday. Instead of waiting for data from the stores themselves, they used location data from mobile phones to infer how many people were in the parking lots of major retailers. Combining this with data on average spend per shopper enabled them to estimate a retailer's sales, even before the company had recorded it themselves.
This is just one example. Judgments that used to depend on human intuition alone are now supported by insights gleaned from complex analyses and predictive modeling. Retailers combine data on demographics and weather to predict sales and develop merchandising plans. Banks and lenders have predictive analytics engines that tell the lender the probability that a customer will pay them back. Housing market price changes can be more accurately predicted from analysis of Google searches than by a team of expert real estate forecasters. Investment is rushing into big data analytics as firms seek to find ways to first understand and then take advantage of the possibilities on offer. There has been a rapid uptake in health care, consumer marketing, crime reduction, agriculture, scientific research, and many other areas.
One area so far relatively untouched is organizational change management. That's not because there isn't a problem to solve. The failure of major transformation projects to deliver the expected benefits is a well-documented phenomenon: many change programs simply do not achieve their business goals.
It's time for that to change. The combination of predictive analytics, large data sets, and the processing power of today's computers is starting to transform change management. Just as the discipline of marketing has transformed from soft to hard science in the past 20 years, so too will the practice of change. But before that can happen, we have to understand why data has failed to catch on in change management to date.
A big obstacle is the change management profession itself (of which we are all proud members). To date, change management has not been based on a data-driven model. When a change practitioner talks about data, typically that is qualitative information, generated by a root cause analysis workshop or similar. That's because so many of the issues we deal with have to do with human behavior. These intangible factors like culture, leadership, and motivation do not yield easily to empirical analysis. This makes it difficult to set up controlled experiments to validate cause and effect to demonstrate how specific change interventions deliver the intended outcomes.
Academics may not have helped us. Most models for change management are rooted in research from the 1940s that was originally designed to explain how small groups adapt to change, not large complex organizations. The most popular managerial approach to change management is John Kotter's eight-step model. Researchers have pointed out that though the model makes sense, there has been little empirical data to support it. Subsequent research to validate this model has been unsuccessful. There are also heated academic debates over how to measure leadership effectiveness, motivation, and culture. One recent study by leading academics in the field concluded that there is no agreed definition of "culture" and that the tools used for measuring it were either methodologically flawed or designed to measure something else altogether. These are not the foundations of a proper science that verifies knowledge through experiment and replication of findings in peer-reviewed studies.
When change management process does work, it is due to the work of skilled, experienced professionals who know how to weave together a set of practices to help a business reach its change goals. The issue is that they operate as artisans, not scientists. Change practitioners struggle to reach the levels of proof that are standard in other professions. They enable many successful transformation efforts, but then lack the data to demonstrate the link between cause and effect that marketing or supply chain professional takes for granted. This makes it difficult to justify investment in change management with the rigor that a data-driven CEO or CFO might expect.
The result is a cycle of underperforming transformations; with no data to validate the return on investment, benefits of change management doesn't attract the resources it requires and the outcome is down to the quality of the artisans at work. Improving the tools the artisans use may make for better outcomes, but it won't enable demonstrable cause and effect. Converting change management from an art to a science is the key to unlocking this problem.