TWO of the most dramatic changes in contemporary information technology (IT) architectures are the rise of the cloud for processing and storing data, and the modernisation of data management tools to accommodate unstructured data and open-source technologies. These two trends are happening simultaneously, but their relationship is unclear. Both are critical to success for today’s organisations, but which drives the other?
It is, of course, always difficult to establish causal relationships among trends, but results of our recent survey (see sidebar, “Survey methodology”) reveal that these two developments are reinforcing each other. On the vendor side, cloud providers are offering not only cloud-based storage and processing, but data modernisation capabilities as well. On the user side, many leading firms are doubling their data footprints once or twice each year, so they are looking to simultaneously reduce costs, take advantage of new forms of big data, get greater flexibility in analysing data and employ powerful artificial intelligence (AI) and analytics algorithms. These benefits are available to firms that modernise their data and move to the cloud at the same time.
In April 2019, we conducted a survey of 504 respondents in the United States, all of whom work in IT groups of medium to large companies. These companies have annual revenues in excess of US$500 million and 60 per cent have revenues of more than US$1 billion. In terms of respondents’ levels, 46 per cent are C-suite executives, 30 per cent senior executives/head of business units and 24 per cent managers or programmers. All respondents reported being involved in or making decisions about cloud and/or data management issues.
Simply put, data modernisation means moving data from legacy databases to modern databases. It is particularly critical for any organisation that needs to shop unstructured data—images, customer voice audio, social media comments, clinical notes in health care and so forth. Data modernisation offers substantial cost advantages over previously used data management technologies.
Over several years, many large organisations have shifted from a data architecture based on relational enterprise data warehouses to data lakes based on Hadoop and other open-source tools.1 However, they may not have eliminated warehouses completely—warehouses are still useful for many applications involving structured data—but these companies are deploying modernised platforms for many new business applications.
How prevalent is data modernisation? Very much, indeed. Our survey results reveal that the companies we surveyed are already well along in the process of data modernisation, with the great majority (84 per cent) having already started their journeys towards that goal. Around one-third (34 per cent) claim to have such initiatives fully implemented, while half have data modernisation initiatives underway (figure 1). Financial services firms are the most likely to have initiated data modernisation. Somewhat surprisingly, technology, media and telecom firms are the least likely to have started down the path, despite the technical orientation of their products and services. Even then, 81 per cent respondents from the sector report having begun modernisation initiatives.
While the majority of respondents (62 per cent) expect to succeed in their data modernisation efforts, such initiatives are not always formalised (figure 2). Less than half (48 per cent) of respondents say they have a specific, formal initiative for data modernisation, but many others are modernising their data storage anyway. Interestingly, the majority of respondents—57 per cent—report that they are doing data security planning as part of their data modernisation activity and “data modernisation” may even be the primary banner under which they will describe it (figure 3). Respondent companies are cognisant of the real-world needs of their businesses and consider the tools and technology that are available in the marketplace the third most important reason in their decision-making.
Although there is clearly progress on data modernisation, it is not without obstacles (figure 4). The most common obstacle cited by respondents is “budget/cost concerns overall” (55 per cent). The other factors include the lack of understanding of technology (44 per cent), the lack of consensus among decision-makers (41 per cent) and the absence of clarity on success metrics (40 per cent).
The cloud is both a means to and an important consequence of data modernisation. Survey responses indicate that the cloud is already a dominant location for data storage—more than nine in 10 organisations (91 per cent) primarily keep their data on cloud platforms. Of the remaining nine per cent that primarily keep their data on premise, nearly all plan to migrate to the cloud. What’s more, on average, 57 per cent of the surveyed organisations’ businesses are actually operating on the cloud—meaning that all their important applications and data are on it. That’s saying a lot, since many organisations—including the financial services firms in the survey—often have to keep some applications and data on premise for regulatory reasons.
It is possible to pursue only on-premise modernisation projects, but given that many firms are moving data to the cloud, they often put it on modernised platforms at the same time. This makes the cloud a non-negotiable part of data modernisation. The cloud/modernisation connexion is strongly apparent to survey respondents (figure 5)—55 per cent see data modernisation as a key component of or reason for cloud migration. It is second only to security and data protection. The C-level respondents actually rank it the highest as a rationale, with 65 per cent overall saying it is a driver of cloud migration.
There are many other reasons for moving to the cloud. Traditional IT architectures and on-premise data centres often come with their own cost concerns, which make cost a key driver of cloud migration. In the survey, 32 per cent identified the “general cost and performance of IT operations” as the top-most or second driver for moving to the cloud. Middle-level executives were most worried about this issue, C-level executives somewhat less so. Since companies expect to save money by migrating to the cloud, they could perhaps mitigate some of their concerns about the cost of data modernisation by leveraging cloud migration as the primary means to modernise. Other drivers are the availability of cloud-based software and access to external data.
Many companies are discovering that the core capabilities of understanding customer journeys, making “next best offers,” and monitoring customer experiences are difficult or even impossible to achieve with a traditional premise-based analytics footprint. Cloud providers typically have strong capabilities in data management—even including relational data warehouses in the cloud—as well as high-quality algorithms for analytics and AI, and a wide range of external data sources.2 As such, most cloud providers enable companies to achieve more than on-premise capabilities.
Several companies have combined cloud migration and data modernisation to deliver on their strategic goals. Given below are some examples of how these initiatives have panned out.
These companies are examples of rapid advancement on both cloud migration and data modernisation dimensions. Cloud migration and data modernisation can help solve complex business and information challenges, including personalisation of customer information and predictive analytics for operations.
While no survey can identify the next trend in IT and information management, results of our current survey confirm the direction and strength of the existing trends—almost all data management approaches will likely eventually be modernised and almost all data and applications will be in the cloud. Survey results confirm that both cloud migration and data modernisation initiatives are well underway in most medium to large organisations in the United States and both initiatives—whether formalised at the corporate level—are generally successful.
Notably, cloud migration and data modernisation are mutually reinforcing trends—they seem to support and overlap each other.
As such, most companies would do well to embrace both trends. If factors within your organisation are aligned to only one, then perhaps you can pursue a stealth strategy that also embraces the other. If you are moving to the cloud, consider adopting some data modernisation approaches too. If your primary need is to modernise data, you can do that most effectively in conjunction with cloud migration. Since the two trends are overlapping, a “two for one” strategy is eminently reasonable.
Achieving your business outcomes, whether a small-scale programme or an enterprisewide initiative, demands ever-smarter insights—delivered faster than ever before. Doing that in today's complex, connected world requires the ability to combine a high-performance blend of humans with machines, automation with intelligence and business analytics with data science. Welcome to the Age of With™, in which Deloitte translates the science of analytics—through our services, solutions and capabilities—into reality for your business.