A guide to measuring the business impact of Generative AI

By Jessica Constantinidis, Innovation Officer EMEA, ServiceNow

Generative AI (GenAI) is the A-lister of modern tech. It has turned heads and stoked controversy as much as any emergent technology in living memory. It has potential impact for individuals, businesses, and governments. It is poised to be an ever-greater part of our lives as it finds its way into smartphones, PCs, and other devices. It is here to stay.

PwC recently surveyed Middle East CEOs and found that almost three quarters expected GenAI to “significantly change” their day-to-day operations over the next three years. This finding follows earlier estimates from PwC analysts that predicted a potential economic impact in the United Arab Emirates (UAE) of as much as 14% of 2030 GDP because of general (non-generative) AI. This was expected to be the largest relative impact to an economy anywhere in the region. Additionally, in the latter half of 2023, PwC’s consultancy unit, Strategy&, projected a US$5.3-billion economic boost for the UAE from GenAI alone.

The UAE’s progress is reflected in a global push towards AI adoption. A recent ServiceNow and Oxford Economics survey of more than 4,400 global executives found 81% were planning to increase their AI spending in the next year. We identified the pacesetters and found that two thirds of these leaders (67%) reported a “good visibility [into] AI deployment and utilization” compared with 33% of non-pacesetters. Some 65% of AI leaders said they were “operating with a clear, shared AI vision toward business transformation across the wider organization”, while only 31% of everyone else said this.

Move wisely

But among all respondents, only 35% had established formal metrics to measure the business impact of AI. This is a significant finding for the UAE given the projected economic impact of AI, and specifically GenAI, that is expected to occur here. If organizations wait for others’ successes and failures before exploring their own options, they will miss out any early-adopter advantages. To move wisely, an organization must be able to measure impact as adoption proceeds. One pro-tip when it comes to metrics is to not confine measurements to one corporate area.

First, look at financial effects. Metrics must be able to capture more than just cost savings and revenue boosts. The business should account for productivity gains and changes in the execution speeds of processes. Also consider the adding or subtraction of risk that can be attributed to GenAI as this has a direct bearing on future costs.

Operational impact can be measured by going beyond saved time and resources and looking to tie those outcomes to improvements in customer and employee experiences. It is also helpful to introduce measurement of the quality of adoption — the frequency and breadth of use of GenAI tools — so business leaders can better determine the technology’s contribution to operational and financial outcomes.

Measure twice…

While the enterprise is measuring impact, it should remember that apparent gains may have hidden, or even obvious, costs. GenAI may allow great efficiency in a call center, for example, by enabling self-service within the customer support function. This benefit may emerge in the metrics but behind the data may be disgruntled customers and demotivated employees. This would be very bad news for a brand that was previously lauded in the marketplace for its customer-first ethos.

When we look to the AI pacesetters and examine their approaches, the main takeaway must be that while they do use discrete data points like cost reductions and time savings, they do not stop there. If GenAI is to live up to its potential, adopters must approach AI metrics holistically. Finance and operations are entangled with the technologies that enhance them. Metrics must produce consistent, contextual results to be actionable and impactful.

We can think of a GenAI value triangle in which financial and operational impacts are stacked against quality of adoption to reveal an overall business value. If every GenAI project is evaluated in these terms, the enterprise is on the road to AI maturity. A good investment in GenAI is one in which the organization can point to measurably positive operational outcomes accompanied by clear popularity for the solution among users. Adoption tells a story that no amount of on-paper operational benefits can refute. That is why measurement of user engagement is critical to the success of AI in general, and generative AI in particular.

Just getting started In the digitalization race what we are seeing right now with GenAI is a revving of the engine. When it reaches full speed, we need to be ready to apply the brakes and turn the wheel accordingly. If we do not learn our lessons now, we risk exclusion from a technological revolution that will stamp organizations as winners or losers for years to come. Holistic approaches to value will allow GenAI adopters a measure of control over which stamp they receive.