DEATH TO THE SLA (as the only measure of IT success)

Whatever our sector or profession, we all have (or should have) ways to gauge how well we’re meeting our customers’ and clients’ needs and expectations. We often gather this feedback by informal, ad hoc means, but many of us also have formal, contractual arrangements in place. Within the IT world, these take the form of service-level agreements or SLAs.

An SLA defines the levels of service the customer can expect, the metrics by which they’re measured, and any remedies or penalties that apply if they’re not met. The details can of course be exceedingly complex; but the underlying principles are fairly simple for clients and suppliers to get their heads round. Hence they’re still ubiquitous after all these years and show no signs of going anywhere.

The way we used to be

So why is Acora confidently proclaiming their demise? To be clear, we’re not – or not in so many words. What we are saying is that, in our view, time’s up for the SLA as the only measure of IT success, for a number of reasons.

Some of these are fairly basic. SLAs are (and the clue’s in the title) entirely service-focused, so tend to deal with hygiene factors – the things that, as a client, you should simply expect as a matter of course – and responses to specific issues and incidents. Perhaps more significantly, they generally define minimum levels of performance only. This means that provided they’re fulfilling what they’re contractually obliged to do, there’s no incentive for suppliers to do more or better. Nothing’s actually broken or fallen over, ergo everything must be fine. Logically impeccable, but very unlikely to drive improvement; or, worse, may encourage people to ‘let sleeping dogs lie’. Indeed, you could argue that an SLA has never actually been a measure of success, and more a measure of failure.

Plus, the very longevity of SLAs disguises the fact that their essential form and function haven’t changed all that much – but the world and technology have moved on enormously. Look at SLAs from 10 years ago and chances are they’ll be remarkably similar to those in force today. The risk is that they end up regulating and measuring old aspects of performance that no longer apply, while not adequately addressing new ones that do.

A moment in time

Then there are the metrics. A standard means of measuring performance in most SLAs is the customer satisfaction score or CSAT, usually conducted quarterly. It’s obviously better than nothing, but CSAT provides a snapshot that may not accurately reflect overall experience and satisfaction levels. Because while CSAT is structured and formalised, you’re still dealing with people; and if they didn’t like how you dealt with their recent issue, or they’re just having a bad day, they’re going to mark you down. As with most things in life, we’re all only as good as our last gig.

The importance of the UX

But for us, the biggest drawback with using SLAs as the sole measure of success is what we call the Performance Paradox. This is where all your SLAs are green, but users and staff are still grumbling, frustrated and unhappy with the IT service they’re receiving. The cause is simple but fundamental. SLAs basically measure users’ satisfaction with IT support in response to specific trigger events: they’re not designed to assess or track the overall user experience (UX).

This really matters. The COVID-19 pandemic has thrown new light on the relationship between people’s interactions with and experience of technology, and their productivity and wellbeing. And with the shift to remote and hybrid working looking increasingly widespread and permanent, this correlation is only going to grow in importance.

Introducing the XLA

It’s worth saying that this isn’t a fault inherent in SLAs: historically, measuring UX has just been very difficult. Constantly asking users how they’re feeling risks simply irritating them; what evidence you do obtain tends, unsurprisingly, to be unstructured, qualitative and essentially anecdotal.

But advances in technology now allow us to measure the UX in real-time and, crucially, in non-intrusive ways. This isn’t off-the-shelf stuff; we’ve invested significantly in developing and making it available to clients as part of our managed services offering.

By combining this UX data with our standard SLA, we’ve created something new and enhanced, which we call the Experience Level Agreement, or XLA. And what XLA does is bring the SLA into line with today’s IT environment, by adding two crucial measures of success.

New measures

The first we call the Digital Experience – capturing how people actually use and interact with technology and applications day-to-day. What they tell us about real-world reliability and performance help us understand exactly how IT is contributing to their stress, frustration and dissatisfaction levels, and its impact on their wellbeing and productivity (or lack of it).

We also measure adoption success: are people using that shiny new piece of software you’ve provided for them, or are they still using the old one they know, understand and therefore prefer? Anecdotally, high adoption levels frequently correlate with high user satisfaction: people are generally happy when they’re consuming new technology. Measuring Digital Experience can provide empirical evidence of this added value, bolstering CIOs’ investment cases. It can also help identify users who are perhaps resistant, reluctant or having difficulties, making it easier to provide appropriate, targeted training and other support.

Our second new metric is Sentiment data. When a service event is triggered, we ask the user to share with us how they’re feeling at that moment. In our experience, engaging with users directly and proactively while we’re actually dealing with their issue produces response rates up to three times higher than CSAT. Plus, we’ve found it builds confidence and helps restore good relations between the user and IT/Operations.

Together, the Digital Experience and Sentiment provide hard, empirical data about what’s wrong, and what we need to do to change and improve things for people.

New solutions to age-old problems

The XLA is a much-needed update and upgrades to the SLA in line with today’s requirements. By directly addressing the UX as well as ‘pure’ IT service metrics, it provides an answer to the Performance Paradox we talked about earlier.

More than that, it also goes some way towards resolving another familiar problem for IT leaders. The CIO Dilemma is the perennial conflict between investing in maintaining BAU with no IT issues and transforming the organisation’s IT through innovation and new technology. The XLA provides hard, empirical data to support business cases: it tells us that if we invest in A, it will improve the UX by a factor of B, with all the productivity and wellbeing benefits we know flow from it.

For CIOs, this brings much-needed clarity, focus and objectivity to the debate, and helps maximise return on investment in terms of an improved experience not just for internal customers, but external ones, too.

Better all round

Over time, this becomes a virtuous circle. The more we measure the UX, the better we understand what people want and need from their IT experience, allowing us to target investment in areas that improve the UX. For CIOs and other decision-makers, this also shows a clear path to value, further strengthening the business case and overall accountability.

So no, we’re not reading the last rites over the SLA just yet. By introducing the XLA, we’re enhancing, strengthening and extending its usefulness, and making it fit for purpose in today’s IT environment.

To find out more about how we’re using the XLA in our services, please follow the links below.