Offering a "full-stack" "full-lifeycle" experience to maximise the opportunity for our customers.
Designed to keep you at the cutting edge of AI, Cyber, and IT advancements, we are voicing the unsaid and shaping the future of business technology solutions.
Your challenges
As a Business Technology Services Partner, we offer the "full-stack" "full-lifecycle" experience, built to maximise the AI opportunity for our customers.
OUR CULTURE
Thank you for your interest in Acora. We'd love to hear from you! Please feel free to drop us a message via our contact form.
Follow Us
Work with us
To view this video please enable JavaScript, and consider upgrading to a web browser that
Home News room The importance of ‘live testing’ – what to learn from TSBs error
I look after technical presales and architecture functions across Acora’s customer environments, our private cloud platform and our own internal infrastructure. In my 20+ years, I’ve worked on service desk, in engineering and design, and as a consultant and technical account manager, giving me a deep understanding of a whole range of client issues. I’m here to help clients make good technology decisions and ensure a smooth implementation journey that delivers real value and business benefits.
There are several very interesting points that can be derived from this headline, and indeed the subsequent articles, the most standout points to me are:
These points, and a firm grasp of the reason for them, should drive home the importance of testing. It should also highlight the importance of the frequency, depth and duration of that testing – and most importantly – the level of risk associated with the activity that the testing supports.
It’s quite common in IT-driven companies to have significant and segregated test platforms, these are called “pre-production environments/platforms.” They mirror the technology and create a configuration of the live system on independent hardware and systems. Less commonly, they will also mirror the scale and capacity of the live systems (depending on the level of risk involved in that system) and it may be needed to prove the performance. It is important to note that this is not the same as the development or traditional test environment, but one that is controlled in the same way that the live environment is – quite often it is not managed by the development team, but by the infrastructure team to ensure this control.This pre-production platform is where most testing takes place, it provides a safe space, somewhere that is controlled in terms of change. It ensures the test is true and that the results can be trusted. It can be quite expensive to deploy, maintain and manage – but that effort and expenditure are worth it to mitigate the risk of untested changes impacting the live environment.However, in a system that is sufficiently large-scale, or perhaps public-facing, it is far more difficult to simulate the load and behaviour of the REAL user base in a pre-production platform. The sheer scale of the production platform and the associated workload can affect how a system accepts change or behaves, compared to the test platform. Add to that the complexity of human behaviour, which can be quite unpredictable, and you introduce an element of risk that demands attention. This is where live testing comes into its own.In live testing, when a change is pushed to live, it is effectively on a trial basis. It may be to a subset of users initially, allowing a control group to interact and ‘use’ the system on a live basis. Or, it may be that a maintenance window is created to introduce that control – but the key is the change is not signed-off until all the test criteria have been met in production, and the performance and stability of the platform are observed as correct. Until that point, the rollback plan, which should have also been thoroughly proven in pre-production, should be ‘waiting in the wings’ ready to run.
I can only imagine the internal conversations that happened at TSB in the hours and days following the outage, but I imagine they included such statements as “it was tested” and “we got sign-off for the change”. But I would ask, was the risk and impact of that change properly understood when it was signed-off?The key question a leader should be asking when anyone in IT, or indeed the whole business mentions words like ‘patching’, ‘updates’, ‘roll-out’ or ‘migration’ is “what is the worst that can happen?” Closely followed by “what is the test plan?” and “what is the rollback plan, has it been tested?”What they should be asking themselves, and demanding their reports ask themselves before presenting the answers is, “Does the depth and rigour of testing, and rollback planning, reflect the level of operational and/or financial impact associated with an outage to the specific system/service in question?”Sometimes it is as simple as numbers (e.g. retail), sometimes it is SLAs or contractual obligations (e.g. finance or law), sometimes it is reputation (public transport) and sometimes it is a combination of these factors.The important point to understand at C-level, and to drive into the culture within IT, is that change control and risk decisions are not just IT decisions anymore – they are business decisions. It is the responsibility of everyone involved, regardless of whether they work in IT, to take the time to effectively understand and communicate both the risk and the impact all the way up to C-level, and inspect all the way back down to the planning level, to ensure that what has been planned is appropriate.Then, when the worst happens and something comes out of the blue, we can roll back – and we know the rollback will work… because we tested it. For more information about pre-production environments and IT project planning, contact us. We’re happy to help.
2022 has been about growth, people and innovation. Company-wide, we have seen lots of staff members return back to the office which is essential for communication, personal development and collaboration. We are seeing the benefits of face-to-face internal team and…
Acora is pleased to announce it has secured a new minority investment with LDC, the private equity arm of Lloyds Banking Group. The new investment round, supported by debt funding from Ares and HSBC, will provide long-term financing for the…