Network fault management

88%

Reduction

in ticketing

255k

Minutes of

Engineering Capacity restored in year 2.

90%+

Reutilisation

of automation meant no incremental costs

Background

A Communication Services Provider (CSP) with an international voice network wanted to maximise the quality of service to their customers whilst minimising the cost.

A set of network traffic monitoring systems reported incidents on the network traffic to specific destinations. As well as this, a central routing engine helped engineers make changes to the routing as and when required.

Due to the complexity of correlating the alarms and the evaluating the best routing alternatives, it proved difficult for the customer’s engineers to keep up with the changes, leading to them being overwhelmed and their ability to apply fixes to the network diminishing over time.

The fault management system had a huge list of alerts which were either not handled in time or were cleared by users without any analysis.

The fault management system had a huge list of alerts which were not handled in time

Objectives

The Customer understood this to be a business critical process which was constantly deteriorating under the pressures and demands of the wider business growing around them, but the departmental ability to keep up with said growth, not scaling sufficiently.

An automation initiative was devised to increase the velocity and accuracy at which this team could operate. Additionally, the business needed to evaluate and remedy all unhandled alerts, as well as those which were handled either slowly or incorrectly, to reduce the errors in the network. The end goal of the project was to achieve a zero-touch automation of the routing changes based on traffic alarms.

To carry out the project, an internal automation team was selected to undertake the work, from the knowledge extraction to deployment in production. The team were not experienced in automation techniques and strategy. Equipping the customer to be self-sufficient was key to the selection of CORTEX. The customer needed to be able to increment changes, as well as find extensions to the implementation, which in turn would continue to increment the value of the solution and provide robust long term value for the investment.

Solution

CORTEX was integrated with key tools of the business, but primarily the Fault Management System. This is so CORTEX can gather the network alarms.

Using information collected from the inventory CORTEX triggers the diagnostics to the network through scripted task automation that the CSP had already implemented. Unlike other automation tools, CORTEX’s legacy integration capability allowed for automation to be reused saving time and money.

During the project it became clear that the operations team had inconsistent and missing documentation on their diagnostic process for a given alert type and its subtypes. Whilst this can de-rail other automation projects, owed to the human-in-the-loop capability of CORTEX, automation remained possible.

Where business critical automation cannot fail, and similarly, you cannot yet rely on AI, a human is best placed to review the process and then rapidly deploy an automation flow to prevent it re-occurring. Human-in-the-loop dramatically reduces the time it takes to get solutions to production; you build what you can and then iterate as you learn or you find edge cases. This is far more agile than classic waterfall developments that often underpin automation for CSPs.

As the solution immediately addressed the bulk of the highly repetitive situations, this created capacity to begin to address the others. This not only enabled our client achieve a shorter development and testing cycle, but they were able to go on gradually delivering the automation of rarer problems and edge cases into production.

The fault management system had a huge list of alerts which were either not handled in time or were cleared by users without any analysis. Cortex’s scalability allowed every alarm to be addressed in real time.

Further in the implementation process, the existing Fault Management System required enhancement to receive alerts from a wider list of domains increasing the reach of the automation, minor rework through Cortex’s Flexibility when reutilising existing automations, enabled the quick management of these alarms.

The inventory system exposed additional services for automation so that Cortex could use the Service Impact Analysis on demand, minimise the number of tickets raised, and populate the trouble ticket with useful information.

Results from the automation were logged in Cortex but also updated in the Fault Management System which enabled users to be aware of the progress of the automation, intervene as required, and manage changes required in the automation.

Outcomes

The Customer achieved all of their objectives and more.

The Fault Management system was enhanced to receive additional alerts, from a wider list of domains all of which the customer self-serviced in line with their original objectives.

One year after the first deployment, more than 50,000 events were managed by the automation and distilled down to 6,000 automatically generated actionable tickets allowing for greater oversight of issues arising.
85,000 minutes of manual effort saved. With a potential to increase this significantly (estimated 3x) during the following year. Freeing up NOC engineers to take on work that would not have been possible before CORTEX was implemented.
The NOC engineers were educated with automation skills and a framework to deliver successful automation yielding value to the business.

The automation initiative has brought together the people, processes, and technology required to guarantee success, enabling people to implement and support automation, establishing the appropriate processes underpinned by the right technologies.

Why choose CORTEX

Our software alone is not the reason why people buy CORTEX, we know what really matters to CSPs. Discover the difference with We Are CORTEX

Time to Value

Deliver business value quickly

I have a long term technology strategy but I need my automation to start delivering quickly and cannot wait for all of the legacy systems.

Scalability

Hyper scalability for Communication Service Provider operations

We work in a high volume business and our automation tools need to execute at speed and scale out to handle huge volumes of transactions.

Enterprise Governance

Enterprise-grade automation platform

We want to automate business critical operations and we need central controls and governance around our automation estate to avoid citizen-developers running unnecessary risks.

Explore more Use Cases

As a dedicated CSP business we deep dive into every area of a CSP, take a look at some of our other use cases here.

Billing Operations

Customer billing relies on an extensive underlying infrastructure to monitor, record, maintain data and apply required contractual parameters for accurate billing.

Connectivity Circuit Writing

The current IT stack used within the circuit writing process required manual data input leading to inefficiencies.

Network fault management

Manual network fault management on trouble tickets for network infrastructure that were raised from customer complaints, by the operations teams.

OLO Network Traffic Management

The customer wanted to maximise the quality of their international voice network services whilst minimising costs.

A Fully Digital Service

Voice Resource Management was a manually performed process, from order to allocation and further provision.

Change Request Management

Minimising unexpected downtime or systems availability for customer impacting Change Requests.

Network fault management

88%

Reduction

255k

Minutes of

90%+

Reutilisation

Background

A Communication Services Provider (CSP) with an international voice network wanted to maximise the quality of service to their customers whilst minimising the cost.

Objectives

The Customer understood this to be a business critical process which was constantly deteriorating under the pressures and demands of the wider business growing around them, but the departmental ability to keep up with said growth, not scaling sufficiently.

Solution

CORTEX was integrated with key tools of the business, but primarily the Fault Management System. This is so CORTEX can gather the network alarms.

Outcomes

The Customer achieved all of their objectives and more.

Why choose CORTEX

Explore more Use Cases

Share

Lets Talk!

Follow Us

Contact Us

If you want more information about our automation solutions, about a Partner, or you are ready to fully engage on your project, let's get talking!