Data-driven operations

Good analytics needs good data

The quality of any assessment is only as good as the information used to make it. For data-driven operations, this means having a complete set of real-time, precise key performance indicators (KPIs) for both the network and the services it carries. In fact, in an SDN/NFV network where there may be no direct correlation between the physical topology and service topology, it’s important to have KPIs for every service. Correlating changes in service KPIs, even before they indicate an issue, can lead to the discovery of hidden but more widespread problems, impacting many services. 

Ideally, KPI generation should be part of the service definition itself, being measured at the service endpoints and in a consistent manner for all services. This end-to-end view provides the closest measurement to what the customer is actually experiencing.

Finding a needle in a haystack

The good news is you’re generating KPIs for every service in your network—100% service visibility. The bad news is you’re now being swamped with KPIs—200,000 services generating 5 real-time KPIs per second means you’re generating 1.44 billion KPIs per day! With this volume of data, detecting major service issues could be challenging for a human. But what about trying to find services that are just starting to trend towards problems or looking at trends in utilization that could trigger network expansion?

This is where big-data analytics shines. Making sense of a sea of data means looking for subtle changes in KPIs that may otherwise go undetected. And being able to correlate changes in KPIs across many, possibly unrelated, services can lead to network insights that would have otherwise been hidden. Proactively detecting and managing changes in both network and services is the key to data-driven operations.

It’s never just one thing

The challenge for any operations team is deciding which problem to fix first. Seldom do problems develop in isolation, nor is it likely the root cause is located where the problem was first noticed. Take latency for example. Noticing a trend toward higher latency at an endpoint is often a sign that the network is becoming congested somewhere—quite possibly an issue with over-utilization of an Ethernet port or a protection switch elsewhere. Without the ability to scan millions of KPIs, detect the ones that are trending in the wrong direction and then correlating them to find a common potential source, your teams may spend days chasing the wrong problem.

Seeing the ‘big picture’ and being able to correlate and prioritize all the known issues is what data-driven operations is all about. Fixing the right problem first can often mean solving many problems at the same time.

Keeping track of the ever-changing network

One of the key benefits of an SDN/NFV-based network is the ability to continuously optimize the infrastructure and service topology to address changes. If a virtual router is reaching capacity, create a new virtual router to share the load. If there’s a failure or high congestion on a particular link, re-route traffic to avoid the problem link. And since SDN/NFV networks rely on orchestration to make these changes, there may not even be a human involved.

Keeping track of how the network and services change, in real time, is essential for both service provisioning and fault isolation. Knowing there’s adequate bandwidth available to provision a new service, or knowing which virtual and physical elements a service runs through, are critical for effective operations.

Real-time topology mapping must become a fundamental building block of any solution, both for virtual and hybrid virtual/ physical networks. Without it, operations teams can waste significant effort trying to piece together what they need—and even then, because of automated orchestration, things may change without notice.

Cost-effective operations

With the scale and complexity of today’s networks growing all the time, there can be no doubt that automation will need to touch almost every aspect of operations—from capacity planning to service provisioning to troubleshooting and maintenance. Carriers are making significant investments to adopt and embed automation into as many aspects of their business as possible. An important benefit of this transformation will be offloading of many manually intensive tasks which typically require highly trained and experienced technicians. For example, troubleshooting and root-cause analysis often required a 2nd or 3rd level technician with a deep understanding of the way the network works to be able to come up with a suitable solution. With a data-driven operations model, this knowledge is embedded in the automation systems allowing optimal solutions that can take into account additional dependencies and network impacts, to be developed in minutes rather than hours or days; and, with fewer expensive technicians, less chance of human error and consistency with network policies.

Proactive rather than reactive operations

Imagine being able to identify and fix issues before they become problems. Imagine if your customers never complained about service quality because it’s never an issue. Imagine never getting penalized for violating an SLA. 

While this may seem like a dream, this is exactly what data-driven operations is all about. Identifying negative trends as they appear, understanding any potential impact to customers and the network, prioritizing fixes to maximize impact and minimize service issues, reducing cost of operations and providing insight for capacity and service planning so that your network can continue to grow. 

Having the 100% visibility of service and network performance and knowing how best to address them is the key to proactively eliminating issues before they become service impacting.

First things first

Anyone who works in a network or service operations center can tell you there’s no such thing as an issue-free network. In fact, some days it can feel like the network is a ‘sea of red’. On those days, when the pressure is on to get the most important services back up and running, sometimes the biggest problem is knowing where to start. Which customers are the most critical? Is there a common root cause? Can services be routed without causing even more problems? Which faults are ‘must fix’ versus ‘nice to fix’? And as networks become virtualized, it becomes harder for humans to visualize the service and network topology making it more difficult to know where to start. 

Data-driven operations can eliminate this all together. By applying big-data analytics, driven by consistent policy enforcement across all network and service KPIs, a plan of attack can quickly be developed that takes into account everything from getting critical customers and services back up, to total time of outage, to impact to revenue and cost—without any guesswork or emotional bias.

Managing change

There’s a new paradigm emerging for customer managed services. Driven by competition and a demand for more application-specific services, customers are now able to acquire and manage their own services. Through online portals, customers can now do things like order new services or change an existing service, perhaps requesting more bandwidth, and have it happen within minutes. And while this is great for attracting and retaining customers, it makes the job of service and capacity planning more challenging. Can the network support an additional service without impacting existing services? Is there enough capacity to support the additional bandwidth requested? 

Without this information, there’s a real risk of not being able to meet customer commitments. Data-driven operations can address these questions in real-time. By having visibility into things like unused capacity, a new or modified service can quickly be configured. Additionally, if the new service looks like it might cause problems elsewhere in the network, optimization of service routing across a larger portion of the network can quickly be done to ensure the new service can be added. Finally, since the system is always aware of capacity throughout the network, capacity planning methods can be developed to support a just-in-time methodology to upgrades, thereby reducing or eliminating unused capacity.