Operations Measures

Recently I was asked a question about a performance metric one might use to measure the performance of a NOC. Before responding I contacted my "personal network” of data center experts. This is a group I worked with in previous worlds all of whom have built data centers from the ground up. In the exchanges that ensued, all of us agreed on one fact: Everything really comes down to how one performs against the SLA as well as efficiency and effectiveness.

However, I have never been one to tie everything to a single metric without giving it some thought. There is a need for a balanced set of metrics; not just one. The real issue is to decide...
1. What is important?
2. Can we measure it?

Importance - What Is Important to the Company?
What ever metric we come up with we should think first about what is valued by the company. This presents an opportunity and it should be looked upon as just that: An opportunity to get some buy-in by management. Get them involved! Facilitate an exercise to agree how what we do in the NOC can impact the organization's strategy. Use this discussion to prioritize what is important to the company and to highlight those things over which you have no control! Management may never have given thought to the importance of measuring what is, essentially, their strategy. At the least they should recognize the value of aligning what is important to you with what is important to them and measuring that in some meaningful way. As a seed to what might be important, consider some of the issues that are typically important to management such as projects focused on the direction of the company, the company goals, strategy, and plans, or improvement efforts either internally with processes and internal transactions or externally with customers. And of course there is always interest in the improvement of the cost/value ratio.

Capability - Can We Measure It?
The relevant question is whether you have the capability, tools, and controls in place to measure. Most organizations struggle at this point. They are caught in a "rock and a 'hard case'" between measuring "what they can measure because they can" and measuring the wrong thing because the metric is there: convenient and available. While you don't want to measure the wrong thing, you also need to balance the investment required to measure the right thing. Rest assured that if there is some activity or transaction, there IS a way to measure it. But the rule that applies here is the K.I.S.S. principle. Just keep it simple for now. As your internal processes mature and your capability increases, you can always change the metric later. But don't get yourself in a situation where you have to measure something that is not measurable ("saved jobs" comes to mind) or a metric that requires labor and costs that exceed the value of the measure.

Other Considerations
A single metric is not ideal. One metric, taken in isolation, may reflect a bias or be based on a systemic fault that is not detectable in just one number. This could lead to decisions that are based on invalid assumptions. I would also have the following concerns:

1. We have already evaluated what we can measure. But are we sure we are measuring what we think we are? Challenge the metric and see if it answers the question(s) you are asking. Determine if this metric tells you or the target audience what they need to know to take action.

2. Are we sure our internal staff, who may become aware of how they are being evaluated or how they personally will be compensated, cannot distort the data? An astute NOC manager, engineer, or administrator can easily game the system to favor desirable measures.

3. If our metric is aligned with growth, what controls do we have over the sales cycle and growth? We need to think of what we are measuring in context of the system within which we operate.

Balance In Context of a Capacity Plan
For all of these reasons I would favor a critical few set of metrics balanced across the four domains (financial, internal, customer, L&G) that many will recognize as Kaplan and Norton's Balanced Scorecard. You may think of any number of measures:

• Improved time to market (getting the customer up and running)
• Costs balanced with growth and business need*
  
• Asset utilization balanced against spend*
  
• Ratio of staff to transaction volume*
• Revenue or profit growth versus investment in the data center*
• Capital preservation*
  
• Reduction in cost of operations (cost reduction over time indexed against customer or transaction volume)*
• Cost-effective delivery of service
• Revenue/ sq. ft.*
• Renewed contracts/repeat business

These may come together under four high-level domains. Essentially, whatever we come up with to evaluate performance should incorporate measures balanced across these four domains:

1. Financial - cost, savings, efficiency, capital preservation, asset utilization
   
2. Internal - effectiveness, total number of proactive actions or corrective actions
   
3. Customer - response time, customer satisfaction, SLA performance, trend analysis to prevent re-occurrence
   
4. Learning and Growth - staff certifications, staff retention, goals alignment
   

The asterisked measures in the above list might be identified in a capacity plan. In its fully developed state, a capacity plan is the STRATEGY of the organization – what’s important to management and/or ownership. Most successful organizations want to measure what is important to them. In turn, you want to tie the metric that is used to evaluate the NOC to something over which you have some control. And, if things change such as business direction, resistance to fund an initiative, recession, etc., you want to have a means to identify that variable early. The capacity plan does just that. It allows you to get agreement on what is important and gives you the control over how to measure the strategy.

The capacity plan outlines the current state, prospects for future state, and the recommendations and plans to meet the prospects for the future. This would allow one to work closely with the company management, ownership and sales force to measure how "the company" performs against the plan. Since, by definition, a strategy has to be signed off by senior management, it represents the stated goals of the organization and buy-in of all running the company. A capacity plan has to include both LEADING and LAGGING indicators. A policy to review regularly, would ensure the capacity plan was evergreen and your group's performance was aligned with the business conditions. If leading indicators suggest an issue, an adjustment is made to the plan.

In Summary
Whether a capacity plan is used or not, if a single metric is required, consider building a matrix of multiple functional measurement elements:

1. Agreed and documented responsibilities in NOC
2. Metric associated with each of those responsibilities – how they are being measured – if they can’t be measured, then they don’t go into this first round
3. Add a weight to the responsibilities based on importance to the organization (you’ll still need to get agreement here just as you would have for the capacity plan)
4. Do the math and come up with a score
   

Finally, I would revisit the metric(s) at least annually (if not more frequently - perhaps quarterly if the organization is young and undergoing dynamic changes) as your capability to measure will have improved, metrics may have evolved, and the company values may have shifted.