The wonderful madness of metrics: Different things to different folk

GIGO

A lot of comms links' up/down times are measured from a log of SNMP polled events. When we looked closely at some anomalies we began to realise that market leader products mistakenly presented these SNMP events in aggregated SLA reports as unambiguous up/down states.

For example: a link up or down state was often only represented at SNMP level by a "changed" indicator - from which the reporting application deduced an up or down event. Repeated change events within a few minutes led to an incorrect deduction about the state of the link at that point.

When there were multiple resilience links the matter became even more complicated. Contractual outages were being reported when in fact the end to end path was unbroken.

It took us some thought and effort to write our own link break interpretation algorithm. Some unstable line conditions required a human to make an educated assessment of what had probably happened. The reporting application supplier wasn't really interested - apparently on the basis of "no one else is complaining".

We tried to be scrupulously honest about the interpretations so that the reported SLA for an end to end link was as near to the probable truth as possible.

This was only a few years ago - it would be interesting to know if the standard SNMP signalling and interpretation have overcome these deficiencies.

You get exactly what you didn't pay for....

and since you were a cheap b@st@rd you got nada.

Let's face it, even the expectation of 5 nines is unrealistic in most cases as there are just too many vendors and thus too many points of failure involved.

Use a datacenter located nearby and expect to have a backup ready and waiting if you need that kind of reliability.

Charge what that is worth to the customer.

If they can't afford it, propose less critical service and go to the cheapest datacenter vendor.

Explain the proposal in detail so they have a real understanding and get a signature that notes they have read and agree to the T's &C's if they sign an agreement.

Five Nines

Since 25 seconds or even 5.x minutes are pretty hard to detect or even poll reliably, I mentally use a 'five nines' that an old IBM mainframer told me - 'Yep, we reckon on an hour of downtime a decade'.

I'll bet nobody out there is delivering true five nines ;-)

Targets

Give me a stupid target, and I'll find a stupid way to meet it.

Best advice I've been given in IT.

detection

Had a vendor sneak in a clause which allowed 20minutes to detect an issue. This first 20m would not count as downtime...

Transit vans

Interestingly, the use of a "fully loaded transit van" as a yardstick (metric?) for measuring the porosity of legal agreements fits nicely with the theme of the article. Is that a transit van stuffed to the roof inside but looking from the outside no different from an empty transit van? Or is it like the sort of thing that you find in e.g. Asian countries, piled high on the roof with luggage and with people clinging to every possible handhold on the outside and with a profile that is about twice the size of an empty van?

Bah!

The shenanigans start even before the SLA is proposed and signed. Long before this the marketing drones will be in-theater bleating about 8X performance gains over the market leader.

Just don't ask anyone what "X" means. It is supposed to confuse you into believing it means "times" but careful examination of the performance graphs on the powerpoint slide being shown "to illustrate" will show that by their own figures, 8X (Market Leader) is closer to 6*(Market Leader).

Real example from real vendor's shill. Unfortunately they also fielded The Germanically Efficient Attractive Woman In A Suit Who Knows Computers and The Unbelievably Scruffy Bearded Guy Who Talks UNIX Internals For Fun to bracket the one who was supposed to be watching and catching this stuff, and he was so busy showing off his own chops that he forgot to pay attention.

Well played [REDACTED] corp. Well played.