If you ever maintained a live system - I’m sure you must have used the min-max alarms. However, are they always the best tool for the job?

Alarms should be the first line of defense against issues on production. It shouldn’t be a manual process that you discover something isn’t working as expected. It shouldn’t be a developer lurking into the logs. The alarm should notify you about a problem - page your mobile phone for serious issues, otherwise just leave a trace in yours ticketing system for further investigation.

Simplifying, in a monitoring system there are two main components - metrics and alarms. Metrics are the data sent by producers (hosts, application, infrastructure etc.), and alarms are just the components for monitoring the values. Let’s focus on the alarms.

Min-max (range alarm)

Assume that you want to monitor CPU utilization on your hosts. The natural choice here is min-max alarm. Having the utilization too high may indicate problems with scaling, bugs etc. Having CPU utilization too low usually also isn’t good - perhaps it’s a waste of resources, perhaps your traffic dropped too much.

Min-Max alarm on CPU utilization

Derivative alarm (rate of change alarm)

Now, imagine you are a website owner and you want to monitor the parameters regarding number of user registrations on your website. Usually, there is some kind of email validation algorithm in place - it can be dead simple like a regex, or more complex like an artificial intelligence checking if the emails are legit. Would you use a min-max alarm to monitor the number of invalid registrations on your website? What threshold would you set? Remember, that usually there are fewer registrations in night, than in during the daytime. You could try to divide the day and night periods and try to come up with min-max alarm based on that. However, the main disadvantage is that those alarms will become obsolete as hopefully your website will grow and the number of registrations will naturally increase.

In that case, you can monitor the rate of change - because the problem is when the metric changes rapidly. The rate of change is mathematically a derivative of a function, noted as f’(x).

The alarm rule looks like this: alarm if value > 3*f’(x) + 5.

Derivative alarm on the number of invalid email addresses used during registration

Another solution for described use case would be to set up the alarm on a mathematic formula - the percentage of invalid registrations (invalidNumberOfRegistrations/totalNumberOfRegistrations).

Standard deviation alarm (σ alarm)

Sometimes (usually…?) the production runs on many servers. In that scenario the standard deviation alarm come in handy. It is working on additional “dimension” - it doesn’t process one metric, instead it processes many instances of the same metric from many servers. It checks if the values are similar to each other - mathematically - it calculates the standard deviation. The most useful case for it is when you compare 1-Box environment to production environment. I wrote about that before: Rock solid pipeline - how to deploy to production comfortably? For example, can monitor the number of inodes used on yours hosts (because you can run out of space… without running out of space).

Obviously, the alarming state is when one of the host in your fleet uses more inodes than “normally” the rest of the fleet, for example, 2*σ+5000 inodes.

Standard deviation alarm on the number of used inodes

Generally, whenever there is need to compare an experiment (new deployment/new feature/new algorithm) to an existing one, σ alarms seems to be helpful.

So what metrics can you monitor?

…all of them! Stay tuned for the next post!

Operational Excellence series

1. Intro: What is Software Operational Excellence?
2. Deploying: Rock solid pipeline - how to deploy to production comfortably?
3. Monitoring&Alarming: Types of alarms - what’s beyond min-max checks?
4. Monitoring: What service metrics should be monitored?
5. Scaling: (Auto) scaling services by CPU? You are doing it wrong
6. Scaling: How do you know the right maximum connections?
7. Scaling: How to estimate host fleet size? Why keeping CPU at 30% might NOT be waste of money?