Sorry for my absence, dear readers, but I was out for a well-deserved vacation time with my family. Not that I don’t like you, but you know the security observer needs, sometimes, to take a moment to refresh his mind and be prepared for another year of security monitoring and analysis. And, by what I can see right now, the year is going to be full of surprises.
Speaking of surprises, today I wanted to talk about the security mistakes we frequently see in the course of our work: companies aren’t prepared for the worst.
Disaster Recovery: Are You Ready?
One of the first mistakes I often see in companies is their lack of attention to their disaster recovery plan. I’ll give a good example. I used to work for a big IT company which was doing business worldwide. The data centre was in a major North American city. I was charged to evaluate the security of this data centre. Globally, it wasn’t that bad. It wasn’t excellent, but at least they had a good grasp on basic security principles.
The major problem was when I asked them if they taught about natural disasters. What would they do if they were in a situation involving the complete destruction of their data centre. Well, they had a backup centre, which is great! The tiny little problem was linked to where the backup centre was. After conducting a full audit, I was informed that the backup centre was at a distance of… 2 miles!
I guess you can see what the problem is here. If there is an earthquake or a major flood, both data centres are at risk. That is an excellent example of a common mistake in security: you think you have your back covered, but you don’t because you never taught about some dreadful possibilities. And, sometimes, you can’t even imagine the worst thing that can happen. Let me give you an example.
The Case of the Domino Effect
One of my colleagues in the security field recently told me a story that is so scary that at first, I thought he was kidding. It is a very good example of how a harmless situation can rapidly turn into a real security mess that might be dangerous for your whole company.
This story starts with a bolt. A loose bolt actually. There was a loose screw on a container outside of the wall of this company. The security agent is doing his patrol and sees it. He gets back to his office and grabs a tool to screw it back in. He is a nice guy, you see! So, the security agent tried to screw back the bolt. He used his tool but unfortunately, the bolt broke. And it broke badly. So badly in fact that the liquid inside the container leaked out and burned the agent. That liquid was a coolant used to cool the server room.
So yeah, now it is a dangerous mess. The liquid is falling on the ground, it might put your servers at risk, and you have an injured person. The liquid is also an environmental hazard, so the authorities have to be contacted quickly since an intervention is required.
But that’s not all!
The liquid is leaking and, against all odds, it finds its way into the building just between the ground and the concrete slab. It found its way, of course, right into the server room. At that moment, the chemical reaction between the concrete and the liquid coolant created fumes. Those fumes were detected by the fire detection system which was then activated.
Since it is in a server room, the system is using gas; a gas that is supposed to remove oxygen from the room to kill the fire. The gas is pushed from a gas container. Unfortunately, when the system started, the gas container shook way too much. So much that it shocked the hard drives in the server room, killing many of them. Those hard drives were used in a cloud system and were hosting critical corporate software. Yes, there was a backup elsewhere, but the damage done to the server room took three days of work to get back into working order, creating millions of dollars of losses for the enterprise.
The lesson here? A security incident can have implications that are way worse than what can be expected. A domino effect is always possible, and you cannot prevent it. So, all in all, the problem is not the fact that company leaders are making mistakes in regards to security questions. The problem is that they don’t care enough about security, or that they don’t learn from their mistakes. But, as you can see, you better be prepared for the worst because even if you think all your bases are covered, something even worst can happen.
