Planning for a Disaster
After being in business for a while you hear some crazy stories. I’ve been lucky to be involved in a couple of them, and some of them are bad for business and I’m going to be talking today about how they could have been avoided. One of my favourites is this potential client that called me up and they were still in the process of using our services.
What to Do When the Server’s on Fire?
They called us up and said, “Josh, I need you to get here right now. Our server’s on fire.” And I said, “Okay, now is it really?” They said, “Yeah, yeah, the server’s on fire!” I said, “Have you tried turning it on and off again?” They said, “No, it’s actually on fire!” I said, “What? Why are you calling me? I’m not the fire brigade. I can’t just reboot that and turn that off remotely. If it’s on fire, it’s on fire. That’s stuffed.” They said, “Ah! Ah!” Anyway, they got off the phone and they did what they could do to get it rebooted and by that I mean, with a fire extinguisher. Once they did that they called us back and they said, “Okay, what can we do about the data?” I said, “Well, if you were under a managed agreement with us, we would have had a backup for your data, backup for everything, and we would’ve made sure that the hardware set that you had was under warranty.” I said, “Because you were still in the discussion with us, we haven’t done all that research yet for you guys, but let’s have a look.”
Dust sucks
We went along and had a bit of a sneak peek at this server. We took the cover off (which is standard practice when you bring on new business) and it was full of dust. Now not as much dust on this one anymore because the dust had burnt away, but it caused a fire and it just goes to show the amount of downtime that can be had if you don’t do some of the simple things. You only have to take a water drip dripping for many years and it’ll erode away a bunch of lands, and this was dust that had just built up for many years and it completely shut down the operations of their business with 65 employees for two weeks.
All About Preventative Maintenance
An ounce of prevention is worth a pound of cure and that is exactly what we’re talking about. That was just a crazy situation and they came across to us and we were able to get everything up and running reasonably quickly. However, if they had already put preventative methods in place and a disaster recovery plan in place, we could’ve much more easily been able to help them out. They would’ve been up and running in hours, even with a situation that was out of their control as opposed to something that could’ve been prevented.
Lights, Camera, Action!
The next story, the outcome isn’t quite as great. We were talking to this client and we said, “Look, we want to see you guys have some security cameras.” And they said, “Yeah, yeah, we’re going to get them. Too expensive, too expensive, too expensive.” Well, they were and a whole bunch of their vehicles was stolen. Now when that happened, they’re like we got to get security cameras now, right now. I said, “Yeah okay, yeah we’ll get onto it. We’ve just got to order them.” We started going through the process of ordering them. Only a few days later, they were ram-raided a second time. This time a police car was stolen and they’re like, “We’ve got to get the cameras up now.” I said, “No worries, we’ve got the cameras up,” and so everything was all sweet. You know what they say, it comes in threes, and this was no exception. The third terrible thing happened for their business and that was a fire. Now the business is a smash repair company and the fire happened in one of their spray booths and when that happened, you saw it on the security footage, it was there, it was there, a huge fire, the camera went black, and all their power went out everywhere. We called them up because we noticed all their equipment went offline and they’re obviously right in the scene of the fire at this stage so they couldn’t really talk about the servers. As you’d imagine, pretty, pretty full-on.
Insurance Matters
These are the things can be the end of a business if you don’t have the right insurance. The amount of bad luck that they had, now luckily those three things happened and that was a number of years ago and they’ve not had any bad luck like that since. But these things happen as it goes in business and you’ve got to be prepared, you’ve got to make sure you’ve got the right insurances, you’ve got a disaster recovery plan in place because the last thing you want to become is one of these stories.
The Final Word
As much as I appreciate the content, I would much prefer to be talking about a happy story and how something really great was avoided rather than the other way around. You need to be prepared for the worst at all times. Unfortunately, it is a part of life. I hope this has been a bit of a lesson for everyone and if it has and you’ve enjoyed it or you’ve got your own story that you’d like to tell us about, let us know. Stay good.
