Never EVER leave vendors alone (in your server room)

Gather round kiddies and I’ll tell you a story that is 100% true and happened to myself, my team, and my company. Since all humans make mistakes and those shouldn’t be held against them, I will not be naming the parties involved. The moral of the story is the very title of this post, never ever leave vendors alone. Especially never let them alone in the server room. Sure, there are security implications (they could really be pentesters, or just plain bad guys attacking your network), but mostly it’s not safe for you or your equipment.

Allow me to set the stage, for those who don’t know. I work in a decent sized startup based out of San Francisco. As it happens I’m the IT Manager, in charge of all things internal technology related. We operate out of a multi-story office building and while we’re 100% cloud based, we do have decent little server room (~750 sq ft) that doubles as our IDF and storage space.

On a particularly busy day, I had an interview for a new team member at the same time a technician from a local ISP was going to be onsite lighting up a new internet connection. Both of these things are very important to me, but sadly I can only be in so many places at once (still working on fixing that little problem). My team was briefed and aware of the technicians pending arrival before I went off to my interview. Most importantly they knew exactly where in the server room the drop was going to be installed.

Shortly after I went to interview the candidate, the ISP technician arrived and my team escorted him into the server room. Now, neither they nor I knew exactly what the tech was going to be doing (which is, of course, an oversight), but they provided they tech the information needed and let him go about his merry away. Being the early days of the company, the IT department’s team wasn’t very large, but I trusted my team to take care of business when I’m not available. Unfortunately being a small team means that sometimes people get pulled away in the middle of “very important” tasks for “even more important tasks”. The usual “If the CEO says ‘jump’, you jump first and ask ‘How high’ later” sort of thing.

As my interview is winding down, I get a text message from a friend in Accounting (which sat right next to our server room) that said, very casually, “I think your team may have set the server room on fire”. However casual the message may have been, it concerned me a great deal. Partially because the server room on fire is a “Bad Thing” and partially because this information is not coming from my own team. I bid my candidate a hasty adieu and rushed upstairs.

What I found was not a fire… fortunately. However, all the doors to the server room were propped open; several box fans were going at full power; my team look winded; and it smelled funny. A short investigation was launched and it didn’t take long to get the full story.

As was eluded to earlier, the team member in charge of keeping an eye on the ISP Technician got called away for a “higher priority” issue. They were a junior member of the team and didn’t understand the importance of keeping an eye on vendors. As soon as the technician was left alone, he pulled out a 120V UPS to provide battery backup for the head end equipment he was installing. In any normal circumstances, I would have considered that a bit silly since most server rooms tend to be battery/generator backed already. It was made doubly silly in this case since he put his UPS on top of one of our existing UPS units. So where did it all go wrong?

Well, for whatever reason the server room was wired by the previous tenants at 220V AC. We never changed this since it isn’t a problem for most equipment to use. Almost everything is designed to be 100% international these days, all you need to do is look at the back of any power brick and you’ll see the input is rated for 110V-240V (or somewhere thereabouts). However, 120V UPS are NOT designed to take voltages much beyond 130V. What happens when you plug in a 120V-only device into a 220V socket?

Magic Smoke!

Yup, he managed to find the one standard looking outlet in 20 feet (a C14 to NEMA 5-15R adapter we had left plugged in, and unoccupied), and plugged into it without question. Somehow the ISP’s UPS did NOT immediately fry, so he had enough time to plug in his other piece of equipment and leave. A few minutes later… it died in a rather spectacular and final fashion. About the time my team realized the server room was filling with smoke they yanked the power and started ventilation. Again, fortunately, no fire.

Since there was no damage and my team handled everything fairly well (considering) in the emergency situation, I was mostly amused by the entire situation. Even more amusement came later when I called up the sales rep at said ISP and inquired why their technician had attempted to set our office on fire then promptly left for lunch. The technician was called and came running back, with no idea what he’d done wrong. That made me slightly sad, but I’m just hoping he was brand new and learned a REALLY good lesson that day about double checking before plugging in.

The moral of the story is that magic smoke is bad for battery backups and more importantly… NEVER LEAVE VENDORS ALONE.

(PS. A year later when we see that ISP’s vehicles around the area we still grab for the fire extinguishers. Ya know… just in case.)