Never EVER leave vendors alone (in your server room)
IDF and storage space.
On a particularly busy day, I had an interview for a new team member at the same time a technician from a local ISP was going to be onsite lighting up a new internet connection. Both of these things are very important to me, but sadly I can only be in so many places at once (still working on fixing that little problem). My team was briefed and aware of the technicians pending arrival before I went off to my interview. Most importantly they knew exactly where in the server room the drop was going to be installed.
As my interview is winding down, I get a text message from a friend in Accounting (which sat right next to our server room) that said, very casually, “I think your team may have set the server room on fire”. However casual the message may have been, it concerned me a great deal. Partially because the server room on fire is a “Bad Thing” and partially because this information is not coming from my own team. I bid my candidate a hasty adieu and rushed upstairs.
As was eluded to earlier, the team member in charge of keeping an eye on the ISP Technician got called away for a “higher priority” issue. They were a junior member of the team and didn’t understand the importance of keeping an eye on vendors. As soon as the technician was left alone, he pulled out a 120V UPS to provide battery backup for the head end equipment he was installing. In any normal circumstances, I would have considered that a bit silly since most server rooms tend to be battery/generator backed already. It was made doubly silly in this case since he put his UPS on top of one of our existing UPS units. So where did it all go wrong?
Well, for whatever reason the server room was wired by the previous tenants at 220V AC. We never changed this since it isn’t a problem for most equipment to use. Almost everything is designed to be 100% international these days, all you need to do is look at the back of any power brick and you’ll see the input is rated for 110V-240V (or somewhere thereabouts). However, 120V UPS are NOT designed to take voltages much beyond 130V. What happens when you plug in a 120V-only device into a 220V socket?
Yup, he managed to find the one standard looking outlet in 20 feet (a C14 to NEMA 5-15R adapter we had left plugged in, and unoccupied), and plugged into it without question. Somehow the ISP’s UPS did NOT immediately fry, so he had enough time to plug in his other piece of equipment and leave. A few minutes later… it died in a rather spectacular and final fashion. About the time my team realized the server room was filling with smoke they yanked the power and started ventilation. Again, fortunately, no fire.
The moral of the story is that magic smoke is bad for battery backups and more importantly… NEVER LEAVE VENDORS ALONE.
(PS. A year later when we see that ISP’s vehicles around the area we still grab for the fire extinguishers. Ya know… just in case.)