March 5, 2021

1794 words 9 mins read

Ciao Chao Smule

After 3.5 fun filled years, it is unfortunately time for me to depart from the Smule family. There have been great changes, both for me and the world, with many fond memories. I can’t possibly recount them all in a single blog entry, but I wanted to share a few thoughts.

My time at Smule started in November 2017. While I had previously been doing a lot of IT Management, I’d recently pivoted over to DevOps with a heavy security background. Smule started me out as a regular run of the mill SysAdmin which I was perfectly fine with. Terrifyingly (for both knowledge and security reasons) by the end of my first day I had root access to more than 3,000 physical servers spread across 4 data centers on 2 continents. While I’d had access to many AWS instances before, it just seems so much more real when you can actually touch the hardware.

Many months were spent onboarding, documenting, and doing various Ops team side projects that no one else wanted to deal with. Learn Cassanda? Sure, why not! Move the wordpress blog? Can do. Try to write an admin tool to keep track of every piece of hardware? I’ll give it a shot. But seriously, there was a lot of documentation to be done in those early days. Confluence had lots of specifics, but not much generic day-to-day information when came to administrating the servers. When you hire only one sysadmin every 6 months, I get it. However I wanted to make sure I didn’t have to train the next guy, so I wrote down as much as I could. Years later, a large portion of my Ops runbook is still in place - even though I haven’t been doing sysadmin work for quite some time. If you’re a sysadmin (devops, SRE, whatever), go to your runbooks now and see if there is anything you can update. Bitrot is terrible. Up-to-date docs are wonderful.

As time went on, honestly not that much time, Smule started to experience more security concerns than could be casually addressed. The Ops team put together a special strike team to address these concerns. With my background in Security (most recently having been in FinTech) I, of course, volunteered… and somehow ended up as the leader. This quickly became a full time job and being a Sysadmin turned into a hobby. A hobby that I would continue to keep until the very last days. Why does security own the Blog and GitLab? Well because Jon set it up and no one else ever needed to deal with it, of course.

In the first years of Smule, it was such a joy to be able to run about doing all sorts of random projects. We had one dedicated team member working on the data center Operations and his job was not made easy by us in sysadmin. Since I love playing with hardware, I managed to convince him to give me access (ok, we all had DC access for “emergency/backup” reasons) and used basically any excuse I could to go to the data centers (at the time our closest DC was just a few minutes from the office). With the migration to the cloud it’s so rare to actually be able to enjoy the sights and sounds of a real data center. There is something truly amazing about tens of thousands of servers in one big room, humming (or screaming) away.

As the Security Operations grew in work, so did my team. It was a bit of a rocky start at first as our Security team members were hired in Sofia, Bulgaria. Managing someone is a skill, doing so when they have a 10 hour time difference is an entirely different set of skills. After 6 months of my first team member in Bulgaria I finally got a chance to go out there (in the spring of 2019). Unfortunately the one team member I had left the company only a few weeks before I arrived in country. However as luck would have it I would get a chance to interview new security engineers in person. One of those candidates would be the right person for the job and would make an excellent addition to the Smule team. Two years later they are still here doing amazing work.

That trip to Bulgaria was also notable because my wife came with and we turned the trip into quite the European adventure. We included a little extra time to see the sights of Sofia (which if you’ve never been there, is really quite a lovely place to visit). After that we went to South East Germany (Bavaria) to visit some of my family for a few days. We also stopped in Wein (Vienna), Austria. Lastly we spent a week in Minsk, Belarus where my wife also had an office to visit. Needless to say that trip could cover several blog entries (and hey, who knows, since we still can’t travel… maybe I’ll reminisce).

Smule is a company about music, especially bringing the social aspects back to music. So it was a very relaxed and fun environment. Many hijinks were had in the office, though less nerf wars after the CEO found us a little too… disruptive. Many a late afternoon was spent arguing about the best affogato or the best way to make it. To this day I still don’t know why I was ever involved in those conversations. I’m lactose intolerant so they know I can’t enjoy it… and maybe subtly torturing me was the reason. Ramen lunches were always the best. There is nothing that will ever be better than delicious (and free) ramen served to you in office with a buffet of toppings for customizing to your hearts delight.

Teams would all grow and change and adapt to the times. While I was still an honorary member of the sysadmin group, they also made sure I didn’t do too much damage with my hobbies. After all I was the first to stand up Kubernetes at Smule, to run all the security tools. I was the only person in Security running tools (in the early days), the reliability of Kubernetes made sure I got to sleep through the night. I don’t regret that choice in the least.

Come the spring of 2020 we added a new team member to the Security group, also in Bulgaria. My boss insisted I should go out there and work with them in person for a little bit. My plan was to go in late April after I got back from a trip to Japan in March which had been in the works for almost 2 years. However at the boss' insistence I ended up headed to Bulgaria at the end of February. It’s wonderful to spend time with the existing teammates you’ve spent so much time with on Slack and Zoom. Being able to just go out to lunch and “hang out” makes all the difference sometimes. It was also amazing to be able to help onboard my new hire. The tail end of that trip was a few days in Munich, Germany seeing family.

The day after I landed back in the USA was the announcement of the travel ban from Europe, because of COVID-19. Going to Europe had been a little stressful since Italy was just starting to get bad. By the time I left, Lufthansa was starting to cancel flights left and right. But the US Travel Ban just threw everything into high gear. I was very grateful of the timing of my trip, even though it had been slightly stressful. Of course at this time lock downs also became a world wide phenomenon followed quickly by a rapid work-from-home shift.

For our team in the USA, it wasn’t a big deal per se. Everyone had previously been allowed to work from home at least one day a week, maybe more. We knew what to do already and most people had some version of a home setup. Of course everyone would have to adjust to kids being home, everyone being locked in, effectively permanent WFH… but the important part was that we could do so from our technology and security side with relatively little pain.

For at least a year prior to COVID-19, I had been working on a project that I called “BeyondSmule”. Which is clearly named after Google’s BeyondCorp concept of zero trust security. Originally I had started it because I was both annoyed that it was hard to find internal resources… and because I was lazy. When working from home it was obnoxious to get on the VPN just to check the user corporate directory (for example). As I was doing less sysadmin work, having to use the VPN to access web based resources just seemed like overkill. Not to mention it was silly to have access to EVERYTHING when I only needed access to one small thing. With everyone working remotely post-COVID-19, the usage of my BeyondSmule technologies went through the roof. While we would eventually replace my homegrown scripts with Cloudflare Access, the project of moving to a VPN-less world has not stopped at Smule.

Midway through 2020 I was also given the IT team to manage. While I had spent a fair amount of time pivoting away from direct IT management, the unfortunate loss of our previous IT Manager necessitated my return. Most of my time was spent understanding how things were set up and updating policies & procedures. The Sr IT Engineer on the US side had taken over a majority of the tasks IT managers often deal with: purchasing, all hands announcements, and credit card reconciles. We didn’t find a reason to change how things were working since he had almost 4 years in Smule IT and knew exactly how everything needed to be. Because of the excellent work of all the IT team members (both US and BG), managing the team was a breeze. If every IT Manager gig I had was that easy, I would have never pivoted out.

In the last few weeks of my time at Smule I’ve taken a little time to stop and reflect at what a wonderful group of people we had. Not everyone got along all the time (hey, often security’s job makes other peoples' lives more difficult - it doesn’t always make you friends), but we had a great group of smart people. Managers left us to do what we did best… and also helped us when we needed it.

To Parker, Alexander, Elena, Chamreun, Aleksander, Dimitar, Alexey, Asen, Mike, Michael, Richard, Ray, Stefan, Ventsi, Alex and all the rest (both past and present). Thank you. It was truly a wonderful place to work.