One of the best projects I worked on had zero-overhead communication

There’s a project I worked on some time ago that provided me one of the most pleasant experiences I had. When I say pleasant, I mean it in the project management sense.

The work itself wasn’t that fancy: I had to migrate a lot of machines to a new system. Nothing cutting edge, no new products, just some migration to a much better system that would allow us to control the machines smoothly. The trick is that these machines were serving a really big number of customers, and we couldn’t just stop them. On top of that, the code running on the machines was in some cases over 10 years old, and of course we didn’t have a single person with knowledge about the entire thing.

I don’t really want to talk about the technical side of the project because it’s boring: you perform a few tests with machines not serving production traffic, then you start to slowly roll out the migration to machines serving customers, you’ll then learn of a few scenarios you missed because the scale of the traffic you tested with wasn’t big enough (a good problem to have?), and then you figure out how to deal with that, incorporate the learnings into the product code, and resume the migration.

What I want to focus on, though, is my experience leading this migration, and why I think this was a very pleasant experience.

Given the nature of the problem, it was somewhat easy to convince management that there really wasn’t a timeline we could realistically follow. Machine migrations take time, you need to do it under heavy control to avoid cases where too many machines are brought offline and you’re unable to serve your traffic. Sometimes, you’ll also need to replace certain machines, because they could have some underlying issues that you just can’t manually investigate due to the scale (this was a migration of over 20,000 machines). We also had 2 people dedicated on this, and we knew we’d hit some roadblocks along the way, but we didn’t really know which ones. Nobody knew 100% of the code running on the machines, as it was a really big codebase.

Even though management was ok without some kind of “hard” deadline, we still aimed at moving things as fast as possible. Some prep work would need to be done before the migration, and since we’re talking about a Big N company here, you can imagine all kinds of internal systems we had to mess around with to make this possible.

Here’s the really nice part: I felt like the other person working on this project was completely in sync with me. They didn’t know everything involved in the migration (I handled a lot of human coordination transparently for them), but they knew enough about the code running on the machines to understand all the prep work we’d need. They knew about some edge cases we’d need to cover. When talking about the initial steps for this project, things went so well and we finished discussing so quickly that I even became worried they hadn’t understood some of the things we’d need to do. I let them roll with it, though, and when reviewing the code patches I was really surprised: in the few minutes we talked about the prep stuff, we’d synced about the approaches to take, and then we just started doing it. Contrast this to either of the following two (way more common) scenarios I encountered in the past: someone needs a lot of help to understand parts of the code or the system, and needs some mentoring to figure things out; or someone holds some strong beliefs about the way things should be done, and an unproductive discussion arises about how to do things, when pretty much every alternative proposed is good enough and there is just no right answer.

When we found a roadblock, syncing up was also so easy and smooth, because nobody held any strong opinions on which approach to take. We knew which alternatives would solve the problem, and we favored gettings things done rather than finding some mythical perfect solution. Reporting on the progress of the migration was also pretty smooth. We kept a simple spreadsheet with a bunch of information of things that were already done, and things that still needed to be done, and both of us dedicated a few minutes every day to update things there. If we were to track those using the company’s task management system, it would’ve taken ages to update information the same way we did with the spreadsheet. There were days we didn’t even talk at all, because we were aware of what needed to be done, and we knew what things were already done just by looking at a somewhat simple table. No standups required.

Overall, working with this person felt like I was just working with an extension of myself. I say this in the good sense. Working with people all in the same mindset might skew projects towards some below average solutions and not bring enough diversity to the table. This wasn’t the case here. We both thought about different approaches to solve the problems we faced, but we also knew well how to evaluate them, and knew when they were good enough and discussing more wouldn’t help much.

After we finished this project, I realized that I had become addicted to the way this project was going. It was just so pleasant that I didn’t want it to end. To this day, I still regularly think about this project, and wonder if I’ll ever be part of or help build a team full of people like this. Is this how high performing teams work? Is this how working at very productive companies feels like? If it is, I want more of it.