Mixing clever and stupid

The most valuable technologies I have built had at their core some pretty clever ideas. That wasn’t all of it, though. They also had some stupid ideas. Not stupid bad but stupid crude and stupid simple.

For instance, I worked on a load-balancing system with some ingenious methods for distributing load across a pool of servers. I also had a service registry. Kind of. They were text files with hostname:port pairs, one per line. The core idea was really clever. The service registry? Dumb as a sack of hammers. This is the sweet spot.

The easy case is something that is stupid top to bottom. If there are no clever bits, what is it doing for you? It might be doing something useful, but it’s probably not that useful. It’s probably inflexible and narrow, like a basic shell script.

Consider a project that is clever all the way through. Maybe the core of it is some insane gradient descent algorithm, and that should be clever. But what about its logging? What about how it takes input? What about how it allocates and frees memory? Do those need to be clever? Not only do they not need to be clever, but it’s almost certainly harmful if they are.

Every bit of cleverness adds implementation cost, maintenance cost, failure modes, and difficulty debugging. Software programs follow the Pareto Principle, but to an extreme. Instead of 20% of the code contributing 80% of the value, it’s probably 4% of the code contributing 96% of the value. The rest is just bookkeeping and logistics. You certainly want those things done well and efficiently, but they’re clear candidates for satisficing. Save your maximizing for the 4%. Making the 96% fancy won’t make your product any better, and it’ll almost certainly make it worse.

Putting flaws in a plan in context

Every plan has flaws. If someone presents a plan to you, it’s definitely helping them to identify flaws in their plan. When you do, though, distinguish between fatal flaws and fixable flaws. Does the flaw you’ve found mean the plan should be thrown out entirely? Or does it identify a way to improve the plan? What I’ve seen most often is that someone identifies a fixable flaw but it’s heard as a fatal flaw. This does not tend to be an enjoyable experience for either party.

How I do skip levels

I have a large organization to keep track of. It’s easy for me to get out of touch with the day-to-day work of people actually building the software that powers the business. It’s also easy for them to get disconnected from leadership. One of the ways that I try to address this is with periodic skip-level meetings.

A skip-level meeting is just a meeting with someone who indirectly reports to you. I have set up recurring 1:1s with every manager and senior individual contributor. I’ve set the schedule at one half hour every six months because I don’t have the capacity to talk to every one of those individuals more often. I think this is the lowest acceptable frequency Quarterly is good as well.

One thing I try to do is to sent them a message explaining the meeting when I schedule it. Getting an invitation to have a mysterious 1:1 meeting with your VP could be nerve-wracking. I want to prepare them so they’re not nervous, or at least less nervous.

I see the agenda as being similar to a typical one-on-one with a direct report. It’s primarily their time. I do go in wanting to know how they’re doing, both psychologically and practically. I also want to understand if there is guidance they want me to give to their manager. Other than that, though, I want to talk about what they want to talk about. Sometimes it’s questions about our strategy, sometimes it’s about their career and growth, sometimes it’s about policy, sometimes it’s about specific technology… It doesn’t matter to me. I tell them not to worry about whether something is important to me. If they’re in my organization, and something is important to them, it is at least important enough for me to spend a few minutes discussing it with them.

There are three things I avoid doing in skip levels. One is asking for status updates. I should be able to get information on the projects and products some other way. I don’t want to spend their time getting me up to date. That serves me, and the goal is to serve them.

I also don’t give them instructions. That’s for two reasons. I don’t want to spend the time that way, and I want them to lean on the organizational structure for that. I want them to see their direct manager as the primary source of instructions, and only in urgent circumstances should I give instructions myself. This is also why I prefer to do these no more frequently than once per quarter. If they have a 1:1 with me every few weeks, then there’s a higher chance they’ll come to me for something instead of going to their direct manager.

Finally, I don’t make any decisions or commitments to do anything other than to look into something. Maybe they disagree with a technology choice, think our strategy is dumb, or have a problem with their coworker. These concerns are all important for me to hear, but there’s no way I can expect to fully understand the situation just hearing it from them. I’m not going to make a commitment to do anything before I understand the situation, and often the circumstances where they most want me to do something are the ones that are most complicated where I need to get multiple perspectives.

When we finish the discussion, I stress to them is that the recurring schedule represents a minimum. They can email me, chat me, or set up a (currently virtual) meeting in a free slot in my calendar. I want them to know I’m available to them if they need me. They need to know that I’m not too busy for them overall. I may be too busy at specific times, but I’m nothing without my team. If I can’t make time for them, and there’s no emergency going on, where are my priorities?

Go ahead, reimplement the wheel

Any time there’s a discussion on a technical team about whether to create something new, someone states the cliche, “don’t reinvent the wheel.” That statement is often enough to stop the idea in its tracks. It’s a dumb cliche, and invoking leads to dumb decisions.

Consider the actual wheel. Imagine you’re building a physical machine to do something. You go to your city’s one wheel store. Said store sells exactly one kind of wheel. You buy some of those and finish your machine? That seems pretty unlikely.

The problem is the verb “reinvent.” It doesn’t actually mean anything. Invention is an idea. Usually it’s reified in a prototype or implementation, but it’s just an idea. If you come up with the same idea independently, you’ve reinvented it. There’s nothing wrong with that, although it’s rare. More often you’ve learned the idea from elsewhere and thus can’t reinvent it. What you can do is reimplement it, to take that existing, already-invented idea and reify it again in some way suited for your needs.

The technology world is replete with reimplemented wheels. Storing data on your local file system? You’ve got flat text files, structured text files, binary databases files, relational database management systems, document databases, columnar databases, and so forth. And that’s without getting into network or cloud systems.

Let’s go back to this physical machine. Are you building a vehicle for transporting people? A fancy office chair? A treadmill desk? A back massager? A trebuchet? Each of those uses a wheel, but none of them uses the same wheel. They have different weight, strength, stiffness, size, and other requirements. Using an automobile wheel for a back massager would make for an unusable, ineffective back massager. It would probably injure you. The wheel is an idea, and there are many, many different implementations of it.

Does anyone say you shouldn’t reimplement the visitor pattern or the strategy pattern? These patterns are just ideas, and different products assemble different patterns in different ways that are better for some circumstances and worse for others. If there isn’t an existing component that mixes patterns in a way that’s good enough for your need, then, yes, reimplement the wheel!

Structured for conflict

A common pattern is to assign different domains to different people. Maybe you divide by function, like sales versus product management. Maybe you divide by territory, like Northeast and Midwest. Maybe you divide by product line. Then you set goals and incentives appropriate to each person’s domain. This works great. That is until you need these individuals to cooperate.

Remember how you gave them separate goals and separate incentives? You just guaranteed that they will find it difficult to agree. Client Services wants to increase customer satisfaction. Sales wants to squeeze out more revenue. When there’s an initiative that increases revenue but makes customers unhappy, how do you expect your heads of Client Services and Sales to work it out? Yes, you can ask them to look past their incentives and do what’s best for the company as a whole, and, if you hired and managed well, they will.

Conflict escalation is of course part of being the boss. You can solve this problem by overruling one of them. But you created a conflict that will make agreement harder and less frequent. You can make the decision, but you’re not getting agreement and buy-in. One of those people or groups is going to resent you for taking something away from them, and they should. This is entirely a consequence of your choices. You gave them goals and then contradicted yourself. Over the long run, incentivizing conflict will sour relationships and cause organizational dysfunction.

What you need to do instead is to set up responsibilities to internalize the differing goals. You want to push the trade-offs into one person’s head, ideally as low in the organization as you can. Where you need to split responsibilities, give similar goals to parties that need to cooperate. Then it’s not one person’s loss versus the other person’s gain. It’s one person having to deal with a gain and a loss and the other person having to deal with a similar gain and a similar loss. They will each have to make more complicated decisions, but you’ll incentivize cooperation not conflict.

It’s your job to disagree

I hate disagreement. I hate arguing. Every time I say this, my coworkers don’t believe me. That’s because I disagree a lot. But it’s not because I like it. It’s because I have to.

Imagine a diagnostic panel for a nuclear power plant. Tons and tons of lights for radiation sensors, temperature, water flow, and so forth. All of them are green. Except one that’s blinking red. How do you react? Do you ignore it? Do you chastise it for not agreeing with the others? Do you bang on it to try to make it green? Obviously not. That’s because that red light is telling you something none of the other lights is telling you. In fact, its whole purpose is to tell you something the other lights don’t tell you. If it’s green when the other lights are green and only red when they’re red, it serves no purpose.

When you’re in a discussion about some topic, whether it’s about hiring a candidate, implementing a new strategy, or how to reorganize a team, you’re there to add information and perspective that would not otherwise be there. If those are already represented by other people, you should not be there. You’re wasting your time and possibly theirs. If you have an objection that you’re not voicing because you’re afraid of disagreeable, why are you there?

There is one case where you should keep quiet. That’s when you are in a company where disagreement is frowned upon. If you disagree, stay silent while you look for a new job, and then get out of there. It’s toxic and will limit your career success as long as you remain.

Force versus frequency

My job is getting people to do things. The way I want it to work is that I tell people one time what needs to get done, emphasize its importance, and then it gets done. That’d be nice, right?

As far as I can tell, if you tell people something once, it’s as though you never said it. It doesn’t matter how loudly you say it or how important you say it. Saying it once doesn’t stick.

What does work? Saying it over and over again. Not even loudly. Quietly works too.

This is pretty rational. We all have moods and reactions to things. When I tell someone something once, who’s to say I’m not just in an unusual mood? Who’s to say the thing that I think is so important right now is going to be important to me in a week or a month? There’s a good chance it won’t be. But if I say calmly that something is important today, tomorrow, next week, the week after, the month after… You know I mean it. You know I’m serious, and you’ll be serious too.

Achieving mastery

When we see someone who is amazing at something, we assume they have a deep passion for that. That’s probably true, or at least it probably was true, but either way it’s not the exception. Passion won’t get you mastery. A desire for mastery is the only thing that will get it.

Passion matters. Passion will get you started. But passion becomes hard to sustain. Achieving mastery doesn’t come from doing something once, or twice, or even ten times. It comes from doing it thousands or even millions of times. There’s the debatable claim of 10,000 hours, but whether that is the exact number is beside the point. There are many hobbyists who become pretty good and are happy with their level of skill. That’s wonderful, but it’s not mastery.

To achieve mastery, you have to repeat what you’re doing far, far past the point of novelty. Achieving mastery is, well, boring. If you get bored and stop, that’s fine, but you won’t achieve mastery. “An amateur practices until they can get it right. A professional practices until they can’t get it wrong.” People say a lot of good things about deliberate practice, but nobody ever says it’s fun. It’s often the opposite. Passion doesn’t prevent the boredom, and it’s not passion that makes people keep trying through that boredom.

What motivates people who achieve mastery to push through the boredom and tedium and frustration is the goal of mastery itself. I sometimes think that the desire for mastery of a thing at some point can exceed the passion for that thing. Perhaps not just “can” but “must.” It’s not about joy or passion because those become harder and harder to find. It’s not about extrinsic rewards, because those are elusive and for most skills not that great. Maybe being competitive will get you there, but arguably the desire to beat everyone else is a version of mastery. Eventually all motivations get burned away except the desire for mastery itself. To achieve mastery, your goal must be mastery. It won’t come any other way.

People don’t understand what they don’t understand

I’ve had many discussions with other managers about the performance of people on their team. Everyone is imperfect, and one of the jobs of managers is to identify weaknesses in people’s performance and give guidance to them to address them. In these discussions, something that I’ve seen over and over is managers coming up with a message the recipient is almost certain to misunderstand.

Suppose you have a software engineer who is shipping buggy code. This seems pretty clearly to be a result of them not being careful, so their manager says, “be careful.” Stop for a moment. When the software engineer is writing the buggy code, are they thinking “I’m being careless?” Unless they’re a jerk, they’re not. They think they’re being careful already. They don’t realize that you mean something else by that word. They don’t understand that what they think of as “careful” is not what you think of as careful. That’s excusable, but for you as the manager to give that advice shows that you don’t understand what the problem is. It’s not about the intent. It’s about how the intent translates into action and judgment.

There are infinitely many examples of this advice anti-pattern. In any kind of technical discipline, you’ll hear people say, “use the right tool for the job.” Suppose you have someone who uses the wrong tool. Telling them to use the right tool helps nothing because they believed they were using the right tool. You have to show them why what they thought was the right tool was in fact the wrong one.

Another popular topic is development process. If I had a dollar for every time I heard someone say they wanted “just enough process,” I’d have like a hundred dollars. Who ever says they want too much process? Who ever says they want too little? Everyone who is doing something too much or too little probably thinks they’re doing it just right. If they didn’t, they wouldn’t be doing it that way!

The root of this problem is ironically closely linked to the problem the manager is trying to fix: the manager’s guidance can only be properly applied if the team member already understands the thing they obviously don’t understand. You can’t tell them to do what they already think they’re doing and expect a different outcome. You have to give the guidance in a way that highlights how what they think they’re doing is not what they’re actually doing. Otherwise you’ll live the cliche: “the greatest problem in communication is the illusion it has taken place.”

Nuance doesn’t scale

I have an organization of a few hundred people reporting to me. These people are intelligent, knowledgeable, and sophisticated. They’re motivated to do their jobs well and accomplish our mission. We’re trying to do difficult and complicated things, but I have to explain everything like they’re five because nuance does not scale.

Think about how you handle problems with a small team. You talk a lot. You listen a lot. You propose ideas, pick holes in them, patch them up, shoot them down. It takes a lot of time. Now embed that team into a larger organization supporting a multi-billion dollar business. This team has to make sure that what they build fits into the strategy. They have to collaborate with a half a dozen other teams.

Building a software product is hard. It’s not one decisions but hundreds of thousands of decisions, maybe even millions. Building dozens of software products is proportionally harder, especially with your teams distributed around the world. Even if I had the knowledge to make all of those decisions well, I certainly don’t have the time or bandwidth. Instead, the decisions are distributed all throughout the team. Some of the decisions are things that can be computed from available information to derive the “right” answer. Where that isn’t possible, or it isn’t possible with the information available to the individuals on the teams, is where I need to fill a void.

Somehow I have to fill that void with useful guidance and information at scale. That means I have to keep the messages simple. I can’t meet with everyone on each team to explain what I meant when I chose my words clumsily. I can’t explain how abstractly-expressed guidelines apply to each individual situation. I can’t describe a complicated method for identifying situations where the guidance applies and where it doesn’t. I can’t leave out important decisions as exercises for the reader, assuming everyone will know exactly how to fill in the blank. I have to give guidance that is unambiguous, readily applied, complete, and with clear guidelines of when to apply it. If not, the best case scenario is that I’m inundated with questions. Worse is the possibility they think they understand, but what they understood and what I meant are different.

It would be nice if this was just about how the guidance was phrased, but it often means that the underlying ideas need to be simplified as well. That means that our API first strategy applies to every externally-facing product, not just the ones where we expect to have a customer. It means our accessibility audit deadline is the same for every team, no matter how important their product or which countries they serve. It means that we focus the complex and sprawling idea of “application fidelity” on the idea of showing the job seeker the exact application the employer will see for them to confirm and submit. It means that I list exactly which time zones are part of Asia/Pacific and which are part of the Americas, and I deny that Europe exists.

These are not definitely the best decisions to make for each situation facing each team. But they are the best decisions to make when I’m deciding for every team at once. The bigger the organization, the simpler and clearer your ideas have to be. There may be better ideas, but you won’t be able to get them implemented because doing so means you have to anticipate or react to a blizzard of questions and clarifications that you either will handle imperfectly or burn enormous amounts of time trying to handle perfectly (and probably still fail).