An Engineering Philosophy

13 minute read

It’s been far too long since I’ve blogged. In this post, I want to reflect on some things I’ve learned as a senior engineer and try to synthesize them into a personal engineering philosophy.

table of contents

impact over productivity

If you have to optimize for something as an engineer, it should be impact. Impact can be a squishy thing to define and - to make matters wo- it can look vastly different at every company. So how do you optimize for it?

A main or even primary part of your job as an engineer, I think, is to help accomplish the broader goals of the organization you’re part of — not to “engineer”. Engineering might look differently at different places, but ultimately the engineering itself is a means to an end instead of an end in of itself. Some thoughts on this to drive this home:

  • choose what is best for the org over what is most personally interesting
  • sometimes, the most interesting thing is also best for the org, but often not
  • technical choices and efforts can have outsized impacts that trickle down to the entire org
  • impact can be measured in monetary (driving KPIs, bottom-line revenue, cost reduction), organizational (retention, happiness, etc.), and other means (overall reputation of a company). Figure out what metrics, concrete or not, your company cares about and prioritize driving them over the sheer number of tasks completed.

“It depends”

I’ve noticed that the more time I’ve spent in software engineering, the more I find myself saying “it depends”. Earlier on in my career, I was much more likely to think that there was a “one and proper” way to do a given thing. And, to be fair, there are a non-zero number of areas where you can pretty safely land on an approach in nearly all situations. But even then, it’s more a “do this nearly all the time” thing and less a “do this all the time” thing.

The reality is that nearly all situations where software engineering is applied involve a myriad of competing factors and there’s almost never an identical approach to the same problem in different circumstances. It’s best to realize that most questions and scenarios in engineering are very much “it depends” scenarios. It’s best to stat with this as a baseline and work from there instead of blocking yourself in by assuming a single solution to a problem.

Cleverness kills

“Programs must be written for people to read, and only incidentally for machines to execute.”

― Harold Abelson, Structure and Interpretation of Computer Programs

This quote has always stuck with me. It’s brief, memorable, and focuses on the importance of people in computing. People, after all, are where all the code comes from (until the robots get smarter, anyways). There’s a temptation to write code that is intricate, optimized for computers, finds an uncommon way to express a common concept…“clever”, we might say. In almost every case where code like this is written, it creates a negative directional tradeoff away from code changeability and flexibility and towards brittleness and difficulty. Whether it’s another team or just your future self, clever code will be difficult to work with and is more likely to produce bugs or just plain frustration. Write for future you and other teammates so they can easily delete or change your code, not so you feel clever in the moment.

“Business knows best”

Have your ever seen “The Big Bang Theory”? It’s a show about a handful of scientists at CalTech in Pasadena (my hood!) who live together and the adventures they have. One of the running jokes of the show is the lack of respect between applied and theoretical physicists and between engineers. The gist of the gag is that the theoreticians don’t think the applied scientists are as rigorous in what they do but they both agree that engineers are doing, by comparison, scientific grunt work.

I think we - software engineers - tend think or at least find it very easy to feel that the engineering side of the house is the domain of the pure, theoretically signifcant work and that what the rest of the business does is uninteresting grunt work. I’ve seen this happen and while it’s funny to watch on “The Big Bang Theory”, it can be incredibly harmful in practice. For the majority of engineers who aren’t working in a research or academic capacity, our work is applied and oriented around the problems our business faces. Furtermore, we’re just one part of the overall organization working towards a set of common goals. Our goal is to help the business solve problems and in turn solve the problems of our users. And, hopefully, remain profitable along the way.

Sounds simple enough, but often engineers can give in to the temptation to “engineer for engineering’s sake”. This might look like picking a project because it’s more technically interesting than impactful to the business or creating a system that’s more complicated (but more interesting!) than what the business problem calls for. If this happens enough, you end up with a beautiful system that doesn’t meet the needs of the business and can ultimately fail to produce revenue. So make sure to ask yourself “is this {thing, solution, project, etc.} what’s best for the business?” whenever you’re writing code to solve a problem.

Ignore best practices (sometimes)

“Best practices” are durable ideas, patterns, or similar that have developed overtime and become well-known within a field. You probably know some more general best practices like “don’t repeat yourself”, “refactor your code”, “keep it simple” or others that are more technology-specific — “Use capitals for component names” in React or “handle errors as values in Go”. It can be very easy to adopt, even just subconsciously, a best practice on face-value alone. It’s a best practice after all - who doesn’t want to do and be best?

The problem with this practice is that it’s not, err, best. “Best practices” should really be called “often-good things you might consider doing”, but that’s not as easy or fun to say. The problem with best-practive-driven engineering is that it is both context-independent (the best practice might make absolutely no sense for your business or project) and often lost in translation (reduced to a pithy phrase like “DRY”). This can often show up in engineering teams where there is a lack of more senior engineering leadership and the team is searching for standards or goals to align around.

The good news is that you can get out of this cycle by adding context and digging into the practice itself. For example, advocating for engineering time to adopt best practice XYZ, ask yourself and your teammates whether the actions will really make sense for you. Does it solve a business need? Does it have short-term costs but long-term gains that are concrete? You can ask these and other questions to force yourself to start to see the tradeoffs (!!!) of a practice and evaluate it beyond it just being “best”.

The other thing you can do is dig more into the practice itself to check your understanding. I’ve seen many folks, for example, hold up the DRY (don’t repeat yourself) principle as something we should all practice. It might be used to justify changes in code review, architecture, and other technical areas. The issue, though, is that DRY doesn’t really mean “never repeat yourself”. It’s been widely misunderstood as such, to the extent that the authors of the The Pragmatic Programmer felt the need to clarify in the 20-year edition of their fantastic book:

Let’s get something out of the way up-front. In the first edition of this book we did a poor job of explaining just what we meant by Don’t Repeat Yourself. Many people took it to refer to code only: they thought that DRY means “don’t copy-and-paste lines of source.” That is part of DRY, but it’s a tiny and fairly trivial part. DRY is about the duplication of knowledge, of intent. It’s about expressing the same thing in two different places, possibly in two totally different ways. Here’s the acid test: when some single facet of the code has to change, do you find yourself making that change in multiple places, and in multiple different formats? Do you have to change code and documentation, or a database schema and a structure that holds it, or…? If so, your code isn’t DRY.

So, look into what best practices actually entail before adopting and wielding them.

Code is the easy part

I used to think that the more senior your role was, the more complicated and intricate the problems you worked on were. This can be true, of course, but one thing I’ve found and since come to believe is that code is the easy part of the software engineering job. The challenges of aligning, collaborating, and working with other people are often far more complicated and have significantly larger consequences than most any block of code will. Part of the reason for this is that coordinating and organizing dozens, hundreds, or even thousands of engineers to work well together to produce high-quality software is just plain difficult. Whereas you are often working with a deterministic world of electrons in software, people are infinitely more complicated (and interesting).

One of the takeaways from this is that senior engineering leadership roles are almost all “people roles” to some extent, even if they’re technically on the “individual contributor” track. It also follows that the most successful engineers will be skilled at working with people. In fact, I would go so far as to say that the best code, systems, and software, in general, will nearly always be the result of high-quality collaboration and interpersonal coordination rather than the “lone genius” style of working. Software supported by flexible and resilient teams will be just that - flexible and resilient.

“No Mysteries”

We’ve all dealt with a particularly nasty bug. If you haven’t, odds are you will. It can be frustrating and make you even question your abilities as an engineer. And more often than not, the bug is either some incredibly obscure edge case or a tiny but fixable issue that will make you close your laptop for the day in exasperation.

As frustrating as these are, they’re not nearly as bad as another type of bug: 🔮 the mystery 🔮. This is the sort of behavior that can occur infrequently and often gets shrugged off as transient or a one-off occurrence. This type of issue is so pernicious because it may indicate a significantly larger issue beneath the surface that won’t come to light until later and may have much more drastic consequences. There’s also the possibility that they are less severe but incredibly persistent and show up just often enough to be a silent drag on team resources or user experience.

I’ve found it helpful to adopt a “no mysteries” approach. This involves agreeing to, either individually or collectively as a team, ensure that flaky, not-understood, or inconsistent behavior is tracked down and demystified as much as possible. There have to be limits, of course - you can spend an inordinate amount of time chasing vagueries that might be linked to network weather or other “normal” inconsistencies. But the general idea is to make sure that you or you and your team understand the behavior of a system as much as possible. When done properly, this can reduce bugs, increase engagement, and lead to better system evolution. And, most importantly, save you the awkward “huh…never seen that before” when someone brings up strange behavior.

Abstract & optimize later

This could almost fall under cleverness kills, but it’s so important that I think it deserves its own section. Engineers are trained, generally, to look for and create ways to facilitate efficiency in code. This might be packaging up code into a sharable library, moving related areas of code into the same repo or folder, or refactoring the architecture of a project to be more amenable to re-use and sharing. This is usually good and well, but it can accidentally make abstraction and optimization goods in of themselves.

Abstracting or optimizing too early might look like seeing a handful of shared characteristics between two models and thinking “I should abstract this!” or possibly trying to come up with a data model that meets the needs of several distinct data entities that won’t be easy to change together. Whatever it is, this early abstraction or optimization has the potential to completely stifle out and kill the development or success of a project. That’s because you have likely made a prediction about the future (“this will solve all future use cases!”) that is almost certain not to be exactly right and placed the weight of that prediction on your code.

Because predicting the future is impossible and because rewriting code you just wrote isn’t fun, it’s better to instead optimize for changeability over efficiency. If you see a pattern start to emerge — say, something happens 5 times — you can allow yourself to ask the question “should we abstract this into something?” That doesn’t mean you should, just that you get to ask the question. You should let actual data points (“this seems like repeated logic”, “this and that both seem to always need to change together”, etc.) and not your predictions about the future drive your design decisions, your code will be much more robust and resilient. It’ll be easier to work with and thus easier to bend to your needs. You’ll also find it won’t need to be completely rewritten when business needs change, as they inevitably will.

You Aren’t Going To Need It (Probably)

You’ve probably heard or read “YAGNI” somewhere. It stands for “you aren’t going to need it” and it can mean a wide variety of things, depending on the person. I’ve generally found it helpful as a reminder that a simpler, less clever approach or architecture will often serve you better over the long run. Worried about changing databases one day? That’ll be such a massive undertaking at any sizable company that the current thing you’re fighting for won’t make a difference — ”you aren’t going to need it.” Did you hear from a product manager that “we might need to add this other feature here in 6-9 months”? You aren’t going to need it. Thinking about trying to squeeze out some tiny performance wins from a function you’re writing? You (probably) aren’t going to need it.

Most of the time, it’s better to focus on easily-changeable code and architecture rather than try to make sure you solve a future problem in the here and now.

I would say this rule applies maybe 95%+ of the time. There are cases, however, where it might not. If you know that you’re going to, for example, run out of storage space on your primary database in two quarters, you should do something to fix it. Hopefully, there’s an engineer at your company whose role (maybe “Staff Software Engineer”, “Principal Engineer”, etc.) who is empowered to think ahead and avoid such a problem. In most cases though, it’s better to ask yourself if you’re really saving any time by trying or effort to make a very tenuous guess about the future.

(this post will be updated over time)