Internal experiments are a product problem, not an engineering problem
The Overlap #8
Welcome to The Overlap, a biweekly newsletter that explores the relationship between product & organization design.
We’ve been pursuing a ton of internal experiments at my company, Sanctuary Computer, which I’ve been damn stoked about. One thing we’ve learned is that internal experiments are more akin to a product problem than an engineering problem.
An internal experiment is a small, bite-sized change that aims to solve an organizational problem. Specifically, experiments address:
How we meet
How we make decisions
How we strategize
How we distribute authority
How we reduce dependencies
How we mentor and upskill
How we invest our time and money
How we share information
How we do the work
Some experiments we’re currently running are:
A career growth buddy system: everyone is matched with a buddy of similar discipline and skill level. You and your buddy assess each other’s development based on our internal Skills Tree (which we’ve shared publicly!). If you and your buddy agree to any changes in your leveling, you can propose this level change (and salary bump, as a result) to our Finance lead. This is meant to make career growth more bottom-up and individually-driven.
Twist instead of Slack: we’ve been using Twist instead of Slack for all internal communication. This has been going really well—we love that Twist pushes us to communicate asynchronously.
Having more developers lead technical strategy: to distribute more authority and provide more growth opportunities, more developers are leading technical strategy work with our clients. Our devs are stoked about the opportunity to practice a different skill. And this helps us do better work by involving more voices in product discovery.
We are co-designing the way we work
Through these experiments, we aren’t just trying to adapt our culture to an all-remote world. We’re co-designing a new version of our culture. Two very different approaches.
Adapting our culture to full remote feels obligatory and non-inspiring. Co-designing our ideal culture feels inspiring and abundant.
Props to my colleague Hugh Francis for seeing this pandemic as an opportunity to pursue the latter instead of just the former.
Not only are we open-minded to remote work, we’re seeing this as an opportunity to distribute authority, decentralize where we should decentralize, and evolve our way of working.
Experiments as a product problem rather than an engineering problem
As an org designer who’s done experiments work with both large and small companies, I’m always excited to discover new metaphors for talking about this work. One metaphor that Hugh and I recently discovered came from a former colleague and mentor of mine, Aaron Dignan. That metaphor is this: experiments are a product problem rather than an engineering problem.
When solving an engineering problem, we’re concerned with making sure the software works. We don’t want the software to break when pushed to a server. We want to have a solution for all the possible edge cases before deploying. We want the code to be reliable, feasible, and scalable—and we’re fixing bugs that get in the way of those three things.
If we treated an experiment like a pure engineering problem, we want to make sure the experiment works. We’d write endless if-else statements.
But what if we missed an if-else statement?
What if the experiment is not meant to work?
What if you read the Spotify Squad structure whitepaper, applied that to your company to try to decrease dependencies, and still found that teams rely on each other?
I’ve noticed that most leaders treat experiments as if it’s an engineering problem. The experiment not working? We’ll make it work! We just have to double down and work harder!
However, forcing an experiment to work when it’s not meant to loses valuable information that tells us why it’s not working in the first place. Spotify’s Squad structure doesn’t work for your company because your company isn’t Spotify. Great! Let the experiment fail, so that you can identify what exactly isn’t working and design a different experiment to solve that.
Smarter leaders are open to the experiment not working. They seek to understand why the Spotify squad model isn’t working. Why dependencies still exist. Why they still feel like they’re a bottleneck. And ultimately try a new experiment.
That is treating experiments like a product problem.
When solving a product problem, we’re concerned with whether our product is a viable and valuable idea. We want to identify people’s unmet/underserved needs and understand whether our idea meets those needs. We’re sharing out small ideas early and trying to learn quickly and inexpensively.
To be fair, the similarity between engineering problems and product problems is this: if it fails, we want to minimize the consequence. We want to know all the possible ways it can go wrong early and safely. That’s why engineers use staging environments, that’s why QA phases exist, and that’s why Continuous Integration / Continuous Delivery is a widespread philosophy. So the metaphor may not be entirely one to one.
However, experiments as bets instead of features to be implemented is a useful framing. It pushes me out of the mindset that every experiment must work. And into the mindset that every experiment is a guess.
Learn quickly, safety, and inexpensively
Rather than asking “how can we make sure this works?”
ask “what might we learn if this doesn’t work?”
That might mean scaling the effort to just one team rather than the entire org. Or deciding on the duration of the experiment to be 2 weeks instead of 2 months.
The specific design of your experiment will depend on the a) problem you’re trying to solve and b) your organization’s context. But thinking about continuous improvement as a product challenge before thinking of it as an engineering challenge is a useful guideline.
—
What I’m Reading
I tend to think that experiments must be small, short, have no dependencies, and have a low blast radius. It’s hard to learn quickly if the experiment is large, contains multiple dependencies, and requires approval from multiple stakeholders. But John argues that “large” experiments also have their upsides. Always going small doesn’t prompt you to think about how to scale your experiment, and scaling is important.
A crowdsourced list of what makes good retros good. Great post.
Online targeted harassment is a problem in tech and we need to change this. Kat Fukui, a senior product designer at Github, shares her personal experience of getting harassed by actual software developers.
Make sure you have a policy and detailed playbook, and definitely don't expect your marginalized employees to fix these problems for you. Don't wait until an incident arises—it's always an "edge case" until someone's personal safety is threatened.
⚠️ Content warning: racist, sexist, transphobic, hateful language, and online abuse.
A super intriguing vision of a new, more simple, and more participatory version of the web.
Social networks are universally more restrictive than web pages but also more fun in significant ways, chief amongst them being that more people can participate. What if the rest of the web has that simplicity and immediacy, but without the centralization? What if we could start over?
See you in two weeks,
–tim
Artwork by Hassan Rahim, commissioned for this article by Whitney Mallett on ssense.
Excellent frame
Hey Tim, hope you're well! FYI, original art is by Hassan Rahim, commissioned for an article by Whitney Mallett on ssense.com
https://www.ssense.com/en-us/editorial/culture/the-optimization-impulse