This article was written by Ken Leaver who comes from a product & commercial background. He has founded multiple companies and held senior product positions at SEA tech companies like Lazada and Pomelo Fashion.
Ken runs his own agency that helps early stage companies execute faster and cheaper. Check out his linkedin at: https://www.linkedin.com/in/kenleaver/
Guest Author: Ken Leaver
My “Rule of three” states simply that:
- regardless of how much interviewing you do….
- …if you test three people for the same role for at least a few weeks,
- then on average the end result will be 30% better than if you had just hired someone based on interviewing alone.
Let’s take an example…. recently I was hiring for a BI/analytics person and I started by talking to a handful of people on Upwork and then choosing who i considered the two best.
I gave them 2-3 charts to make and they needed to pull the data (via Make.com), clean the data (in Google Sheets) and then present the data (in Looker).
The first guy was pretty good, but had a tendency to do the minimum required. It was clear he was not going to think about how to overdeliver and make something great. In part because he probably thought he should paid more (note I paid everyone the same amount).
The second guy would work in spurts. He’d do a bunch of stuff and then we’d barely hear from him for a few days. So he was a bit unreliable and hard to communicate with.
So I hired two more folks from Upwork and started testing them as well. They both seemed quite easy to communicate with and faster to get stuff done.
One of them, however, clearly started to emerge as having better technical skills and delivering higher quality stuff. And so I will let the other three wind up what they were doing this coming week and just keep the one.
I would estimate that this person I am selecting is significantly better than either of the first two people I tested.
Meaning I expect if I hired him full-time vs. one of the first two folks.. he’d probably get 30 – 100% more done.
Also, this is not some outlier… this is what I typically see when running these tests.
And running these types of tests on Upwork the past ~4 years has taught me some valuable lessons about recruiting that the prior 20 years of hiring hundreds of people had not taught me.
The reality is some people are just much better than others
In engineering the saying for a long time was that one great engineer was more valuable than 10 average ones. And this was generally accepted by most as being fact.
While this power law might be strongest in areas like software development, I very much find that they hold true in all other types of positions. But perhaps to a lesser degree.
Think about it… don’t the top founders & CEO’s say that they’re always looking to hire the absolute best people? And so they invest tons of resources into this endeavor.
Why do they do that? Because there is a power law in effect. And the reality is that someone that is very good often achieves several times more than someone who is not good.
I’ve seen this consistently to be the case my entire ~20 year career… ever since I was a strategy consultant back in 1999.
Another reality is that testing someone is far more accurate than interviewing them
This is the lesson I learned these past few years. And I learned it because I’ve hired 200+ contractors on platforms like Upwork during this time.
At the beginning I’d interview a bunch of people and hire one. And i’d invest weeks or months into them. But sometimes i wouldn’t get the level of quality output that I was seeking.
So I’d stop working with them and hire someone else on Upwork.
Till one day i said to myself… “Hey wait a minute Ken… why lose months in the wrong person? Why not just hire three at the same time and keep the best one?”
This way I can compare and contrast the three to see who is clearly better. Natural selection at work!
So this is exactly what i have done. For over three years now.
And what I have learned from it is simply mindblowing… in my view they should literally throw away all the books that talk about interviewing in a massive bonfire. As they are just completely wrong in my view.
I ALWAYS end up with a better outcome by using this system of testing.
On average I find that you’re at least about 30% better off
To understand how I arrived at this number of ‘30%’ we need to go a bit deeper into how this plays out when you repeat the experiment many times. And note that i’ve probably run this experiment on the order of 50-70x in close to 4 years.
And i’ve run it for all types of positions…. from tech roles to admin roles to content to marketing stuff and everything in-between.
So how does it work?
I still always start off with some interviews. I’ll generally start off by interviewing 5-8 Upwork people with the goal of finding 3 that I want to test.
And I keep interviewing till I find these 3.
I keep these initial interviews pretty short (~10 minutes) and look at it almost as if i’m just ‘qualifying a lead’. I want to see that the basics are there to be a decent candidate.
- Do they speak english well enough?
- Do they have some relevant experience?
- Does their attitude seem good?
That’s it. I don’t go that much deeper.. because i’ve been duped many times by people that express themselves very well but are crap at execution.
“Homey don’t play dat no mo!” hahaha
Then I hire all three on upwork for a limited scope that typically takes from 1 – 3 weeks and doesn’t cost that much. Sometimes I kick them off at the same time, sometimes I don’t.
Plus I always try to keep a mental note of who i would have hired based on the interview alone.
As I do this work I begin to learn how well they think, communicate, adopt my system, etc. I start to get a pretty solid view on how they actually operate.
A winner usually starts to emerge pretty quickly.
When I compare this winner against who i ‘would have hired’ from the interview… sometimes it is the same person. I’d say that this is less than half the time.
A lot of times it is someone different.. and they end up being MUCH better. Sometimes even 2-3x better
Repeat this same exercise many times… and you realize that on average you are at least 30% better off. It’s just math. For example:
Round 1: Winner = person i would have chosen from interview
Round 2: Winner = different person who is 60% better than person I would have chosen in interview
Round 3: Winner = different person who is 30% better than person I would have chosen in interview
Sum of three rounds = I hired 3 different people that ended up a total of 90% better than if i’d hired the person solely from the interview.
90% / 3 rounds = an average of 30% better.
Voila! And this is conservative…
This is not an idea, in my view this is a law
I have repeated this enough times over a long enough period that I am now confident enough to say that in the longer run it will ALWAYS be true.
Meaning that if you were to repeat this experiment enough times in whatever your domain is… you will begin to get the exact same results.
Kind of like if you play roulette long enough you will get closer and closer to the mathematical probability of each possible outcome.
And therefore… this evolves from being “Ken’s opinion” to Law baby!
When I googled ‘what is a scientific law?’, this is what I got:
Scientific laws are typically conclusions based on repeated scientific experiments and observations over many years.
So basically I encourage others to try this same experiment and let me know if you find different from what I am. I don’t think you will to be honest.
Then later on after there is sufficient data… I think we can all safely assume this is law. And we can all stop wasting time on perfecting our interviewing skills. Because it will go the way of the VCR. Sayonaraaaaaaa!
And thus there is absolutely no sense in interviewing UNLESS you absolutely cannot test at least three people for the role.
Which happens a lot less often than you would think… if you are creative. .
How do you test people for full-time roles?
Some folks have told me that they don’t think this works for full-time roles because they cannot test them.
And my answer is always the same… “Do paid work trials”.
Karri Saarinen, the founder of Linear, has been doing this ever since founding the company. And loves to tell all his naysayers that paid work trials absolutely works for full-time hires pretty much all the time.
Linear literally uses them when hiring every position I heard.
How does it work? Basically the person is paid to work part-time for the company with the exact team he would be hiring into.
And he is given some limited scope project, which he implements over the course of some weeks or months.
But what about if the person has a full-time job? Fine. They can work at nights and weekends.
What if the person doesn’t wanna do it? Too bad. They don’t get hired then.
Parting words
The Rule of 3 takes a bit more time but it pays off massively over the longer-term. I consider it no-different than the ‘compounding’ effect in finance.
If you consistently hire people that are 30% better then over a number of years… your team is a shitload better.
And the results are a lot better.
If you got 30% better results over just four years… then the compounding would mean your result is nearly 3x better than if you’d hired people by interviewing alone.
And if that is not argument enough… then I don’t know what is.
