The one thing that will hold out forever or will at least be the last thing to fall is critical thinking. So I think you simply interview for that.
At my small company we interview all candidates, regardless of experience level, using the same set of questions during the primary round. They’re a combination of open ended scenario type questions and pure debate style questions. We try to design them such that when you’re answering them, it’s similar to when you’re on the job collaborating on a technical solution, or triaging an incident.
We provide limited information at first to force the candidate to immediately ask for more info, to see what kind of questions they ask and what kind of information they care about. When they start to provide the solution, we like to throw in alterations or wrenches to see how they pivot. It’s very much “turn based”.
If you’re taking it seriously, it’s very obvious when candidates are either solving it for the first time, are recalling true experience, or are simply reciting information like they studied it. That’s why many of our questions work for many levels of engineering.
Additionally, because there is so much back and forth, you get to also “vibe-check” the candidate engineering wise. In our experience most candidates have enjoyed the process, so we feel like we uncover a lot about personalities here as well.
Unfortunately, the heel to this approach is that you pretty much have to conduct these in person. Because an LLM hypothetically would do well in this style of interview. We’re hybrid one day a week so it works for us, but it doesn’t exactly scale.
It'll be interesting to see if we get better / worse at figuring out what effective LLM interaction should look like, since that seems prerequisite to this kind of evaluation. Right now there's so little data on whether e.g. casual conversational interaction works better than structured prompting, and so, so, so much superstition - I don't know if I'd really trust judges to have good instincts about what's effective rather than what resembles their own use patterns.
I would like to preemptively acknowledge that this is not the center of the article, but:
> Did you know IQ tests are effectively illegal for hiring in the US because they disproportionately screen out minorities? I learn things when I research these posts.
The provided link discusses how this is itself a common misunderstanding and they're not illegal if they have to do with the job, but you have to justify that they do have to do with the job.
> Using fully automated scoring would also be against the law in some places: New York City requires annual bias audits of automated hiring tools, and you don’t want that heat.
The elision of the gap between "[it's illegal to do this]" and "[it's illegal to do this without checking for bias]" is interesting.
Thanks for posting this clarification. It's actually even simpler than that: for white collar employment there's going to be a presumption that IQ is relevant to the job, and there's no procedural hurdle you have to clear to IQ test applicants. The sole concern is, presumably, whether a candidate who is rejected can mount a successful disparate impact claim.
Several household name corporations routinely apply general cognitive tests to candidates. The companies who deliver these tests have logo crawls just like any other SAAS company. If this was a lucrative field for plaintiff's lawyers, you'd expect that not to be the case.
IQ tests aren't more widely used because they're not very useful for qualifying most candidates for most jobs. You generally want to know if a candidate can do the job, not how well they do on IQ tests.
Why hire juniors? There seem to be plenty of unemployed mid- and senior-level people and the market is so poor that some of them (e.g. me) will work for cheap.
Juniors become your next Seniors over time. You need both, but filling roles with Senior engineers on the cheap isn’t good for longevity.
Anecdotally, though I’m sure many hiring managers would agree, junior engineers tend to be more coachable, raise fewer demands, and create less conflict.
The most dysfunctional teams I’ve worked with were heavily Senior. Though, the most functional teams I’ve worked with were 2-3 Senior devs. Maybe the future is primarily teams of a couple of midd/seniors, a single manager over 5 of these, and a lot of tokens. Paid what we pay juniors today, of course, as we race to the bottom.
Here is the problem with almost all of these attempts: there is no need to be a “pop quiz” where you have the answer, and the candidate is made to guess what you feel is right (even if you're wrong) and is walking in blind. This literally never happens in the real world - and if it does it means you’re getting fired. It’s just sadism you are pushing to make the in group people feel smarty pants and superior.
Tell the candidate up front what you want to talk about. Do it on a subject you dont know either so you are on even footing. Let them do research before the “quiz”. Since tokens are pretty cheap, ask them to build a working application and then discuss it with them.
This really isn't hard if you get off your imaginary high horse, and show candidates some basic respect.
The one thing that will hold out forever or will at least be the last thing to fall is critical thinking. So I think you simply interview for that.
At my small company we interview all candidates, regardless of experience level, using the same set of questions during the primary round. They’re a combination of open ended scenario type questions and pure debate style questions. We try to design them such that when you’re answering them, it’s similar to when you’re on the job collaborating on a technical solution, or triaging an incident.
We provide limited information at first to force the candidate to immediately ask for more info, to see what kind of questions they ask and what kind of information they care about. When they start to provide the solution, we like to throw in alterations or wrenches to see how they pivot. It’s very much “turn based”.
If you’re taking it seriously, it’s very obvious when candidates are either solving it for the first time, are recalling true experience, or are simply reciting information like they studied it. That’s why many of our questions work for many levels of engineering.
Additionally, because there is so much back and forth, you get to also “vibe-check” the candidate engineering wise. In our experience most candidates have enjoyed the process, so we feel like we uncover a lot about personalities here as well.
Unfortunately, the heel to this approach is that you pretty much have to conduct these in person. Because an LLM hypothetically would do well in this style of interview. We’re hybrid one day a week so it works for us, but it doesn’t exactly scale.
It'll be interesting to see if we get better / worse at figuring out what effective LLM interaction should look like, since that seems prerequisite to this kind of evaluation. Right now there's so little data on whether e.g. casual conversational interaction works better than structured prompting, and so, so, so much superstition - I don't know if I'd really trust judges to have good instincts about what's effective rather than what resembles their own use patterns.
I would like to preemptively acknowledge that this is not the center of the article, but:
> Did you know IQ tests are effectively illegal for hiring in the US because they disproportionately screen out minorities? I learn things when I research these posts.
The provided link discusses how this is itself a common misunderstanding and they're not illegal if they have to do with the job, but you have to justify that they do have to do with the job.
> Using fully automated scoring would also be against the law in some places: New York City requires annual bias audits of automated hiring tools, and you don’t want that heat.
The elision of the gap between "[it's illegal to do this]" and "[it's illegal to do this without checking for bias]" is interesting.
Thanks for posting this clarification. It's actually even simpler than that: for white collar employment there's going to be a presumption that IQ is relevant to the job, and there's no procedural hurdle you have to clear to IQ test applicants. The sole concern is, presumably, whether a candidate who is rejected can mount a successful disparate impact claim.
Several household name corporations routinely apply general cognitive tests to candidates. The companies who deliver these tests have logo crawls just like any other SAAS company. If this was a lucrative field for plaintiff's lawyers, you'd expect that not to be the case.
IQ tests aren't more widely used because they're not very useful for qualifying most candidates for most jobs. You generally want to know if a candidate can do the job, not how well they do on IQ tests.
Why hire juniors? There seem to be plenty of unemployed mid- and senior-level people and the market is so poor that some of them (e.g. me) will work for cheap.
Juniors become your next Seniors over time. You need both, but filling roles with Senior engineers on the cheap isn’t good for longevity.
Anecdotally, though I’m sure many hiring managers would agree, junior engineers tend to be more coachable, raise fewer demands, and create less conflict.
The most dysfunctional teams I’ve worked with were heavily Senior. Though, the most functional teams I’ve worked with were 2-3 Senior devs. Maybe the future is primarily teams of a couple of midd/seniors, a single manager over 5 of these, and a lot of tokens. Paid what we pay juniors today, of course, as we race to the bottom.
Here is the problem with almost all of these attempts: there is no need to be a “pop quiz” where you have the answer, and the candidate is made to guess what you feel is right (even if you're wrong) and is walking in blind. This literally never happens in the real world - and if it does it means you’re getting fired. It’s just sadism you are pushing to make the in group people feel smarty pants and superior.
Tell the candidate up front what you want to talk about. Do it on a subject you dont know either so you are on even footing. Let them do research before the “quiz”. Since tokens are pretty cheap, ask them to build a working application and then discuss it with them.
This really isn't hard if you get off your imaginary high horse, and show candidates some basic respect.