How AI Screening Tools Actually Rank Candidates, and Where the Bias Creeps In
A look under the hood at match scores, embeddings, and knockout logic — and the specific points where a screening model quietly learns the wrong lesson.
Every screening vendor shows you a number. Eightfold gives you a match percentage. SeekOut ranks a talent pool. Workday's recruiting stack (with HiredScore folded in) surfaces a shortlist with a tidy label like "highly likely to advance." The number feels objective because it came from a machine. Understanding how that number is built is the difference between using the tool well and being quietly steered by it.
Here is what actually happens between a resume landing in your ATS and a candidate showing up near the top of your list.
Step one: turning text into math
A resume is unstructured text. A model cannot compare "led a cross-functional pod" to "managed a matrixed team" as strings — they share almost no words. So the first thing most modern tools do is convert the resume, and your job description, into embeddings: long lists of numbers that represent meaning. Phrases that mean similar things land close together in that numerical space, even when the vocabulary differs.
This is the genuinely useful part. Semantic matching is why a good tool can tell that a "revenue operations analyst" and a "sales operations specialist" are cousins, and why keyword-only ATS filters from a decade ago were so brittle. The candidate who didn't happen to paste your exact phrase no longer gets silently dropped.
It is also the first place bias enters. Embeddings are trained on enormous amounts of human-written text, and they absorb the associations in that text. If the training data consistently pairs certain names, schools, or phrasings with certain roles, those associations carry through. The model isn't malicious. It is a mirror, and it reflects patterns you may not want to hire on.
Step two: scoring against a target
Once everything is embedded, the tool needs something to score against. There are two common approaches, and they fail in different ways.
Job-description matching. The simplest version compares each candidate to the text of the req. Cleaner in one respect — the target is a document you wrote, not a person. But the score is only as good as the JD. A job description stuffed with "rockstar," "digital native," or ten years of a tool that has existed for six will pull the ranking toward whoever mirrors that language, not whoever can do the work.
Success-profile matching. The more powerful version learns from your own historical hires or high performers. You point it at the people who worked out, and it finds more like them. This is where trouble concentrates. If your past successful engineers came disproportionately from three schools and one gender, "more like them" can mean exactly that. The model has no concept of why those people succeeded. It only has the pattern, and the pattern includes everything — including the parts that were an accident of who you happened to hire in 2019.
This is the mechanism behind the cautionary tales you've heard. A model that penalizes a resume for containing the word "women's" (as in "women's chess club captain") did not decide to discriminate. It observed that resumes without that token were historically selected more often, and it optimized accordingly.
Step three: knockouts and hard filters
Layered on top of the score are the rules you or the vendor configured: minimum years of experience, required credentials, location radius, work authorization, sometimes a required assessment score. These run before or alongside the ranking, and they are blunt.
Knockouts are where a lot of quiet, legally risky exclusion happens, because they feel neutral. "Must have a bachelor's degree" screens out capable people and correlates with demographics in ways a court cares about. "Must have no employment gaps" penalizes caregivers and people who were ill. "Within 25 miles" can map onto neighborhoods in ways that track race. The tool will happily enforce whatever line you draw, and it will do it silently to thousands of applicants at once. Scale turns a mediocre policy into a systemic one.
Step four: the ranked list you actually see
Finally the tool sorts. And here is a subtle point that trips up a lot of teams: the score and the rank are not the same thing. Two candidates at 88% and 84% may be functionally identical, but one sits above the fold and gets a call and the other is on page two and never does. Recruiters trust the order. The order compresses a fuzzy, uncertain model output into a hard sequence, and the uncertainty disappears from view.
Where bias creeps in — the short version
- In the embeddings, from the language the base model was trained on.
- In the target, when you train on past hires and inherit past patterns.
- In the job description, when biased or lazy language becomes the thing candidates are matched to.
- In proxies — the model can't see race or gender, but it can see ZIP code, school, resume gaps, name, and the sports and clubs people list, all of which correlate.
- In the knockouts, where blunt requirements exclude protected groups at scale.
- In the interface, where a ranked list makes small, noisy differences look decisive.
What to actually do about it
You will not audit the embedding math yourself, and you don't need to. Here is what is in your control.
- Interrogate the target. Ask your vendor, in writing, what the score is computed against. If it's a success profile trained on your historical hires, ask what checks exist for the demographics baked into that history. "We can't disclose that" is an answer, and it's a bad one.
- Clean the job description before you clean anything else. Tools like Textio, or even a careful prompt to a general model asking it to flag exclusionary and coded language, will catch the "rockstar/digital native" problem cheaply. The JD is the input you most fully control.
- Look at your knockouts as a policy, not a setting. Every hard filter is a decision about who never gets seen. Degree requirements especially deserve a fresh look — many roles that list one don't need one.
- Use the score as a sorting aid, not a verdict. Set a threshold and review everyone above it, rather than trusting the exact rank order. Deliberately pull a sample from just below your cutoff and read those resumes yourself. If you keep finding strong people there, your threshold or your model is miscalibrated.
- Watch the adverse-impact numbers over time. Track pass-through rates by group at each stage. A screening tool that ranks well but advances one group at half the rate of another is a problem you can measure before a regulator does.
The honest trade-off: these tools genuinely surface people your old keyword filters buried, and they save real hours. That upside is why they're worth using carefully rather than banning. But a match score is a hypothesis about a person, generated from patterns in old data. Treat it like one.
This article is general education for HR practitioners, not legal advice. Employment discrimination law is fact-specific and varies by jurisdiction — loop in qualified counsel before setting screening policy.
Put this into practice
Work out what an AI model actually costs per month from your token usage, and compare the major models side by side.
Open the AI API Cost Calculator →A note on shelf life. AI products change fast. This guide deliberately focuses on the parts that stay true — how to judge a tool, what the trade-offs are — rather than ranking products that will have changed by the time you read it. Prices and feature claims should always be checked against the provider before you rely on them.