Every recruiting database in the world has the same quiet problem: the same person, living in it more than once.
Nobody does anything wrong to cause this. A candidate applies to your firm with a personal email one year and a law school address the next; same person, two records. A career services office registers a student for your OCI program before an email is even on file. A historic import from an older system arrives missing a surname or a graduation year. A candidate appears as "Bob" on one record and "Robert" on another, or under an entirely different name, as when someone uses a formal given name on one record and a chosen English name on another ("Tse Hsin" and "Jason").
This isn't a failure of any platform or any process. It's data gravity — a universal property of how candidate data accumulates, everywhere, through completely ordinary paths. But the cost is real: an incomplete view of how a candidate has engaged with your firm, hours spent reconciling records by hand, and, because application history ends up split across records, genuine ambiguity in recruit-sourcing fee attribution.
Today I want to talk about how we solved it, and why I'm confident calling the result the most capable duplicate detection on the market for legal recruiting software is accurate.
Why the obvious approach doesn't work
If you've never built one of these systems, the solution sounds simple: match on email. Maybe match on phone number as a backup.
Here's why that fails, and fails in both directions at once.
It fails to find real duplicates, because many of the duplicates that matter most share no email at all. Records created by third parties, historic imports, a candidate who used different addresses at different stages of their career are all possibilities. An email-lookup approach is blind to exactly the records that need it most.
And it fails by inventing duplicates, because recruiting data is full of values that look like identity but aren't. A shared phone number is often a school, office, or agency line. Placeholder emails get entered by third parties. A naive matcher sees two records with the same phone number and confidently declares them the same person, and now your recruiters are wading through false alarms until they stop trusting the tool entirely.
Resolving identity across messy records is not a string-comparison problem. It's a deeply studied, genuinely hard mathematical problem, and treating it as anything less produces a tool people learn to ignore.
What we built instead
Arthur C. Clarke wrote that, "any sufficiently advanced technology is indistinguishable from magic." When recruiters first see the engine surface a cluster of four records — different emails, different phone numbers, two different names — and correctly resolve them as one person, magic is the word they reach for. But the reason it can be trusted is precisely that it isn't magic. It stands on fields with decades of theory behind them.
Probability and statistics. The engine weighs evidence the way a careful human would, at machine speed. No single coincidence makes a duplicate. For each record, it considers the full picture the platform holds: identity details, contact information, the background and history behind the record. Then, a match surfaces only when the evidence, taken together, is decisive. This draws on frequentist models of how much a given piece of evidence really proves: a rare, specific signal in agreement counts for a great deal; a common value in agreement counts for very little. That's how the engine catches duplicates that share no email and simultaneously refuses to be fooled by a shared office phone line.
Graph theory. Records and the evidence linking them form a connected structure: the mathematics of networks. This is what lets the engine do something pairwise matchers can't: rather than emitting a stream of "A might match B" alerts, it resolves complete per-person clusters. If five records describe one person, the recruiter sees one group and resolves that candidate once, completely.
Cluster analysis. The discipline that turns that connected structure into whole-person groups is the difference between a pile of pairwise suspicions and an answer.
All of this is engineering validated against real recruiting data and customer feedback, and its behavior is consistent and repeatable: the same evidence leads to the same conclusions, day after day. That repeatability matters more than it might sound. A detection system whose judgments drift is a detection system nobody audits, and nobody trusts.
The engineering decisions I'm proudest of
We calibrated for precision, deliberately. Every detection system faces the classic statistical tradeoff between precision (when we flag a duplicate, how often are we right?) and recall (of all the duplicates that exist, how many do we find?). The two pull against each other. By chasing every conceivable match, people drown in false alarms; flag only certainties and some duplicates stay hidden. We tuned decisively toward precision, because the entire workflow rests on trust. What surfaces is a high-confidence group a recruiter can act on immediately, not a wall of maybes.
And here's the part I love: recall isn't sacrificed so much as scheduled. Detection re-runs continuously, recomputing the full picture from scratch. A real duplicate that isn't decisive today tends to become decisive as new evidence arrives, and, when it does, the engine finds it on the next pass. No stale one-time scan, no aging index being patched at the edges.
We refused to brute-force it. Duplicate detection is a famously explosive computational problem weighing every record against every other record grows far faster than the record count itself, and simply doesn't scale. The engine instead zeroes in intelligently on the records that could plausibly be the same person and skips the overwhelming majority that plainly aren't. That's the property that lets us routinely recompute an organization's entire pipeline, at any organization size, and keep results fast and fresh.
We built the noise model into the foundation. Real candidate data is messy in predictable ways, and the engine is built for them specifically. It knows "Bob" and "Robert" can be the same person. It knows a placeholder email or a shared phone number is not evidence of identity. It knows a different name is not a disguise. Identity is weighed across the whole record, not read off the name field, records whose names disagree outright still find each other when the rest of the evidence is decisive.
We made human judgment part of the model, not a bolt-on. Nothing is ever merged automatically, not in any configuration. The AI identifies and proposes; a recruiter reviews; every merge and every dismissal is a deliberate human decision, and every decision is recorded. The engine also genuinely takes no for an answer: when a recruiter dismisses a group, that decision persists through every subsequent run. Only genuinely new evidence — a new record connecting to the dismissed group — reopens the question, and then as a materially different question, not a nag.
This is also what makes the failure mode graceful. If the engine ever groups records that aren't the same person, the cost is a moment of review and a dismissal. A wrong suggestion never silently corrupts data.
What it means in practice
For the recruiters who use it, the machinery disappears entirely, which is the point. Duplicate warnings appear across the candidate and job views where they already work. A dedicated Manage Duplicates page collects every detected group in one place. A merge folds the records into one clean profile and brings every application from all of them along, so nothing about a candidate's engagement history is lost, which is what makes the merged record trustworthy for fee attribution and for seeing a candidate's whole story in one place.
There's no separate tool, no export-and-reconcile project, no button to remember, no report to schedule. Detection runs itself in the background on a regular cycle, sweeping in new and changed records automatically. Results are simply current.
And each organization's detection runs entirely within its own candidate records. Your data is never used in another organization's results, and theirs never influences yours.
Where it goes from here
The algorithm is not a finished artifact, and we never intended it to be one. Real-world outcomes and customer feedback continuously feed its calibration, so the engine's judgment keeps getting better. It’s sharper at recognizing decisive evidence and wiser about the noise patterns of recruiting data.
When people ask me how the matching actually works, I give them the honest short answer: it's a multi-signal AI model calibrated to real-world data. The specifics are proprietary. The discipline is not; it's probability, graph theory, and cluster analysis, applied with engineering rigor and validated against the messiest, most human dataset there is.
It feels like magic, but it's actually math. And for the recruiters who get their hours back and the firms who get one trustworthy profile per person, that distinction is exactly why it works.