One Vendor, Four Spellings: How Deterministic Stages Beat Similarity Scores

My agency buys editorial placements from independent publishers. The supplier list is a little over 10,000 websites, every one of them a company that sends us invoices, and keeping that list clean is the least glamorous problem we have. It is also the one that costs money when it goes wrong. A duplicate supplier record is how you pay the same invoice twice, and split records mean split price history, so the rate you negotiated last year is filed under a spelling nobody searches for.

Since January we have been syncing an external marketplace catalog into our internal system through its API, on top of a decade of manual entry by account managers and spreadsheets that arrive by email. The same site now shows up as solartravelmag.com, https://www.solartravelmag.com/, blog.solartravelmag.com, and on a bad day SolarTravelMag.COM with a tracking parameter attached. One marketplace we sync is upfront that its listed prices match reality about 90% of the time, which is honest of them, and which means cross-checking prices between sources is mandatory. Cross-checking only works if you know two rows describe the same site. That is entity resolution, a problem with formal theory going back to Fellegi and Sunter in 1969 [1], and it has a property the tutorials rarely lead with: the string matching is the easy part. Deciding what to do with the scores is the job.

This article walks through the pipeline that cleaned our list, run end to end on a synthetic rebuild of it, with ground truth labels so every precision claim below is checkable. The punchline, in one sentence: two boring deterministic stages removed 76% of the duplicates for free, the fuzzy matcher scored 33.7 million pairs in two seconds, and no threshold existed that could safely merge what remained. The matcher’s real output is a ranked review queue, not a set of merges.

A synthetic copy of a real mess

I cannot publish our vendor list, so I rebuilt one with the same diseases. Seven thousand invented publisher brands received domains across a realistic spread of endings, and then I corrupted the list the way ours gets corrupted: 2,429 surface variants (schemes, www prefixes, random casing, paths, tracking parameters), 888 subdomain rows like blog. and m., 529 country-domain siblings such as brand.com sitting next to brand.de, and 505 one-keystroke entry typos. I also planted 180 traps: domains one edit away from an existing brand that belong to a different vendor entirely, because the real web is full of near-name neighbors and any honest test set needs them.

The result is 11,531 rows describing 7,180 actual vendors. The generator is fifty lines of word lists and deliberate vandalism; the corruption recipe above is enough to rebuild it. Everything ran on Python 3.12 with RapidFuzz 3.14.5 [2] and tldextract 5.3.2 [3] on a single-vCPU cloud sandbox, so none of the timings below required serious hardware.

The boring stages do most of the work

Stage one is normalization, and it is deliberately dumb: lowercase, strip the scheme, cut the path and query string, drop a leading www.

import redef norm_host(raw: str) -> str:    s = raw.strip().lower()    s = re.sub(r"^[a-z]+://", "", s)    s = s.split("/")[0].split("?")[0]    if s.startswith("www."):        s = s[4:]    return s.rstrip(".")

That took the list from 11,531 rows to 9,080 unique hosts in 0.03 seconds. No model, no scores, 2,451 duplicates gone.

Stage two collapses subdomains, and this is where people who split on dots get hurt. The registered domain of blog.solartravelmag.com is solartravelmag.com, but the registered domain of solartravelmag.co.uk is not co.uk, and the only way to know that is the Public Suffix List [4], a maintained catalog of every suffix under which the public can register names. The tldextract library wraps it:

import tldextract# pin to the bundled suffix snapshot so runs are reproducibleextract = tldextract.TLDExtract(suffix_list_urls=())registered = {}for host, row in hosts.items():    ext = extract(host)    reg = ext.domain + "." + ext.suffix if ext.suffix else host    registered.setdefault(reg, row)

Another 873 hosts collapsed, leaving 8,207 registered domains. Two deterministic stages, zero judgment calls, and 3,324 of the 4,351 duplicate rows are already gone. If you take one number from this article, take that one: 76% of the problem never needed a similarity score at all.

What is left is the hard quarter. The ground truth says 1,135 true duplicate pairs are still hiding in those 8,207 domains, split records that only a cross-domain comparison can find: the entry typos and the country-domain siblings.

Brute force is fine, and blocking still earns its keep

8,207 domains make 33.7 million possible pairs, which sounds like the moment to reach for clever indexing. It is not, at least not for speed. RapidFuzz computes the full similarity matrix in one call:

import numpy as npfrom rapidfuzz import fuzz, processM = process.cdist(labels, labels, scorer=fuzz.ratio,                  dtype=np.uint8, workers=-1)

Two seconds, 67 MB. On a list this size, brute force is a coffee sip.

Blocking matters for two other reasons. First, scale: at 100,000 vendors the same matrix is 10 billion cells and 10 GB, and the coffee sip becomes an outage. Second, and underrated: a scored matrix is not a to-do list. You want candidate pairs you can rank and route, not a wall of numbers. So I blocked anyway, comparing only domains that share a key:

from itertools import combinationsblocks = {}for reg in domains:    label = label_of[reg]                 # the part left of the suffix    for key in ("P" + label[:2], "S" + label[-3:]):        blocks.setdefault(key, set()).add(reg)pairs = set()for members in blocks.values():    pairs.update(combinations(sorted(members), 2))

Blocking has a price, and you should measure it instead of hoping. With only the two-character prefix key, the candidate set caught 93.0% of the true duplicate pairs; typos that hit the first two letters escaped their block. Adding the last-three-characters key raised that to 99.8%, at the cost of growing the candidate set from 864,600 pairs to 1,690,124. Scoring all of them took 1.1 seconds. Exactly two true pairs slipped past both keys. On a real list you will not have ground truth, so estimate blocking recall on a labeled sample before trusting it.

Two humps that refuse to separate

Here is where the tutorial version of this story ends and the operational version begins. I scored every candidate pair and split the histogram by ground truth.

Below a score of 80 the data is clean: not one true duplicate lives down there. Above 97 it is clean the other way, with one enormous catch I will get to. The region between 88 and 97 is a genuine mixture, and it is a mixture for a structural reason, not a tuning one. A one-keystroke entry typo of a 14-character domain scores in the low 90s. A genuinely different vendor whose name is one edit away from yours also scores in the low 90s. The string cannot tell you which situation you are in, because the two situations are the same string event with different owners behind them.

The sweep makes it concrete. Auto-merging at 85 has a precision of 0.345, meaning two of every three merges are wrong. At 90, precision reaches 0.764, so one merge in four is still wrong. At 95 you get precision 0.895 and have already dropped recall to 0.738. Only at 98 does precision hit 1.0, with recall at 0.485.

And now the catch. Every single pair scoring in that perfect zone, all 551 of them, is the same name under two different country domains: brand.com next to brand.de or brand.co.uk. The scorer is certain the strings match, and the strings do match, but whether brand.de is the German arm of the same publisher or an unrelated company that grabbed the same word is knowledge about companies, and no amount of certainty about the text supplies it. In our real list both cases occur. So the one region where the matcher is never wrong about the text is the region where being right about the text settles nothing. Those pairs go to a human, by rule:

def route(a: str, b: str, score: int) -> str:    if score == 100 and suffix_of[a] != suffix_of[b]:        return "review"     # brand.com vs brand.de is a business question    if score >= 88:        return "review"    return "ignore"

Notice what is missing: a merge branch. After the deterministic stages, nothing that remained was safe to merge automatically.

Buy recall with minutes, not thresholds

What blind auto-merging costs at every threshold. Precision only reaches 1.0 at a score of 98, by which point recall has fallen below half, and the pairs remaining at that level are country-domain siblings that need a human decision anyway.

Since no threshold merges safely, the threshold’s real job changes: it sizes the review queue. That turns a statistics knob into a staffing decision, and the numbers make the trade explicit. The 88 to 99 band contains 1,354 pairs, of which 565 are true duplicates. Add the 551 forced-review country siblings and a reviewer faces 1,905 pairs. At six pairs a minute, that is just over five hours: one working day to resolve a decade of vendor mess. Widening the band to 85 adds 1,378 more pairs, roughly four extra hours, and recovers exactly 15 additional true duplicates. Dropping the floor to 80 balloons the queue past 30 hours and recovers two more. Whoever owns the supplier budget can now choose a point on that curve while knowing precisely what each hour buys.

I simulated both endings. The review path, with the reviewer resolving the queue correctly, lands at 7,189 entities against a ground truth of 7,180, having repaired 1,116 of the 1,135 duplicate relationships. The 19 misses are 17 pairs that scored between 80 and 87 and the 2 pairs blocking lost. The other ending is the one the tooling makes temptingly easy: blind auto-merge at 90 and go home. That run produced 6,914 entities, and 282 of them are records fusing two or more genuinely different vendors. In accounts payable terms, 282 suppliers whose price history is now contaminated with someone else’s rates, attached to a record that pays somebody. A messy list costs you comparisons. A confidently wrong list costs you transfers.

Two habits made the review day itself cheap. Sort the queue by money, so pairs involving expensive placements get eyes first and the long tail can wait for a slow Friday. And make every merge reversible: keep both source rows, write a merge log, never overwrite. Half the value of a human in the loop disappears if her mistakes are permanent too.

What transfers to your list

Nothing above is specific to publishers. Vendor names, customer accounts, and affiliate partner lists fail in the same shapes: surface noise that normalization kills, hierarchy that a canonical form collapses, and a residue of near-matches where the string alone underdetermines the answer. When you have more columns than I used here, spend them in the review interface rather than the score. A shared contact email or bank detail turns a 91 into a certainty in two seconds of human attention; folding it into a composite score just moves the ambiguity somewhere harder to see. And if your entities are domains, respect the Public Suffix List. Splitting on dots is how co.uk becomes your biggest supplier.

Conclusion

The pipeline that cleaned the list is unglamorous. Normalize hard, collapse to registered domains, block on two keys, score everything once, and route by rule: certainties were already collapsed, impossibilities stay ignored, and the mixed middle becomes a queue a person can finish in a day. On short strings like domain names, a similarity score is an argument for a human to weigh, and the data here shows it plainly: three in four merges wrong at 85, one in four wrong at 90, and perfection arriving only in the exact region where the string stops being the question. Rank the queue, size it to the hours you have, keep the merges reversible, and let the deterministic stages do the bragging.

All charts in this article were generated by the author with matplotlib. No AI image tools were used.

References

[1] I. Fellegi and A. Sunter, A Theory for Record Linkage (1969), Journal of the American Statistical Association

[2] M. Bachmann, RapidFuzz: rapid fuzzy string matching in Python (2026), GitHub

[3] J. Kurkowski, tldextract (2026), GitHub

[4] Mozilla Foundation, Public Suffix List (2026), publicsuffix.org