Near-duplicate detection
memista finds near-duplicates by embedding candidates and running a nearest-neighbour search: results within a small distance are likely duplicates. Runs in-process, best under ~100k items.
Deduplication and near-duplicate flagging are similarity problems: embed each item, and anything with a very close neighbour is probably a repeat.
The approach
- Embed each item and
POST /v1/insertit. - For a candidate,
POST /v1/searchwithlimitsmall. - If the nearest neighbour’s
distanceis under your threshold, treat it as a near-duplicate.
memista’s inner-product metric ranks normalised embeddings the same way cosine would, which is what you want for text similarity. Set the threshold on a small labelled sample and go.
Fit
This is a batch-y, in-process job — exactly memista’s shape. It’s tested under ~100k items, which covers most single-corpus dedup passes. For continuous dedup across a very large, growing dataset, a distributed store scales further; see memista vs Qdrant.
Related: insert vectors over HTTP, glossary.
Frequently Asked Questions
How do I decide what counts as a duplicate?
Pick a distance threshold on the search results. Items whose nearest neighbour is within that threshold are treated as near-duplicates; tune the threshold on a labelled sample.