[{"data":1,"prerenderedAt":72},["ShallowReactive",2],{"help-article-the-dedup-mechanism":3},{"data":4},{"title":5,"excerpt":6,"content":7,"help_category":8,"seo":11},"The Dedup Mechanism","How does m18t stop the same news article appearing twice? Learn how the URL fingerprint deduplicates feed items and why cleared old stories stay gone.","\u003Cp>News Manager keeps a fingerprint of every article it has seen, so re-scouting a feed brings in new stories without piling up copies of old ones.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>What you'll learn\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Why the same feed can be scouted repeatedly without flooding your queue.\u003C\u002Fli>\n\u003Cli>How News Manager decides an article is a duplicate (and the one case it can't catch).\u003C\u002Fli>\n\u003Cli>Why a cleared-out old article won't come back.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Chr>\n\u003Ch2>The concept: seen once, skipped after\u003C\u002Fh2>\n\u003Cp>Feeds repeat themselves. A source keeps its last 20 or 50 items in the feed at all times, so every time you scout it, most of what comes back is what you already pulled in last time. Without a guard, your queue would fill with the same headlines on every run.\u003C\u002Fp>\n\u003Cp>News Manager prevents that with a fingerprint. When \u003Ca href=\"\u002Fhelp\u002Fthe-news-manager-concept\">Scout\u003C\u002Fa> discovers an article, it computes an md5 hash of the article's web address and stores it on the feed item as a unique key. Before adding a newly discovered article, News Manager checks whether that hash already exists. If it does, the article is treated as already known and isn't added again.\u003C\u002Fp>\n\u003Cp>The practical result: you can scout the same feed as often as you like. Only articles you haven't seen before get added; everything already in your queue is left alone.\u003C\u002Fp>\n\u003Ch2>What counts as a duplicate\u003C\u002Fh2>\n\u003Cp>Two things define how dedup behaves, and both are worth knowing:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>It's based on the web address.\u003C\u002Fstrong> The fingerprint is the md5 hash of the article's URL — not its headline or body. Two articles at the same address are the same article. Two articles at different addresses are treated as different, even if the text is nearly identical.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>It's scoped to your brand.\u003C\u002Fstrong> Dedup is per brand. An article seen by one brand doesn't block the same address from appearing in another brand's queue. Each brand keeps its own record of what it has seen.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>When you preview a source's feed, items News Manager already knows are marked with their status, while genuinely new ones are flagged \u003Cstrong>New\u003C\u002Fstrong> — that's the dedup check showing its work before you even scout.\u003C\u002Fp>\n\u003Ch2>Why an old article won't come back\u003C\u002Fh2>\n\u003Cp>News Manager tidies up old items over time — its \u003Cstrong>Cleanup\u003C\u002Fstrong> action slims aged and skipped feed items so the queue stays fast to scan. You might expect that clearing an item would let it slip back in on the next scout, since the feed still carries it.\u003C\u002Fp>\n\u003Cp>It won't. The lightweight row and its url-hash key always survive cleanup; only the heavy raw content is slimmed. So a story you've already seen and cleared stays recognized as already known and won't reappear. Cleanup frees up the queue without resetting the memory of what's been seen.\u003C\u002Fp>\n\u003Ch2>The one case where a duplicate slips through\u003C\u002Fh2>\n\u003Cp>Because dedup keys on the web address, it can't catch a story that lives at two different addresses. The same article is often \u003Cstrong>syndicated\u003C\u002Fstrong> — published by a news agency, then re-hosted by several outlets, each at its own URL. To News Manager, those are different addresses, so they get different hashes, and both can appear in your queue.\u003C\u002Fp>\n\u003Cp>This is a known limit, not a bug to wait out. If you follow several sources that carry overlapping wire stories, expect the occasional genuine duplicate. When you spot one, \u003Ca href=\"\u002Fhelp\u002Fthe-news-manager-concept\">skip\u003C\u002Fa> the extra copy by hand.\u003C\u002Fp>\n\u003Ch2>FAQ\u003C\u002Fh2>\n\u003Cp>\u003Cstrong>If I scout the same feed twice, will I get everything twice?\u003C\u002Fstrong>\nNo. Only articles with a URL you haven't seen before are added. The rest are recognized by their address fingerprint and skipped.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>The same story showed up twice — isn't dedup broken?\u003C\u002Fstrong>\nProbably not. If two outlets re-hosted the same wire story at different URLs, those are different addresses and get different fingerprints, so both appear. That's the known syndication limit. Skip the duplicate manually.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Will running Cleanup make old articles reappear?\u003C\u002Fstrong>\nNo. The url-hash key survives cleanup, so cleared stories stay recognized as known and won't be re-added.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Does dedup work across my brands?\u003C\u002Fstrong>\nNo. It's per brand. The same article can appear independently in two brands' queues, each tracked separately.\u003C\u002Fp>\n\u003Ch2>What's next\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>See where dedup fits in \u003Ca href=\"\u002Fhelp\u002Fthe-news-manager-concept\">The News Manager Concept\u003C\u002Fa>.\u003C\u002Fli>\n\u003Cli>Add the feeds it watches in \u003Ca href=\"\u002Fhelp\u002Fadding-a-news-source\">Adding a News Source\u003C\u002Fa>.\u003C\u002Fli>\n\u003Cli>Once an article is in the queue, the next AI step is \u003Ca href=\"\u002Fhelp\u002Fthe-evaluate-stage\">The Evaluate Stage\u003C\u002Fa>.\u003C\u002Fli>\n\u003Cli>Approved articles become drafts in \u003Ca href=\"\u002Fhelp\u002Fthe-generate-stage\">The Generate Stage\u003C\u002Fa>.\u003C\u002Fli>\n\u003C\u002Ful>\n",{"slug":9,"title":10},"news-manager-help","News Manager",{"meta_title":12,"meta_description":6,"keywords":13,"canonical_url":14,"og_title":5,"og_description":15,"og_image":16,"robots":17,"json_ld":18},"The Dedup Mechanism | m18t Help","news dedup, duplicate articles, url fingerprint, md5 hash, syndication, cleanup, per brand, m18t","https:\u002F\u002Fm18t.com\u002Fhelp\u002Fthe-dedup-mechanism","Why you can re-scout a feed without flooding your queue: how dedup fingerprints an article by its web address, the one case it can't catch, and cleanup.","https:\u002F\u002Fm18t.com\u002Fog-default.png","index, follow",{"@context":19,"@graph":20},"https:\u002F\u002Fschema.org",[21,35,56],{"@type":22,"headline":5,"description":23,"articleSection":10,"inLanguage":24,"datePublished":25,"dateModified":25,"image":16,"author":26,"publisher":30,"mainEntityOfPage":33},"Article","How does m18t stop the same news article appearing twice? Learn how the URL fingerprint deduplicates feed items, the syndication limit, and why old stories stay gone.","en","2026-06-25",{"@type":27,"name":28,"url":29},"Organization","m18t","https:\u002F\u002Fm18t.com",{"@type":27,"name":28,"url":29,"logo":31},{"@type":32,"url":16},"ImageObject",{"@type":34,"@id":14},"WebPage",{"@type":36,"mainEntity":37},"FAQPage",[38,44,48,52],{"@type":39,"name":40,"acceptedAnswer":41},"Question","If I scout the same feed twice, will I get everything twice?",{"@type":42,"text":43},"Answer","No. Only articles with a URL you haven't seen before are added. The rest are recognized by their address fingerprint and skipped.",{"@type":39,"name":45,"acceptedAnswer":46},"The same story showed up twice — isn't dedup broken?",{"@type":42,"text":47},"Probably not. If two outlets re-hosted the same wire story at different URLs, those are different addresses and get different fingerprints, so both appear. That's the known syndication limit. Skip the duplicate manually.",{"@type":39,"name":49,"acceptedAnswer":50},"Will running Cleanup make old articles reappear?",{"@type":42,"text":51},"No. The url-hash key survives cleanup, so cleared stories stay recognized as known and won't be re-added.",{"@type":39,"name":53,"acceptedAnswer":54},"Does dedup work across my brands?",{"@type":42,"text":55},"No. It's per brand. The same article can appear independently in two brands' queues, each tracked separately.",{"@type":57,"itemListElement":58},"BreadcrumbList",[59,63,67,70],{"@type":60,"position":61,"name":62,"item":29},"ListItem",1,"Home",{"@type":60,"position":64,"name":65,"item":66},2,"Help","https:\u002F\u002Fm18t.com\u002Fhelp",{"@type":60,"position":68,"name":10,"item":69},3,"https:\u002F\u002Fm18t.com\u002Fhelp\u002Fcategories\u002Fnews-manager-help",{"@type":60,"position":71,"name":5,"item":14},4,1786312171230]