RAG quality can degrade even when every indexed document is technically correct. When the same information enters the corpus through copied pages, exported PDFs, email attachments, backups and old versions, retrieval can return several copies of one fact while crowding out independent evidence.

Detect exact duplicates during ingestion

Normalize whitespace and common boilerplate, then compute a stable content hash for each document and chunk. Exact matches can point to one canonical vector rather than being embedded repeatedly. Keep the duplicate source references in metadata so citations and access-control decisions still know every location where the text appeared.

Use semantic methods for near-duplicates. Documents often differ only in headers, dates, formatting or a few sentences. Fingerprints, token shingles and embedding similarity can detect material overlap that byte hashes miss. Thresholds need careful testing because two policy versions may be mostly identical while a small changed clause is the most important part.

Make deduplication version-aware

An old manual and a new manual should not collapse into one object just because ninety percent of the text is unchanged. Version metadata, update time and supersession links help preserve meaningful differences. The retrieval layer can prefer the current version while retaining historical provenance for audit or time-specific questions.

Strip repeated boilerplate early. Navigation text, privacy notices, legal footers and template introductions can appear on thousands of pages. Removing or down-weighting them prevents generic text from taking up a large fraction of the vector index. Boilerplate removal also improves chunk boundaries because the first chunk of every page no longer begins with the same site-wide text.

Deduplicate after chunking too

Two documents may be different overall but share an identical policy section or product table. Document-level deduplication will miss this repeated evidence. Chunk-level fingerprints find reused subsections and prevent top-k results from being dominated by the same paragraph copied across many sources.

Preserve source lineage. A canonical chunk can keep references to every source document, location and version that contained the text. This lets the answer cite the most authoritative or user-accessible source without storing duplicate embeddings. Lineage is essential when permissions differ between two copies of the same information.

Diversify final retrieval

Even a clean index can return several chunks from one long document. Apply source caps, section diversity or maximal marginal relevance so the final context covers more independent evidence. This is particularly useful for research tasks where multiple sources should support a conclusion rather than one document repeating itself.

Measure the duplication ratio. Track exact duplicates, near-duplicates and superseded chunks as a percentage of the corpus. Sudden growth can reveal a connector ingesting backups, revisions or mirrored content unexpectedly. Large duplicate clusters should be visible in data-quality dashboards so teams can fix the source pipeline rather than only tuning retrieval.

Consider permissions before canonicalization

Two identical documents may belong to different tenants or access groups. Never merge them in a way that allows one user to retrieve content because another copy was accessible elsewhere. The canonical representation must retain permission boundaries or create separate access-scoped references.

Treat generated summaries cautiously. AI-generated summaries of one source may overlap semantically while emphasizing different facts. Automatically collapsing them can remove useful perspectives or hide the primary source. Where possible, prioritize original documents and label generated derivatives explicitly in provenance.

Use authority when choosing the canonical source

The newest file is not always the best source. Official documentation, signed policy, approved knowledge base or primary research may outrank a newer copied page. A canonicalization rule can combine freshness, ownership and authority rather than relying on ingestion order.

Re-embed only when content materially changes. If a new version differs only in metadata that does not affect retrieval, the system may reuse an existing embedding while updating source references. Larger semantic changes should trigger a new vector. This reduces embedding cost without confusing old and new evidence.

Evaluate retrieval before and after cleanup

Run representative queries and compare source diversity, answer accuracy, citation quality and context token usage. The objective is not the smallest possible index; it is better evidence selection. Our guide to RAG source lineage explains how stable source IDs and versions make these comparisons auditable.

Repeated evidence can make a model look confident for the wrong reason. Deduplication, version awareness and diverse retrieval improve the information environment before the LLM receives any context, often producing better answers while lowering index and token cost.