Why Do Answers Start to Conflict as a RAG Knowledge Base Grows?

Two Answers Before the Morning Meeting

Twenty minutes before the morning meeting, operations lead Xiao Zhou is asked a very specific question: when the same database connection spike alert appears again today, should the on-call team restart the service first, or isolate traffic first?

The team's knowledge base quickly finds two answers. A runbook from three years ago says, "restart and observe." Last month's incident review, however, records that "for this kind of symptom, limit traffic first; restarting directly will expand the impact."

Neither document is nonsense. The first corresponds to an earlier architecture, while the second comes from handling experience after dependency relationships changed. But when both are retrieved in the same search, Xiao Zhou does not get an answer. He gets two choices that both look like answers.

This is the hardest shift many RAG knowledge bases face after long-term use: the materials are not lost, and the hit rate may even improve, but answer credibility starts to decline. The problem is not necessarily that the model is fabricating. It is that knowledge has lost order around version, scope, and effective state.

The Root Cause: Relevant Does Not Mean Valid

RAG works in a clear way: first retrieve fragments related to the question from the knowledge base, then let the model organize an answer from those fragments. It is good at solving "how can scattered materials be found," but it does not naturally solve "should this retrieved sentence still be executed now."

An old SOP may still apply to a legacy environment. An incident-review conclusion may only fit one special incident. A product page may already have been replaced by a later version. They are all semantically related to the question, but they should not have the same decision weight.

When materials only increase and similar topics only pile up, retrieval brings more historical experience into the same context. The model can make the text fluent, but semantic similarity cannot confirm for the team which version is effective, which condition is missing, and which sentence should only remain as historical reference.

The moment a knowledge base truly distorts is often not when content becomes wrong, but when once-correct content leaves its applicable conditions and still participates in answers as current guidance.

Updating Cannot Rely Only on Re-Ingestion

When new materials appear, the most common action is to upload, split, ingest, and expect newer content to naturally outrank older content. This path is convenient, but it leaves blanks during the next follow-up: why did the answer change? Which material introduced the change? Is the original approach still valid in some scenarios?

Directly overwriting the current page with new content erases that context. Keeping both old and new files inside the retrieval scope makes the same question repeatedly surface conflicting fragments. Neither choice works well for long-term maintenance.

A more stable update chain should first turn materials into reviewable candidate content, then decide whether they affect the currently effective knowledge. Taking OpsPilot as an example, materials are built into knowledge pages through a queue, while retaining the source material, build time, and build records. Material updates can generate multiple candidate versions instead of directly overwriting the current effective page.

This candidate stage is not just another approval step. It leaves room for team judgment. Xiao Zhou can first compare the new review with the old SOP: if the review only applies to one special dependency incident, add the missing conditions; if it truly overturns the old steps, let the new version replace the current effective page. In this way, an update becomes a reviewable knowledge change, not a silent file overwrite.

Conflicts Need to Become Pending Confirmation Items

What blocks Xiao Zhou next is not "can another search find a few more documents," but why these two pieces of content answer the same question at the same time.

This kind of situation needs to be explicitly identified as a "same-knowledge conflict": when multiple materials give inconsistent content for the same knowledge item, it should enter pending confirmation instead of staying hidden inside the model's retrieval context. Reviewers need to look back at source, content, applicable scope, and current standards, then decide whether to adopt one item, merge the content, or keep the original conclusion.

It is easy to imagine an LLM Wiki as a system that automatically arbitrates truth. A more accurate position is this: it identifies contradictions in materials and presents them as governance objects. Which version finally becomes effective still needs confirmation from people who understand the business boundary. Unconfirmed content should not be published directly as official knowledge.

This also explains why simply increasing Top K or repeatedly adjusting prompts cannot solve the root problem. At most, they change "which materials are compared." They cannot replace the team's decision on whether a material should become official knowledge.

Technical Insight: Knowledge Trust Depends on State

For a RAG knowledge base to remain trustworthy over the long term, the core issue is not only document quality, but whether knowledge state is clear. The evolution of one knowledge item can be understood through four states:

• Traceable source: know which material the content came from and when it was built.
• Comparable candidate: updates do not directly overwrite effective knowledge, but first form candidate versions.
• Confirmable conflict: inconsistent content enters pending confirmation, where people decide whether to adopt, merge, or keep it.
• Explainable effectiveness: at any moment, the team can answer which version is currently retrieved and used.

If any one of these layers is missing, the knowledge base returns to a state of "many files, conclusions by guesswork." RAG improves the reachability of materials. The governance chain ensures that the materials found are still worth using. They do not replace each other; they are two sides of the same knowledge system.

Directories Are the Second Failure Point

After content is confirmed, the problem is not over. Xiao Zhou may finally get the updated SOP, only to find that it is still placed under the old system's runbook directory. Another review may be automatically classified into an unrelated category and later cited by a new colleague as a general solution.

So directories are not a minor presentation-layer detail. Pages that were manually moved should preserve their manual classification. If automatic classification needs to be restored, that should also be an explicit governance action. For structural changes such as directory merge, disablement, or archival, the team should preview the affected scope first, confirm affected pages and conflicts, and then execute. After structure or page state changes, the team should preview again before executing.

Content review answers "can this knowledge become effective." Directory governance answers "through which entry should it be reused, and by whom." If the former is incomplete, answers conflict. If the latter is incomplete, even correct content can be used in the wrong context.

Connecting One Incident Back to the Governance Chain

Returning to the morning-meeting question, the ideal process should not be Xiao Zhou betting on one of two texts based on experience. Instead, he should first see that the two materials form a same-knowledge conflict, check whether the new review covers the current environment, and then have the responsible owner confirm whether it should be merged into the official SOP or preserved as conditional experience.

After that, the new effective page has clear source and version state. Old materials can still be traced, but they no longer sit at the same level as the current standard. If the runbook directory needs adjustment, the team can see which pages will be affected in advance instead of discovering after cleanup that entry points were disrupted.

This chain turns "answering one question" from a one-time model output into a knowledge asset the team can continuously maintain. Materials can keep accumulating, and history can remain available, but they no longer have to compete for the position of "current answer."

Five Questions Before Getting Started

Before connecting long-term materials to RAG or an LLM Wiki, teams can run a simple self-check:

1. After new materials enter the knowledge base, can their source, build record, and time be traced?
2. When materials are updated, do they first form candidate versions instead of directly overwriting effective pages?
3. When inconsistent conclusions appear for the same knowledge item, is there pending confirmation and an explicit review decision?
4. Before directories are merged, disabled, or archived, is there a preview of affected pages and conflicts?
5. After the foundation model changes, are materials retrained and retrieval results verified?

If the answers to these questions are still uncertain, what usually needs to be added first is not more files, but a mechanism for knowledge update and confirmation.

Conclusion

The value of a knowledge base is not how much it remembers, but whether the team knows which version of knowledge to rely on when action is needed.

RAG can help information reach the model faster. LLM Wiki can bring conflicts, candidates, reviews, and directory impact into daily governance. Once this chain runs smoothly, accumulated materials will not turn into historical fragments that argue with each other over time. They will gradually become traceable, confirmable team knowledge that can be reused over the long term.