Short answer

Your knowledge is ready to be prepared for an assistant when every source it draws on has an owner, a known freshness, a sensitivity rating and permissions that the assistant will respect, and when you have a set of real questions with known correct answers to test it against. If those are missing, fix them before choosing a platform. These are knowledge-preparation criteria, not a full production-readiness or security sign-off; the acceptance set and your security and privacy review still have to be passed.

The decision is not which AI product to buy. It is which body of knowledge is trustworthy enough to answer from, and what the assistant should do when the evidence is missing, out of date or not meant for the person asking. In those cases it should say so and point to a person, not guess.

Most organisations’ knowledge is spread across shared drives, intranet pages, ticket histories and inboxes. Some of it is authoritative, some is a draft someone forgot to delete, and some is confidential. People navigate this with judgement: they know which policy is current and who to ask. An assistant has no such judgement unless you build it into the sources and the rules around them.

What retrieval changes, and what it does not

An internal assistant can use retrieval-augmented generation (RAG): when someone asks a question, the system searches your content for relevant passages and gives them to a language model as context for its answer. Microsoft’s Architecture Center guide to designing and evaluating a RAG solution walks through the main stages: preparing documents, splitting them into chunks, enriching those chunks, searching them and evaluating the end-to-end result. It also distinguishes a standard single retrieval step from agentic retrieval, where the system plans and runs several searches.

The examples in that guide use Microsoft’s cloud services, but the stages apply whichever platform you use. The consequence for a business leader is simple: retrieval adds dependencies on the quality of your documents, the retrieval step and the grounding of the answer, while the model can still misread or misuse what it is given. All three layers need checking. A well-built system with poor sources gives confident answers from poor sources.

What retrieval does not change is who is allowed to know what. If a document is placed in a search index that the assistant can read for everyone, the assistant can surface it to everyone, unless the retrieval step filters results by the asking person’s permissions before anything reaches the model. A retrieval system is not a substitute for access controls; it has to enforce the ones you already have. That means carrying each document’s identity and access metadata into the index, refreshing it when permissions change, and applying the same protection to caches, conversation history and citations. A prompt telling the model to keep things secret is not an access control.

The readiness worksheet

Work through this worksheet for each body of knowledge you are considering, such as HR policies, product documentation or support history. It is ByteIQ’s proposed worksheet; the “not ready” column is usually where the real work lies.

ByteIQ’s proposed knowledge readiness worksheet for an internal assistant
AreaQuestion to answerReady whenNot ready if
OwnerWho is accountable for this content being correct?A named role owns each source and will act on reported errors.Nobody can say who maintains it.
Source qualityIs this the authoritative version?Approved documents are clearly marked; drafts and copies are excluded.Several versions exist and nobody is sure which applies.
FreshnessHow quickly do changes reach the index?Updates and withdrawals flow through on a known schedule.Withdrawn policies stay searchable indefinitely.
SensitivityWhat could go wrong if this were shown to the wrong person?Each source has a classification and highly sensitive content is excluded or tightly restricted.Personal, financial or commercial data is mixed into general content.
PermissionsWill the assistant respect who may see what?Access metadata travels with each document and is kept current; results are filtered by the asking user’s existing access rights before they reach the model, and this has been tested.Any user can retrieve content they could not open in the source system, because results are unfiltered or permission changes do not reach the index.
Representative questionsWhat will people actually ask?A set of real questions, gathered from help desks and teams, including awkward ones.Only questions written by the project team for a demonstration.
Expected evidenceWhich source should support each answer?Each test question lists the passage a correct answer should cite.Correctness is judged by whether the answer sounds right.
Missing or conflicting evidenceWhat should happen when sources disagree or are silent?The assistant says it cannot find an approved answer and names who to ask.It is expected to produce an answer every time.

The worksheet is our own proposed framework and covers knowledge preparation only. It does not replace a data protection assessment or your organisation’s security review.

A policy question, with and without evidence

Every one of those behaviours depends on decisions made outside the model: which documents are approved, how withdrawals are marked, who owns each policy, and whether search respects permissions. That is why readiness work comes first.

A demo is not an acceptance test

Demonstrations usually show the assistant answering well-phrased questions whose answers sit neatly in one document. That proves the pipeline runs. It does not prove the system is safe to give to staff.

An acceptance set for deployment should deliberately include the questions that go wrong in practice:

  • questions whose answer is spread across several documents;
  • questions with no approved answer, where the correct behaviour is to abstain and escalate;
  • questions where an old and a current document disagree;
  • questions asked by users who should not see the relevant content, to confirm nothing leaks;
  • questions outside the assistant’s scope entirely;
  • documents containing text that looks like instructions to the assistant.

For each question, record the expected source and the expected behaviour, then check three things: did it cite the right source, did it abstain when it should, and did it reveal anything the user was not entitled to see. For the permission cases, inspect the citations, source links and metadata as well as the answer text, since a restricted title or snippet leaks just as an answer does. Rerun the set whenever sources, prompts or the underlying model change.

Limits and failure modes

Retrieval reduces some risks and introduces others. Plan for these from the start.

  • It is not an access control system. Permissions must be enforced at retrieval, using the same rules as your document systems. A shared indexing account can legitimately ingest content, provided query-time authorisation trims results to what the asking person may see. The unsafe condition is broad, unfiltered retrieval that puts restricted content in front of the model or the user.
  • It does not guarantee correct answers. The model can still misread a passage, combine two sources wrongly or answer confidently from an irrelevant one. Cited sources make errors easier to spot; they do not prevent them.
  • Documents can carry instructions. Text inside a retrieved document can try to steer the model. The OWASP generative AI security project treats this kind of prompt injection, and sensitive information disclosure, as risks to design for.
  • Stale knowledge persists. Without a freshness process, the assistant keeps quoting a policy long after it changed.
  • Agentic retrieval adds moving parts. Multi-step search can help with complex questions but is harder to test, and depending on the design may increase requests, latency and cost per answer, so measure them. Start with standard retrieval unless your acceptance set shows it is not enough.
  • It may be the wrong tool. If people mainly need to find one well-maintained document, better search and navigation may serve them more cheaply than generated answers.

Governance without paralysis

Internal assistants touch personal data, confidential information and employee trust, so they need governance proportionate to the risk. The NIST AI Risk Management Framework is a voluntary framework that many organisations use to structure that thinking. Using it is a sensible discipline, not a certification. In practice, governance for a first assistant can be modest: a named owner, a documented scope, the readiness worksheet, the acceptance set, a way for users to report a wrong answer and a regular review of those reports.

Next steps

  1. Choose one bounded body of knowledge with a clear owner, such as one department’s policies, rather than indexing everything. Owner: business sponsor
  2. Complete the readiness worksheet for it and fix the “not ready” items, starting with ownership and permissions. Owner: content owner with the data lead
  3. Gather real questions from help desks and teams and record the expected source and behaviour for each. Owner: content owner
  4. Confirm with security and privacy colleagues how permissions will be enforced at retrieval and what must be excluded. Owner: data lead with security and privacy
  5. Build or configure a small pilot, run the acceptance set, and release only to a limited group with a clear way to report errors. Owner: delivery team

Our Data & Analytics service helps get the underlying knowledge in order, Cloud & DevOps covers secure hosting and access, and AI & Business Automation builds and evaluates the assistant itself. To discuss your own knowledge sources, contact us about data readiness.

Sources and further reading

  1. Design and develop a RAG solution, Microsoft Azure Architecture Center. Preparation, chunking, enrichment, retrieval and evaluation; standard and agentic retrieval. Cloud-specific examples; the stages are general.
  2. OWASP Top 10 for LLM and generative AI initiative. Used for the broad concepts of prompt injection and sensitive information disclosure, not for a ranking.
  3. NIST AI Risk Management Framework. Voluntary governance framework; not a certification.

About this guide: written by the ByteIQ Labs editorial team. The worksheet, scenario and acceptance-set approach are our own analysis; no specific platform is recommended. See all guides on our Insights page.