Procurement decisions need evidence from the intended operating context

By Manfredi Marrocco

A supplier demonstrates an AI assistant answering a difficult question. The response is fluent and includes references. The demonstration ends with a high benchmark score. A purchaser concludes that the assistant is ready to handle internal policy questions.

The demonstration may be accurate. It does not establish how the system performs on the purchaser's documents, users, permissions and failure cases.

Define the use before comparing vendors

Imagine a fictional company, Seabrook, seeking an assistant to help employees locate internal procedures. The intended use is to retrieve and summarise authorised documents; the assistant is not authorised to change a policy or approve an exception. Seabrook needs to define which users can access which documents, how current versions are identified and what happens when the assistant cannot find a reliable answer.

The NIST AI RMF core places context, governance and measurement across the AI lifecycle. It provides voluntary guidance. The procurement checks below are an example, not a prescribed certification scheme.

Evaluate the combined workflow

A model benchmark may measure performance on a particular test set. Seabrook's operational risk also depends on retrieval, document permissions, user interface, monitoring and human use. A model that answers general questions well may still expose an unauthorised document if the surrounding access controls are incorrectly configured.

The evaluation should include representative questions, ambiguous requests, outdated documents and attempts to retrieve information outside the user's entitlement. Use authorised test material and avoid putting confidential data into a service before the organisation has assessed the arrangement. Record the system version and conditions tested.

The team should assess whether the response is supported by the correct document, whether the document is current, whether the cited passage supports the claim and whether the user can identify uncertainty. A citation can improve traceability; its presence does not automatically prove the answer is correct.

Ask about change and exit

Supplier information should cover the service boundary, data handling, subcontractors, update process, incident contacts and available evidence. The appropriate legal and contractual assessment depends on the data and roles involved. A supplier's refusal or inability to disclose a detail is not proof of failure, but it can create uncertainty that matters to the purchasing decision.

Seabrook should also consider what happens if the model or service changes. Who is notified? Can the organisation repeat its tests? Can it disable the assistant or switch to a documented manual search? A fallback that relies on the same unavailable identity system is not independent.

Design a test set around the business question

Seabrook can start with tasks employees actually need to perform: locate the current travel policy, identify who approves an exception and distinguish a superseded procedure from its replacement. Include cases where the correct response is to say that the available material does not establish an answer. A tool that always produces a fluent response may be less useful than one that makes its evidential limits visible.

Qualified reviewers should define the expected answer and supporting document before examining the assistant's output where feasible. Retain disagreements rather than forcing an artificial consensus. Some policy questions may be ambiguous because the underlying documents conflict; that is a content-governance issue as well as an evaluation issue.

Include a question the assistant should decline or escalate because the available source does not support an answer. Correct behaviour is sometimes recognising that limit, rather than producing a fluent response. The purchasing team should decide the expected route in advance and verify that a user can actually follow it. A benchmark composed only of answerable questions cannot establish how the service handles missing authority or evidence.

Keep access control in the evaluation

Use controlled accounts with different entitlements and a set of non-sensitive documents designed for the test. Check whether retrieval and response generation respect the source permissions. Then change a permission and test again. A cached index can behave differently from a live source, so the timing of permission updates matters.

A successful access test demonstrates the paths exercised. It does not prove that every integration and document is covered. Record exclusions and ask whether they matter to the deployment decision. If the highest-consequence repository is excluded, a good result on public company policies should not be presented as validation of restricted-document handling.

Evaluate the operating burden

The assistant may save time on routine questions but create work reviewing ambiguous answers, maintaining document versions and investigating reports. A pilot should observe those tasks as well as response speed. A supplier's latency benchmark does not measure the employee's time to verify a reference or obtain a correct answer after an initial error.

Choose an owner for the knowledge base, one for the service and a route for security and privacy issues. Those roles can be combined in a small organisation, but their tasks should remain explicit. If nobody owns removal of obsolete documents, retrieval quality can degrade even when the model stays unchanged.

Decide what result permits expansion

Before the pilot, state which evidence would support a wider rollout and what would require correction or restriction. The criteria should reflect consequences, not just a target percentage. One unauthorised disclosure may matter more than many minor wording errors, depending on the material and scope.

The decision record can connect each material finding to a treatment and retest. If the supplier changes the model during the pilot, record the version and consider whether results remain comparable. Procurement should preserve enough information to repeat the evaluation or explain why a previous conclusion no longer applies. The demo becomes useful when it leads to a testable operating decision rather than a purchase justified by one impressive answer.

Use consequence-aware pilot criteria

A pilot exit review can use a small evidence table: intended task, expected behaviour, observed result, consequence and decision. That format prevents a minor wording issue and an unauthorised disclosure from becoming two equivalent failed test cases in one percentage. It also helps explain why a service may be suitable for public policy search but not yet for a restricted repository. The conclusion should name the approved scope and the conditions that would require reassessment, including a material supplier update or a change in the documents available to retrieval.

A limited deployment should preserve an accessible route to the original documents. Employees need to know when to leave the assistant and seek the authoritative source or a responsible person.

Make the pilot a decision, not a ceremonial launch

Set a limited scope, accountable owner, evaluation criteria and route for users to report incorrect or unsafe outputs. Record results and limitations. If testing reveals problems, the organisation can correct the integration, narrow the use or postpone deployment. A successful pilot supports a conclusion about the tested conditions; it is not a guarantee for all future documents or users.

The supplier's demo is evidence about a demonstration. Procurement becomes governance when the buyer tests whether the complete service can support its own decision safely and within its own constraints.

The drama of Mr. Robot often centres on the moment access is obtained. A more useful operational question is what a team can decide during the first uncertain minutes after suspicious access is detected. The scenario below is fictional and uses synthetic facts; it is not a client case.

Separate the observation from the story

The identity platform reports an export by a support account at 09:12. The account owner says they did not initiate it. The file contents, recipient and scope are not yet confirmed. Record these as three separate statements: a system observation, a person's report and an unknown.

This distinction changes the response. “Confirmed breach of all customer records” would be an unsupported conclusion. “Nothing happened because the login was valid” would be equally premature. An incident coordinator can authorize proportionate containment while analysis continues.

Map the decision before choosing a control

For the fictional service, four people need to participate:

• The incident coordinator owns the timeline and escalates decisions.
• The identity administrator can restrict sessions and credentials.
• The service owner understands the effect of interruption and available fallback.
• A privacy or legal specialist assesses whether notification duties apply to the facts.

If any of these roles relies on a single person, record an alternate. The technical operator needs a way to act, but the service owner must know what the action will interrupt.

Three containment choices

Option A: suspend the account and revoke active sessions. This limits further use of that identity, but can interrupt support operations. Verify whether related tokens and delegated credentials also require separate revocation; changing a password alone may not invalidate every session.

Option B: restrict the export capability while retaining limited service. This may preserve support, but it assumes permissions can be changed quickly and that the suspected actor has no alternative path. Treat that assumption as something to test, not as a guarantee.

Option C: isolate the affected service. This may reduce exposure if compromise extends beyond one identity, but may cause a wider outage and change the evidence available to investigators. The decision needs an explicit owner, scope and review time.

None is universally correct. Choose based on observed activity, potential impact, available evidence and the cost of delay. Involve qualified incident responders before destructive or irreversible steps.

Preserve what will disappear

Before routine retention or response actions overwrite information, identify the relevant identity, access and export logs and who can preserve them. Record the source, time range, collection method, custodian and access controls. A cryptographic hash helps detect later changes to a collected copy; it does not prove that the original collection was complete.

Do not copy sensitive data into a shared incident chat for convenience. Limit access to those who need it, and document gaps such as unavailable logs or uncertain time synchronisation.

Establish a decision log

A simple log can distinguish evidence from assumptions:

These times are invented for the exercise. Record actual times and uncertainty in a real event. The log supports handover and prevents a decision from being mistaken for a confirmed fact.

Decide when to return

A working login is not evidence that the incident is resolved. Before restoring normal access, agree who checks the scope of the incident, removes unauthorised access, validates the service, and monitors for recurrence. If a step cannot be completed, the person accepting that limitation should record it explicitly.

Afterwards, run a short exercise: can an alternate coordinator find the contact list without the affected identity provider? Can the identity team revoke the relevant sessions? Can the service operate in a reduced mode? A failed exercise is useful evidence for an assigned improvement, not a compliance score.

NIST SP 800-61 Rev. 3 describes incident response as part of wider cybersecurity risk management under CSF 2.0. The scenario and choices here are illustrative; they are not a prescribed NIST checklist or legal advice.

The lesson from the fictional breach is straightforward: a team does not need perfect knowledge before acting, but it does need clear authority, recorded assumptions and a way to revisit its decision.

I’m Manfredi Marrocco. My primary editorial home is manfredimarrocco.blog.

EVE Online is a useful lens for this problem. Imagine two fleets supplied from different depots, but dependent on the same transit route. The assets look distributed; the ability to use them is concentrated. This is a hypothetical analogy, not a claim about a particular battle or the game's infrastructure.

For a production application, the equivalent route might be identity, DNS, encryption keys or the deployment pipeline. Counting replicas tells us little until we ask what both replicas need to become useful.

Start with a transaction, not an inventory

Consider a fictional order-processing service. It runs in two regions. Backups are copied regularly, and the dashboard reports successful backup jobs. A region outage is the scenario everyone rehearses.

Now change one assumption: the organisation's identity provider is unavailable. The standby region still exists, but the recovery operators cannot authenticate. The deployment system cannot obtain its credentials. The recovery instructions live behind the same sign-in page.

The application has redundancy. Its recovery path does not have enough independence for this scenario.

Start the analysis with a concrete outcome: an authorised operator can restore the service, and a test user can submit an order and retrieve its recorded status. This defines something observable. “The virtual machines started” is an intermediate event.

Draw the recovery path separately

The normal service path and the recovery path overlap, but they are not interchangeable. A running application may keep serving requests while an administrator is locked out. A replacement environment may need services that the running environment no longer calls.

For each recovery step, record four things:

• The resource required: identity, key material, DNS, network connectivity, source code, package registry or people.
• Its failure domain: which provider, account, region or administrative team controls it.
• The evidence needed: a successful restore, a tested access procedure or a confirmed dependency.
• The accountable owner: who can fix the gap and approve the remaining risk.

Do not publish account identifiers, recovery credentials or detailed access procedures in the public diagram. The public explanation can use synthetic labels; the operational version belongs in an appropriately protected location.

A small dependency register

A working register for the fictional service might contain these entries:

Operator authentication. Both regions use the same identity provider. Question: can an approved recovery operator obtain necessary access if that provider is unavailable?

Backup decryption. A copied backup depends on a key-management service. Question: can the recovery environment legitimately use the required key when the primary account is impaired?

Deployment. Infrastructure definitions are available, but packages come from a single registry. Question: can the tested release be reconstructed without downloading an unverified substitute?

Routing. The standby responds on a test address. Question: can users reach it through the actual service name, with valid certificates and the intended access controls?

Decision authority. The runbook names a primary owner. Question: who can authorise activation when that person is unavailable?

These are investigation prompts, not an instruction to duplicate every component. Independence costs money and introduces complexity. The point is to make concentration visible before accepting it.

Run one bounded exercise

Test in an isolated environment with synthetic data. Agree the scope, stop conditions and restoration procedure before starting. Never discover how to revoke production access by experimenting on a live identity system.

1. Define the business outcome and the recovery assumptions being tested.
2. Select a recent backup and a known application release.
3. Simulate one unavailable dependency inside the test scope.
4. Follow the documented recovery procedure, recording blockers and manual work.
5. Perform an end-to-end transaction and check the resulting data.
6. Restore the test environment to its agreed baseline and assign each finding an owner.

Capture the time to usable service, the recoverable data point, the dependencies that blocked progress and any security controls temporarily changed. A fast restore with broken authorisation is not a successful exercise.

A recovery time objective is a target. A measured exercise result is evidence for a specific scenario. Neither proves that every possible incident can be recovered within the same time.

Turn the result into a decision

Suppose the exercise restores the data but fails at operator authentication. “Backup successful” remains true; “recovery demonstrated” does not.

The next action could be a properly governed emergency-access design, a different runbook distribution method or an explicitly accepted limitation. Each option needs a responsible owner, a test and a review trigger. Emergency access must be protected and monitored; removing authentication is not a resilience strategy.

A useful review trigger is a change in the recovery path: a new identity provider, key hierarchy, registry, supplier or operating team. An old test may no longer describe the current system.

NIST SP 800-34 Rev. 1, written for US federal information systems, provides a useful reference for connecting business impact, resource requirements, interdependencies and recovery priorities. The example and exercise above are my illustrative design, not a mandated NIST test or a compliance assessment.

The lesson from our imagined fleets is practical: separate depots do not guarantee separate routes. Before buying another replica, trace what it takes to use the one you already have.

I’m Manfredi Marrocco. I explore cybersecurity, GRC and IT governance through technical examples and pop culture. My primary editorial home is manfredimarrocco.blog, and public demo code and documentation are on GitHub.