Define the decision the system will support
Name the task, intended users and consequences of an incorrect output. A document-finding assistant and a maintenance prediction tool have different information needs and review requirements. A precise use case makes it easier to judge whether the available data is relevant and whether the proposed system is worth testing.
Inspect representative information
Review completeness, version history, access permissions and how examples were collected. Ask whether the sample includes the conditions the tool will face in operation. More files do not automatically mean better evidence. Duplicated, obsolete or inconsistently labelled information may make an apparently large dataset less useful than a smaller, well-understood one.
Make uncertainty part of the pilot
Include difficult and ambiguous cases in the evaluation plan. Identify when the tool should ask for clarification, decline a conclusion or route the case to a reviewer. Record gaps that need better information rather than trying to solve every issue by changing the model. Agree who owns data quality as well as who builds the tool.
Bring it to the next conversation.
- Define the supported task and user.
- Check representative data and permissions.
- Plan a response to insufficient information.
Further reading: NIST AI Risk Management Framework.
From the Hexifyer team. Examples are illustrative and do not identify client projects. Project requirements and professional responsibilities are agreed for each engagement.