A convincing answer is not proof of accuracy
NIST AI 600-1 identifies confabulation as a risk of generative systems: they can confidently produce erroneous or false content. A grammatically correct answer is therefore not evidence that the information is accurate. [1, section 2.2]
Checking information
NIST recommends fact-checking techniques and evaluating information accuracy, especially when information comes from multiple or unknown sources. A business should establish which content needs review by a responsible person before affecting a customer or business decision. [1, MP-2.3-003]
Sensitive data
NIST identifies disclosure and unauthorized use of personal or sensitive data as risks. Keeping an access key on the server does not by itself prevent data in a request from being sent to the model provider. Assessment needs to consider request content, provider terms, access and data handling. [1, sections 2.4 and 3]
Testing before wider use
NIST addresses risk assessment and management throughout a system’s lifecycle. A limited trial with defined inputs, an accuracy criterion and an error log is one way of applying this approach. The document does not guarantee time savings or financial benefits for a particular business. [1, section 3]