Terminology for Human Oversight Acts in Automated and Agentic Systems

1.1. Problem Statement

Automated and agentic systems increasingly act on behalf of people and organizations: they draft documents, move money, change configurations, and call other systems. Many deployments place a person somewhere in this process, and many records of the process note that a person was involved. Typical forms are an approval flag, a "reviewed" field, a signed confirmation, or a log entry saying that a named user pressed a button.¶

Such records usually do not say what the person did. The same flag can stand for at least four different acts: the person was shown the output; the person determined whether a named property of the output holds; the person chose what should happen next; or the person, using authority they hold, permitted an action. These acts establish different things. Being shown an output establishes nothing about it. Determining that a named property holds establishes that property and no other. Permitting an action establishes that the action was permitted, not that it was correct.¶

When the acts are not distinguished, consumers of the record (auditors, relying parties, and downstream systems and agents) cannot tell which of these things was established. Such a record is easily read as establishing more than it does, so that a confirmation click is treated as though it were a check. The weakness does not lie in any single act. It lies in a vocabulary that has one word for all of them.¶

Existing texts show the gap. Article 14 of the EU Artificial Intelligence Act [EU-AI-ACT] is headed "Human oversight" and describes what the persons to whom oversight is assigned are to be enabled to do; the definitions in Article 3 of that Regulation do not include the term. The NIST AI Risk Management Framework [NIST-AI-RMF] states in its Appendix C that "Human roles and responsibilities in decision making and overseeing AI systems need to be clearly defined and differentiated." An individual Internet-Draft, [I-D.schrock-human-authorization-binding], notes that agent-action record formats reserve a place for "the human authorization" and leave its semantics undefined.¶

This document provides a small set of terms with which such records, and the texts that specify them, can say which act took place and what its record establishes.¶

1.2. Scope

This document defines terminology. It does not:¶

• specify a protocol, message, or data format;¶
• state when human oversight is needed, or which kind of act a given situation calls for;¶
• specify how checks are to be made, how error detectability is to be determined, or how check tests are to be designed; it gives no values, thresholds, or reference data;¶
• interpret any law, regulation, or framework.¶

The terms are intended for authors of protocols, record formats, and policies, and for those who audit systems against them. Within the IETF, they are intended in particular for specifications that record human involvement in the actions of software agents, such as the user confirmation described in Section 10.7 of [I-D.ietf-wimse-aims], the audit records of [I-D.gilda-wimse-agent-audit-record], and the human-authorization evidence of [I-D.schrock-human-authorization-binding] (Section 4.6).¶

1.3. Conventions

This document does not use the requirement keywords of BCP 14. Where a sentence in the definitions reads as a condition (for example, that a check is over a named property), it states part of a definition: something that does not meet the condition is not an instance of the defined concept.¶

Each term is a noun. Each definition is a single phrase that can replace the term in running text; it names a broader concept and the characteristics that set the defined concept apart from other concepts under it, following the principles for the writing of definitions in [ISO704]. Notes and an example follow each definition. Notes add information; they do not change the definition. The corresponding verbs (observe, check, decide, and release) are used in running text with the same meanings.¶

In ordinary usage, a release (Section 2.8) is often called an "approval" or an "authorization". This document avoids both words for it. In [RFC4949], and in the OAuth-based work cited in Section 4.6, "authorization" denotes an approval granted to a system entity, or the process of granting it (Section 4.5); and "approval" is commonly used for records that do not state the kind of act (Section 2.11).¶

The words "evidence" and "relying party" are used in their ordinary senses, not in the senses defined for remote attestation in [RFC9334]. The word "verification" is used only in quotations and in the sense of the document cited where it occurs.¶

In this document, "system" means an automated or agentic system: software that produces outputs or takes actions with some degree of autonomy, including software agents that act on behalf of people or organizations. The "object" of an oversight act is the output, action, state, or class of actions of the system to which the act is directed.¶

2.1. Oversight Act

Definition: An act of a natural person that is directed at an output, action, state, or class of actions of a system and that is recorded, or is meant to be relied on, as human involvement in the operation of that system.¶

Note 1: This document distinguishes three kinds of oversight act by what the overseer does: observation (Section 2.3), check (Section 2.5), and decision (Section 2.7). It distinguishes one kind of decision further, by the authority within which it is made: release (Section 2.8). In this document, "the kinds of oversight act" refers to these four.¶

Note 2: The kind of an act depends on what the overseer did, not on the interface element used to record it. A button labeled "Approve" can record any of the four kinds, several of them at once, or none.¶

Note 3: A single interaction can contain more than one act, and each act can be recorded separately.¶

Example: A reviewer reads an agent's proposed database change, determines that it modifies only the intended table, and allows it to run within authority the reviewer holds. The interaction contains a check and a release; the release is also a decision.¶

2.2. Overseer

Definition: The natural person who performs a given oversight act.¶

Note 1: A system, a software agent, or an organization is not an overseer in the sense of this document, although an overseer may act on behalf of an organization.¶

Note 2: Different acts concerning the same object can have different overseers.¶

Example: The member of staff who releases a queued payment is the overseer of that act. A manager who later countersigns it is the overseer of a second act.¶

2.3. Observation

Definition: An oversight act in which the overseer is presented with the object and makes no check, decision, or release concerning it.¶

Note 1: Monitoring a dashboard, reading a log, watching an agent's activity stream, and having an output displayed on screen are observations. Whether the overseer perceived or understood the object is not part of the definition.¶

Note 2: A record of an observation shows at most that the object was presented to the overseer. It does not show what the overseer noticed.¶

Note 3: An observation may prompt a later act, such as a decision to interrupt the system. The later act is a separate oversight act.¶

Example: A reviewer keeps a live view of an agent's actions open during a session and takes no action. The session contains observations only.¶

2.4. Named Property

Definition: A property that the object of an oversight act may or may not have, stated before the outcome of that act is known and precisely enough that a person other than the overseer can tell what a determination of it establishes and what it does not.¶

Note 1: "Looks correct" and "approved" are not named properties. "Every total in table 2 equals the sum of its rows" and "the recipient named in the message is the recipient named in the request" are.¶

Note 2: A property that is stated, or changed, only after the outcome is known does not meet this definition for that act; see "Property substitution" in Section 5.¶

Example: For a generated purchase order, "the unit prices match the current price list" is a named property. A determination of it establishes nothing about the quantities ordered.¶

2.5. Check

Definition: An oversight act in which the overseer examines the object in order to determine whether a named property holds for it.¶

Note 1: A check is always over a named property (Section 2.4). An act in which a person looks at an object and forms a general impression, without a named property, is an observation, however carefully it is done.¶

Note 2: A check has an outcome: the property holds, the property does not hold, or the overseer could not determine which. The outcome concerns the checked property only. It establishes nothing about other properties of the same object.¶

Note 3: An overseer may use tools, including automated ones, in making a check. Where the overseer adopts a tool's result without determining the outcome, the overseer's act is an observation of that result; any determination made by the tool is not a check in the sense of this document.¶

Note 4: A check is a single act. The point in a process at which checks are to be made is a check step (Section 2.12).¶

Example: A reviewer determines whether every document identifier cited in a generated report resolves to an existing document. The outcome says nothing about whether the report's conclusions follow from those documents.¶

2.6. Checked Property

Definition: The named property over which a given check is made.¶

Note 1: Each check has exactly one checked property. An overseer who determines several properties of the same object makes several checks.¶

Example: In the example in Section 2.5, the checked property is "every document identifier cited in the report resolves to an existing document".¶

2.7. Decision

Definition: An oversight act in which the overseer selects, from the alternatives available at that point, the course of action to be taken with respect to the object.¶

Note 1: Alternatives include using, disregarding, overriding, or reversing an output; letting an operation continue; interrupting it; and referring the matter to another person. A decision to let an action be carried out is a release only where it is made within standing authority (Section 2.8).¶

Note 2: A decision can be made without any check. A record of a decision does not imply that a check was made.¶

Note 3: Responsibility for a decision remains with the overseer, or with the person or organization on whose behalf the overseer decides. Recording a decision does not transfer that responsibility to the system.¶

Note 4: Where a record format assigns the choice of a course of action to a relying party, a decision in the sense of this document corresponds to that choice when the overseer is, or acts for, the relying party.¶

Example: After reading a proposed refund, a member of staff chooses to hold it for a colleague instead of releasing or rejecting it.¶

2.8. Release

Definition: A decision in which the overseer, acting within standing authority they hold, permits a specified action or class of actions to be carried out.¶

Note 1: The standing authority (Section 2.9) exists before the act. A release exercises authority; it cannot create the authority it exercises. A decision to permit an action, made by a person who holds no standing authority covering it, is a decision but not a release, whatever its record calls it.¶

Note 2: A release is bounded by what it specifies. Permission for one action does not extend to a different action, or to the same action with different parameters, unless the release specifies a class of actions that includes it.¶

Note 3: Releasing is not checking. A release establishes that an action was permitted. It does not establish that the action, its inputs, or its effects are correct.¶

Note 4: The reasons for not calling this act an "approval" or an "authorization" are given in Section 1.3 and Section 4.5.¶

Example: A team lead who holds signing authority up to a stated amount permits an agent to submit one specific purchase order within that amount. The team lead's act is a release.¶

2.9. Standing Authority

Definition: Authority to permit actions of a class that a person holds before a given oversight act and that was conferred by a source other than that act.¶

Note 1: Typical sources are an employment role, a documented delegation, a contract, law, or the person's authority over their own affairs.¶

Note 2: Standing authority can itself result from delegation. [PROV-O] describes delegation as "the assignment of authority and responsibility to an agent (by itself or by another agent) to carry out a specific activity as a delegate or representative".¶

Note 3: Other documents use "mandate" for a related but different concept. In [I-D.yossif-agent-mandate-problem], a mandate is "The set of constraints the Principal authorizes at T0, expressed as data the Principal signs"; it is created by the principal's act. In the terms of this document, that act is a release over a class of actions, made within the principal's standing authority. This document does not use the word "mandate" for standing authority, because in that usage a mandate results from the act instead of preceding it.¶

Example: An organization's approval policy that allows members of its finance team to release payments below a stated amount confers standing authority on each of them.¶

2.10. Oversight Record

Definition: A record of a single oversight act that is issued by, or attributable to, the overseer and that identifies the overseer, the kind of act, the object, and the time of the act, and, for a check, the checked property and the outcome.¶

Note 1: For a release, a record that also identifies the standing authority relied on and the action or class of actions permitted allows a consumer to see whether the act was within that authority.¶

Note 2: A record that the system writes about the overseer, without any means of attributing it to the overseer, is not an oversight record in this sense. It states the system's account of the act.¶

Note 3: This document does not define a format. The same record can be carried in several existing formats; see Section 4.¶

Example: A record stating that a named reviewer, at a given time, checked a generated report for the property "every cited identifier resolves", with the outcome "holds", is an oversight record. A second record, stating that the same reviewer then released the report for publication under a named publication policy, is another.¶

2.11. Undifferentiated Approval

Definition: A record of human involvement with a system that does not state which kind of oversight act was performed.¶

Note 1: Approval flags, "reviewed" fields, and confirmation events are usually undifferentiated approvals.¶

Note 2: Without further information, an undifferentiated approval shows only that an interaction was recorded. The interface element that produced it can record any kind of act, several, or none (Section 2.1, Note 2); see also Section 3.¶

Example: A log entry "approved_by: user-17" is an undifferentiated approval.¶

2.12. Check Step

Definition: A point in a process at which checks over a stated named property are to be made, by stated persons under stated conditions, on the objects that reach that point.¶

Note 1: A check step is a procedure; a check is a single act made at it. A check step can produce records without any check being made, for example where a review screen is closed without the object being examined.¶

Note 2: Error detectability (Section 2.13), the property of being fail-open (Section 2.14), and check tests (Section 2.15) concern check steps, not single checks.¶

Example: In a contract workflow, the point at which a member of the legal team examines each generated contract for the property "all required clauses are present" is a check step.¶

2.13. Error Detectability

Definition: A property of a check step that expresses how reliably checks at that step, as made by the persons and under the conditions stated for it, distinguish objects for which its named property does not hold from objects for which it holds, relative to a stated reference population of objects.¶

Note 1: An error, in this entry, is a respect in which an object does not have the named property of the check step.¶

Note 2: The reference population is the set of objects, and of errors in them, on which the checks were made. The same check step can reveal most errors of a kind in one population and few in another.¶

Note 3: Error detectability is distinct from the proportion of objects that a check step flags. A check step that flags every object reveals every error and distinguishes nothing.¶

Note 4: Errors of omission, where something that is required is absent from the object, leave nothing in the object to notice. A statement of error detectability that covers only errors present in the object says nothing about them.¶

Note 5: Where it needs to be distinguished from corresponding properties of automated tools, this property can be called "human error detectability".¶

Note 6: This document specifies no method for determining error detectability, and no values or thresholds.¶

Example: "Reviewers catch most errors" does not state error detectability: it names no check step, no named property, and no reference population.¶

2.14. Fail-Open Check Step

Definition: A check step at which the failure of a check, whether because the check was not made or because it could not tell an object for which the named property holds from one for which it does not, produces the same record as a check in which the property was found to hold.¶

Note 1: A human check step is fail-open wherever the record is written independently of whether the check was made. In that case, a check that was not in fact made still produces the record "checked", and ordinary operation gives no sign of the failure.¶

Note 2: A check step that is not fail-open is fail-closed: the failure of a check produces a record that differs from the record of a check in which the property was found to hold, for example no record, or the outcome "could not determine".¶

Note 3: [RFC4949] notes that "fail-safe" has two opposing meanings. This document uses "fail-open" and "fail-closed" only with the meanings given here. [I-D.schrock-human-authorization-binding] uses "fail-closed absence" for a related but different rule: a relying party is to treat an absent human-authorization binding as insufficient evidence.¶

Example: A review step for the property "all required clauses are present", in which the field "checked" is set whenever the review screen is closed, is a fail-open check step: closing the screen without reading produces the same record as reading and finding every required clause.¶

2.15. Check Test

Definition: A test of a check step in which a check at that step is given an object for which the step's named property is known not to hold, in order to establish whether the check reports that the property does not hold.¶

Note 1: Testing a detection process with inputs known to contain faults is long established, for example in mutation testing of software [DeMillo1978] and, for human inspectors, in threat image projection in aviation security screening [Hofer2005]. A recent statement of the principle for fail-open properties in general is [AIKR-FAILOPEN]. This document uses the term for check steps in the oversight of automated and agentic systems.¶

Note 2: A check test tests the check step, not the object.¶

Note 3: Check tests matter most for fail-open check steps, whose failures do not otherwise show in their records.¶

Note 4: This document does not specify how such objects are made, how many are used, or what result is acceptable.¶

Example: A contract that lacks a required clause, given to a check step for the property "all required clauses are present", is the object of a check test.¶

4.1. W3C PROV-O

PROV-O [PROV-O] describes an activity as "something that occurs over a period of time and acts upon or with entities", an agent as "something that bears some form of responsibility for an activity taking place, for the existence of an entity, or for another agent's activity", and an association as "an assignment of responsibility to an agent for an activity".¶

An oversight act can be described as a prov:Activity associated with the overseer, a prov:Person, through prov:wasAssociatedWith; the activity uses the object (prov:used) and generates an oversight record, a prov:Entity attributed to the overseer (prov:wasAttributedTo). Where the standing authority relied on in a release results from delegation, it can be described with PROV-O delegation (prov:actedOnBehalfOf, prov:Delegation).¶

PROV-O records that a person was responsible for an activity. It does not distinguish kinds of oversight act, and it has no terms for a named property or for the outcome of a check. These are what the terms in Section 2 add.¶

4.2. W3C Verifiable Credentials Data Model 2.0

An oversight record can be expressed as a claim in a verifiable credential [VC-DATA-MODEL-2.0] whose issuer is the overseer. The Data Model distinguishes verification from validation and states: "Verification of a credential does not imply evaluation of the truth of claims encoded in the credential."¶

The same separation applies to oversight records. Verifying an oversight record carried as a credential establishes that the overseer issued it. What the record then establishes depends on the kind of act it records (Section 3): a verified record of an observation is still evidence of an observation only. Validation, which the Data Model describes as "The assurance that a claim from a specific issuer satisfies the business requirements of a verifier for a particular use", is where a verifier would look for the kind of act, the checked property, and, for a release, the standing authority.¶

4.3. EU Artificial Intelligence Act, Article 14

Article 14 of Regulation (EU) 2024/1689 [EU-AI-ACT] is headed "Human oversight". Article 14(4) lists what the natural persons to whom human oversight is assigned are to be enabled to do. Several items on that list correspond to kinds of act defined here:¶

• to "duly monitor its operation" (Article 14(4)(a)) corresponds to observation, and to checks where monitoring is against a named property;¶
• to "decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output of the high-risk AI system" (Article 14(4)(d)) and to "intervene in the operation of the high-risk AI system or interrupt the system" (Article 14(4)(e)) are decisions in the sense of Section 2.7;¶
• Article 14(4)(b) refers to the tendency of "automatically relying or over-relying on the output" (automation bias), which is one reason why checks fail; where a check step is fail-open, such failures do not show in its records (Section 5).¶

Article 14(5) provides, for certain systems, that an identification is to be "separately verified and confirmed by at least two natural persons". The terms of this document can be used to state which kind of act each of these persons performed and, for a check, over which named property.¶

These correspondences are terminological. This document does not interpret the Regulation or state what it requires.¶

4.4. NIST AI Risk Management Framework

The NIST AI Risk Management Framework [NIST-AI-RMF] lists under MAP 3.5: "Processes for human oversight are defined, assessed, and documented in accordance with organizational policies from the GOVERN function." Its Appendix C states that "Human roles and responsibilities in decision making and overseeing AI systems need to be clearly defined and differentiated."¶

The terms in this document can be used to state which kinds of oversight act a process contains and what their records establish. Error detectability and check tests name what an assessment of whether a check step functions would report. The Framework does not use these terms, and this document does not describe how the Framework is to be applied.¶

4.5. Internet Security Glossary (RFC 4949)

[RFC4949] defines "authorization" as "An approval that is granted to a system entity to access a system resource" and as "A process for granting approval to a system entity to access a system resource". In both senses the system entity is the party that receives the authorization; who grants it is not part of the definition. A release in this document is made by a natural person acting within standing authority, and it permits an action of a system rather than only access to a resource.¶

[RFC4949] also records, as a definition of non-Internet origin that it does not recommend for Internet documents, a sense taken from the SET payment specifications: "The process by which a properly appointed person or persons grants permission to perform some action on behalf of an organization." A release is close to that sense. This document uses a different word so that its term is not read in the recommended sense above.¶

[RFC4949] defines "verification", in its first sense, which is marked as applying in the context of authentication, as "The process of examining information to establish the truth of a claimed fact or value". "Check" is close to that sense; it differs in that a check is an act of a person, is always relative to a named property, and can have the outcome "could not determine".¶

"Accountability", which [RFC4949] defines as the property of a system or system resource "that ensures that the actions of a system entity may be traced uniquely to that entity", is served by records of human involvement only to the extent that they identify the overseer and the kind of act.¶

4.6. IETF Work on Agent Authorization and Human Involvement

Several Internet-Drafts address human involvement in the actions of software agents. The terms in this document can be used with them as follows.¶

• Section 10.7 of [I-D.ietf-wimse-aims] states that user confirmation solicited by an interactive agent framework does not by itself constitute authorization and is to be bound to a verifiable authorization grant issued by the authorization server. In the terms of this document, a record of such a confirmation is an undifferentiated approval unless it states which kind of oversight act the user performed. The authorization grant to which it is bound is issued by the authorization server and is not itself an oversight act.¶
• [I-D.schrock-human-authorization-binding] defines how records bind named-human authorization evidence. The kinds of act defined here can state which act such evidence records: a place reserved for "the human authorization" can hold a record of a check, a decision, or a release, and these establish different things (Section 3).¶
• [I-D.yossif-agent-mandate-problem] characterizes the gap between an authorized agent and an authorized action. It uses "mandate" for the set of constraints that a principal authorizes and signs (Section 2.9, Note 3). In the terms of this document, the principal's act is a release over a class of actions. The definition of release in Section 2.8, which requires a specified action or class of actions and standing authority, names the human act whose record would bound an action.¶
• [I-D.rosenberg-aiproto-cheq] describes human confirmation of decisions and actions proposed by agents before they are acted upon. In the terms of this document, such a confirmation can be a decision or a release; it is a check only where a named property is stated.¶
• [I-D.sato-soos-hem] specifies what happens when an agent session requires human judgment before it may continue, and defines human decision types. It also addresses approval fatigue (Section 5).¶
• [I-D.gilda-wimse-agent-audit-record] defines a record format for AI agent authorization decisions. Oversight records are records of human acts; they can be referenced from, or kept alongside, such audit records.¶