| Internet-Draft | Anomaly Detection Evaluation | September 2026 |
| Anjum | Expires 11 March 2027 | [Page] |
The Network Management Operations (NMOP) working group has adopted documents describing an architecture, an operational lifecycle, and a semantics for network anomaly detection. Those documents direct implementers to minimize false positives and false negatives, but do not define how the accuracy of an anomaly detection implementation is to be measured, compared, or tracked over time. This document describes an evaluation methodology for anomaly detectors operating on network and infrastructure telemetry, whether the detector is rule-based, statistical, or machine-learning-based: the metrics to report and their known failure modes, a benchmarking procedure based on controlled fault injection and replay, ground-truth labeling and scoring across multiple telemetry signals, and the properties a benchmark dataset needs to support reproducible, comparable evaluation. The methodology is informational and complements the adopted NMOP anomaly-detection documents.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 11 March 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
The NMOP working group has adopted three documents that together describe how network anomaly detection is to be architected, operated, and annotated: an architecture for Service Disruption Detection [I-D.ietf-nmop-network-anomaly-architecture], an operational lifecycle for iterative refinement of detection systems [I-D.ietf-nmop-network-anomaly-lifecycle], and a semantics for annotating detected anomalies [I-D.ietf-nmop-network-anomaly-semantics].¶
All three documents treat detection accuracy as a central objective. The architecture requires that "the operator can trust that a targeted Service Disruption will be detected (no false negatives), while not spamming the operator with Alarms that do not reflect an actual issue within the network (false positives)" (Section 3.5 of [I-D.ietf-nmop-network-anomaly-architecture]). The lifecycle states that "Providers of solutions for network anomaly detection should aim at increasing accuracy, by minimizing False Positives and False Negatives" (Section 1 of [I-D.ietf-nmop-network-anomaly-lifecycle]), and its refinement stage concludes by "'replaying' the network anomaly and checking if there is any measurable improvement" (Section 5.3 of [I-D.ietf-nmop-network-anomaly-lifecycle]).¶
None of the three documents defines how that accuracy or that improvement is to be measured. No evaluation metric, no formula, and no comparison procedure is specified. The consequence is visible in the architecture document's own deployment report, which counts 20 monitored service disruptions against 52 false positives since December 2022 (Section 4.1 of [I-D.ietf-nmop-network-anomaly-architecture]) but has no framework in which to express whether that ratio is acceptable, improving, or comparable to an alternative detector.¶
This document describes such a companion, an evaluation methodology for anomaly detectors operating on network and infrastructure telemetry: which metrics to report and the known failure modes of each (Section 3), a benchmarking procedure based on controlled fault injection and replay (Section 4), including how ground truth is labeled and scored when a detector operates over more than one telemetry signal (Section 4.3.1), the properties a benchmark dataset needs for reproducible comparison (Section 5), and the minimum contents of an evaluation report so that results from different detectors can be compared at all (Section 6). The methodology is grounded in a published peer-reviewed evaluation of eight machine-learning detection methods on unified telemetry [ANJUM-EVAL], summarized as a worked example in Appendix A.¶
This document is designed to be used inside the lifecycle defined by [I-D.ietf-nmop-network-anomaly-lifecycle], not alongside it:¶
The lifecycle's Validation stage decides, per detected Symptom, whether a real problem was signaled. Accumulated validation outcomes are exactly the labels this document's metrics consume (Section 3).¶
The lifecycle's Refinement stage requires a check for "measurable improvement" after replaying an anomaly. Section 4 turns that check into a defined procedure with defined measures.¶
The architecture's Replaying capability (Section 3.8 of [I-D.ietf-nmop-network-anomaly-architecture]), which preserves the data that led to an Alarm so that the detection algorithms can be rerun on the same input, is the natural execution environment for the replay-based evaluation in Section 4.¶
The semantics document defines the vocabulary in which a validated anomaly is recorded: a Symptom bound to one network plane, with the topology context (for example interface and peer) it concerns. Labels expressed in that vocabulary are the multi-signal ground truth that Section 4.3.1 scores against.¶
One definitional difference must be reconciled explicitly. [I-D.ietf-nmop-network-anomaly-lifecycle] defines a False Positive as "a detected anomaly which has been identified during the postmortem to be not anomalous", an operational, postmortem judgment. Dataset-based evaluation instead defines false positives against ground-truth labels fixed before evaluation. Both definitions are useful and they converge as validated postmortem outcomes accumulate into a labeled corpus; Section 2 defines the ground-truth variants used in this document, and Section 5 describes how operational validation records become benchmark labels.¶
This document specifies methodology, not protocol. It defines no new data models, YANG modules, or wire formats.¶
It applies to any detector that consumes telemetry signals (metrics, logs, traces, flow records, routing events, or their combination) and emits, per detection window or per event, a decision or a score. How the detector was built does not matter: rule-based detectors such as static thresholds and expert rules, statistical detectors such as seasonal baselines and change-point tests, and machine-learning-based detectors are all in scope. The metrics of Section 3, the per-Symptom and per-episode scoring of Section 4, and the reporting requirements of Section 6 are defined on detector output and ground truth, not on the detector's internals, so they apply equally across these classes. The adaptations needed for detectors that emit a binary decision without a score are given in Section 3.1, and the reporting requirements that depend on a score are marked as conditional in Section 6.¶
The metric guidance in Section 3 comes from failure modes observed when evaluating machine-learning-based detectors, but none of it is specific to that class. The F1 floor of Section 3.2 is a property of the evaluated set, not of the detector: a poorly set static threshold reaches it as readily as a collapsed learned model.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
This document uses the following terms as defined elsewhere and does not redefine them: Problem, Symptom, and Alarm from [RFC9940]; Service Disruption Detection System (SDDS) from [I-D.ietf-nmop-network-anomaly-architecture]; and False Positive and False Negative in their operational sense from [I-D.ietf-nmop-network-anomaly-lifecycle] (see Section 1.1 for the reconciliation with the ground-truth sense used here).¶
The following evaluation-specific terms are defined for this document:¶
Any component that consumes one or more telemetry signals and emits, per detection window or per event, either a decision (anomalous or non-anomalous) or a score from which such a decision is derived by a threshold. The term covers rule-based, statistical, and machine-learning-based implementations alike; nothing in this document depends on which.¶
One telemetry stream over which a detector produces its own sequence of decisions or scores, for example metrics, logs, or traces from a set of services, or flow records from the forwarding plane and routing events from the control plane of a network. A detector may operate over several signals; its output is then scored per signal and per event (Section 4.3.1).¶
The authoritative record of which intervals or events in a telemetry corpus are anomalous, fixed before an evaluation begins. Ground truth may originate from a controlled injection schedule (Section 4) or from accumulated, validated postmortem outcomes.¶
The per-interval or per-event value assigned by the ground truth (anomalous or non-anomalous), against which detector output is scored.¶
One anomalous condition as recorded by the ground truth, with one onset and one end, together with the signals in which it is observable and the dimension values (for example interface or peer) it covers. Under injection, one fault-injection run is one event; under postmortem labeling, one validated annotation and its list of Symptoms is one event. An episode (Section 3.3) is an event viewed on a single signal. An Event in this sense corresponds to a Problem in [RFC9940] terms, qualified by the requirement that the ground truth, not a detector, is what attributes the Symptoms to it.¶
The exact collection of intervals or events over which metrics are computed. Threshold-dependent metrics computed over different evaluated sets are not comparable, because they depend on the prevalence of the set (Section 3.2). Threshold-independent metrics are comparable across sets to the extent the sets are drawn from comparable fault distributions (Section 3.1).¶
The fraction of the evaluated set that is labeled anomalous.¶
The fixed time interval into which telemetry is aggregated before a detection decision is made for that interval.¶
The elapsed time between the onset of an anomalous condition per the ground truth and the first detection decision that flags it.¶
Detection quality cannot be summarized in a single number. This section defines a minimum metric set and, for each threshold-dependent metric, the conditions under which it misleads. The guidance here is grounded in observed failure modes from a controlled evaluation of eight detection methods [ANJUM-EVAL]; the worked example in Appendix A shows each failure mode with real values.¶
An anomaly detector typically produces a continuous score per detection window, converted to a binary decision by a threshold. Metrics computed over the score distribution, independent of any threshold, are the soundest basis for comparing detectors:¶
AUC-ROC (area under the receiver operating characteristic curve) SHOULD be the metric of record for ranking and acceptance decisions. It is threshold-independent and insensitive to prevalence, so a change in the anomaly rate of the evaluated set does not by itself move it. An AUC-ROC near 0.5 indicates a detector performing no better than chance, regardless of any other metric reported for it.¶
PR-AUC (area under the precision-recall curve) SHOULD be reported alongside AUC-ROC. It focuses on the anomalous class and is the more informative of the two when prevalence is low, the common operational condition [SAITO2015].¶
The two answer different questions. AUC-ROC ranks, because prevalence insensitivity is what makes detectors scored on sets with different anomaly rates comparable at all. PR-AUC reflects the false-positive burden an operator carries at low prevalence. Prevalence insensitivity is not corpus independence: sets differ in fault distribution as well as in anomaly rate, and Appendix A records one detector scoring far apart on two corpora. Comparison across corpora is bounded by Section 5.¶
An AUC-ROC well below 0.5 is a different condition from one near it. Near 0.5 the score carries no ranking information; well below 0.5 it carries ranking information with the sign reversed, placing anomalous windows below normal ones. The second is a defect in the score convention, not an absence of signal. An evaluation that encounters it SHOULD report the observed value, state that the ordering is inverted, and re-evaluate with the sign corrected before judging the detector's discriminative power.¶
A detector that emits only a binary decision, with no score, has a single operating point, and AUC-ROC and PR-AUC cannot be computed for it. Such a detector MUST be reported with the threshold-dependent metrics of Section 3.2 against the prevalence baseline defined there, and SHOULD expose the quantity it thresholds internally (for example, the deviation from a static limit, or the test statistic of a change-point test) as a score, so that the threshold-independent metrics can be computed and the detector compared with scoring detectors on the same footing.¶
Precision, recall, and their harmonic mean F1 describe a single operating point and remain the terms in which operators reason ("how many alarms are real", "how many disruptions are missed"). They MUST NOT, however, be used in isolation for acceptance or comparison, for two reasons.¶
F1 has a nontrivial floor. A degenerate detector that flags every window achieves recall 1 and F1 = 2p/(1+p), where p is the prevalence of the evaluated set. At a prevalence of 0.4 this floor is approximately 0.57; a reported F1 of 0.6 at that prevalence is therefore nearly indistinguishable from a detector that discriminates nothing. The same degenerate detector falls to F1 of roughly 0.10 at 5% prevalence, which is why F1 values obtained at different prevalences are not comparable at all. An evaluation MUST report the prevalence of its evaluated set and the F1 of the trivial flag-everything detector on that set as a baseline. Where a score is available, an evaluation SHOULD verify AUC-ROC exceeds 0.5 before any F1 value is credited. For a detector that emits binary decisions only no such check is possible, and the comparison below takes its place.¶
For a scoreless detector, compare the fraction of windows it flags with the prevalence of the evaluated set. A flagged fraction near 1 is the flag-everything detector in all but name, whatever its F1, and a flagged fraction well above prevalence buys recall with false positives that F1 at that prevalence hides. An evaluation of a scoreless detector MUST report its flagged fraction alongside precision, recall, and F1.¶
Threshold selection can additionally mask detector quality in either direction. If the threshold is tuned on labeled validation data, reported F1 reflects an optimistic operating point unavailable in production, where thresholds are typically set from the score distribution of normal traffic alone (for example, a high percentile of normal-only scores). An evaluation MUST state how thresholds were selected, and SHOULD report the sensitivity of its threshold-dependent metrics to that choice.¶
Window-level metrics measure classification quality; operators additionally need to know whether whole incidents are caught and how quickly. An episode is one contiguous anomalous condition per the ground truth on one signal (for a benchmark, one fault-injection run as seen on that signal). An episode counts as detected if at least one window within its labeled span is flagged. Episode-level detection rate is a recall-oriented measure and is trivially satisfied by the flag-everything detector, so it MUST be reported together with window-level precision, and with AUC-ROC where a score is available, never alone.¶
The episode-level rate MUST NOT be used to modify the window-level metrics. Windows of a detected episode that the detector did not flag remain false negatives, and an evaluation MUST NOT relabel them as true positives before computing window-level precision, recall, or F1. That practice, known as point adjustment, inflates window-level scores far enough that a random score can outrank a discriminating detector [KIM2022].¶
Range-aware threshold-independent measures [PAPARRIZOS2022] are an alternative to two-level reporting where a detector is scored against anomalous ranges rather than isolated windows. This document does not require them; an evaluation MAY report them in addition.¶
Where onset timestamps are available, detection latency (Section 2) SHOULD be reported as a distribution, not only a mean; Section 4.4 explains why the distribution, rather than its mean, is what predicts behavior on a live network. When a detector operates over several signals, the event-level counterpart of the episode-level rate is defined in Section 4.3.1.¶
The detection window is an experimental parameter with direct effect on results: aggregation over a longer window smooths transient faults, a shorter one raises noise. Conclusions that hold only at one window size are fragile: in a two-fold, single-signal sub-study of the evaluation underlying this document, one detector scored far apart across 10, 30 and 60-second windows (Appendix A records that sub-study's limits). An evaluation MUST state its primary window size and its rationale, and SHOULD re-run its headline comparison at one or more neighboring window sizes, reporting whether the qualitative conclusions are stable.¶
This section describes a repeatable procedure for producing the ground truth and the evaluated set that the metrics in Section 3 require. The procedure is based on controlled fault injection into an instrumented environment; it is the offline complement to the operational Validation stage of [I-D.ietf-nmop-network-anomaly-lifecycle].¶
The environment SHOULD be a representative multi-component system emitting the telemetry signal types the detector under test consumes, unified through a common collection pipeline so that metrics, logs, and traces share a timeline. Two instrumentation patterns exist in practice, and a benchmark gains external validity by covering both: components instrumented natively at development time, and components instrumented after the fact through operator- or agent-based retrofit. A baseline period of normal operation MUST be captured before any fault is injected; it supplies training data for detectors that learn from normal behavior only, the reference against which rule-based and statistical detectors set their limits, and the score distribution from which production-style thresholds are derived (Section 3).¶
Faults MUST be injected under an explicit, recorded schedule; the schedule is the ground truth. A useful minimum taxonomy, drawn from [ANJUM-EVAL], covers: added network latency at graded severities, injected request errors, CPU saturation, memory saturation, and compound cascades that combine a fault on one component with resource stress on a dependent component. Injection targets SHOULD be chosen by architectural position (entry point, mid-chain dependency, backend) rather than convenience, and each scenario SHOULD be repeated multiple times so that dispersion across repetitions can be reported. Each run SHOULD include a defined active-fault phase followed by a cooldown phase long enough for the system to return to nominal behavior.¶
Labels derive from the injection schedule: windows within an active-fault phase are anomalous. Cooldown windows require an explicit decision. The system is recovering but no fault is active; labeling them anomalous inflates prevalence with ambiguous intervals, while silently including them as normal penalizes detectors that correctly flag residual disturbance. The evaluation MUST state how cooldown windows were treated, and SHOULD report its headline results with cooldown windows excluded from the evaluated set, with the resulting prevalence disclosed.¶
Where a detector is fitted, and its evaluation data is therefore partitioned for training, validation, and testing, the partition MUST be by episode (whole runs), never by random sampling of windows; windows from the same run are temporally correlated, and random partitions leak that correlation into inflated results.¶
Operational deployments accumulate an equivalent ground truth over time: each postmortem-validated outcome under the lifecycle of [I-D.ietf-nmop-network-anomaly-lifecycle] is a label. Section 5 describes how such records become benchmark datasets.¶
A detector, or a set of detectors, commonly operates over more than one signal. In the environment of Appendix A the signals are metrics, logs, and traces from the same services. In an operator network they may be flow records from the forwarding plane (for example IPFIX) and routing events from the control plane (for example BGP), each carrying dimensions such as interface and peer, and an event on one plane may or may not show on another. The labels of Section 4.3 are per signal and per window. This section defines how labels and scores are assigned at two levels, the individual signal and the aggregated event, and states the ambiguous cases explicitly. An evaluation over more than one signal MUST report both levels.¶
Labeling at the signal level. Each signal carries its own label sequence over its own windows. Under injection, the schedule fixes the active span of each event; under postmortem labeling, the recorded Symptoms fix it. In the vocabulary of [I-D.ietf-nmop-network-anomaly-semantics], each Symptom is bound to one network plane (Section 3 of that document) and to the topology context it concerns (Section 4.3 of that document), so a Symptom labels the signal of its plane and the dimension values it names, and nothing else.¶
Labeling at the event level. An event (Section 2) is the set of Symptoms, across signals and dimension values, that the ground truth attributes to one cause, with one onset and one end. Under injection, one run is one event. Under postmortem labeling, one validated annotation in the label store of [I-D.ietf-nmop-network-anomaly-lifecycle] (Section 6 of that document), with the list of Symptoms it names, is one event. The ground truth MUST record, for every event, the signals in which it is observable and the dimension values it covers.¶
Scoring at the signal level. Window-level metrics (Section 3.1, Section 3.2) are computed per signal, on that signal's windows against that signal's labels, and reported per signal. They SHOULD NOT be pooled across signals into one evaluated set: prevalence and window counts differ by signal (in Appendix A, prevalence ranged from 0.27 to 0.41 by signal), and pooling changes the evaluated set in the way Section 3.2 warns against. An evaluation that pools anyway MUST report the per-signal prevalences and window counts alongside the pooled figure, so that a reader can see what the pooled set is composed of.¶
Scoring at the event level. An event counts as detected if at least one signal in which the event is labeled observable flags at least one window inside the event's span on a dimension value the event covers. The event-level detection rate is the multi-signal counterpart of the episode-level rate of Section 3.3 and inherits its rules: it is recall-oriented, and MUST be reported together with the signal-level precision, and the signal-level AUC-ROC where a score is available, that it summarizes, never alone. It MUST NOT be used to relabel signal-level windows, for the reason given in Section 3.3. A flagged window that falls inside no event's span on its signal is a false positive at both levels.¶
Two labeling treatments are admissible, and an evaluation MUST state which it used and MUST NOT mix them within one evaluated set.¶
Schedule labeling: every signal is labeled anomalous for every window in the event's span, whether or not the event is observable in that signal. A miss on a signal the event does not reach is then a signal-level false negative. This treatment measures whether each signal's detector catches each class of event, and its informative output is per-event-class recall reported per signal. The evaluation in Appendix A used it.¶
Visibility labeling: only the signals in which the ground truth records the event as observable are labeled anomalous for its span. A flag on an unlabeled signal during the event is a signal-level false positive, and a miss there is a true negative. Postmortem labeling under [I-D.ietf-nmop-network-anomaly-semantics] yields this treatment by construction, since a Symptom is recorded only where something was observed.¶
Under either treatment the event is detected at the event level if any signal in which it is labeled observable flags it. The two treatments give different signal-level prevalences for the same corpus, so results obtained under one are not comparable with results obtained under the other.¶
An event may cover several values of one dimension (several interfaces on one node, several peers of one router) or several dimensions at once (an interface and a BGP peer). The ground truth MUST record every dimension value the event covers, in the terms of the semantics document's service topology model where it applies. A flagged window during the event on a dimension value the event does not cover (an unaffected interface, an unrelated peer) is a false positive at both levels, not a detection of the event. When the detector reports at a coarser dimension than the label (per node when the label is per interface), the evaluation MUST state its matching rule; the default is that a coarse detection counts as detecting every labeled fine-grained value it contains, and the report SHOULD then also give the fraction of events whose labeled dimension values were named exactly, since localization, not only detection, is what the operator acts on.¶
The rules in this section are fitted to the two ground-truth shapes available to the author: the injection schedule of Appendix A, and the Symptom-and-topology vocabulary of [I-D.ietf-nmop-network-anomaly-semantics]. Input from operators holding labeled multi-plane events, for example flow records and routing events across the forwarding and control planes with interface and peer dimensions, is sought so that a future revision can include a worked instance in that vocabulary.¶
Section 3.8 of [I-D.ietf-nmop-network-anomaly-architecture] calls for an SDDS to preserve the data that led to an Alarm so that the detection algorithms can be rerun on the same input. That capability turns the lifecycle's "measurable improvement" check into a defined measurement: hold the evaluated set fixed, rerun the modified detector on the identical preserved input, and compare the metrics of Section 3 before and after the change. A comparison in which the evaluated set itself changed between runs is not a measurement of the detector, and reporting it as an improvement misstates what was measured.¶
Live networks have no injection schedule. Operator triage records true positives and false positives as Alarms are validated, but a false negative leaves no record until its effect is reported by a user, so the labeled set of a live network is incomplete on exactly the side that matters most. This document therefore treats evaluation for a live network as replay against labeled history, in two steps that depend on the lifecycle and semantics documents.¶
Acceptance evidence for a detector intended for live traffic MUST include metrics obtained by replay against the labeled history available: a benchmark dataset (Section 5), the operator's own preserved Alarm data with validated outcomes, or both. Metrics accumulated from live triage alone do not constitute such evidence, because their false-negative count is a lower bound.¶
The labeled set grows as postmortem labeling proceeds. [I-D.ietf-nmop-network-anomaly-semantics] defines the vocabulary in which that labeling is expressed, a Symptom per network plane with its topology context (Sections 3 and 4 of that document), and the label store of [I-D.ietf-nmop-network-anomaly-lifecycle] (Section 6 of that document) persists each annotation and its versions. Every validated annotation, including a user-reported disruption that no detector flagged, adds labels to the history. A detector SHOULD be re-evaluated by replay whenever the labeled set has grown materially, and the report MUST state the size and date range of the labeled history used, so that two evaluations of the same detector are read against the labeled sets they were computed on. A re-evaluation on a grown labeled set is a new measurement, not a measurement of improvement; to measure the effect of a detector change, both versions are run on the same, current labeled set.¶
Detection latency matters more in the live case than in a benchmark. In a benchmark a late true positive still counts as detected within its episode. In a live network a true positive that arrives after a user has reported the disruption is, in operational terms, a false negative, because the user report and not the detector triggered triage. An evaluation intended to predict live behavior SHOULD therefore report detection latency as a distribution (Section 3.3), and SHOULD state the fraction of episodes detected within a time bound chosen and stated by the operator, since that fraction, and not the mean latency, separates a detector that leads triage from one that trails it.¶
A benchmark dataset supports reproducible, comparable evaluation only if it contains, at minimum:¶
the telemetry itself, covering every signal type the benchmark claims, on a common timeline;¶
the ground truth: the full injection schedule (or validated postmortem record), including fault type, target, severity, the signals in which each event is observable, the dimension values it covers, and the active and cooldown interval boundaries for every episode;¶
the baseline period of normal operation, clearly delimited;¶
enough topology and deployment metadata to interpret the faults (component inventory, dependency relationships, instrumentation method per component);¶
documentation of the collection pipeline, since preprocessing choices (aggregation windows, feature construction) are part of what is being reproduced.¶
Datasets intended for comparison across publications SHOULD be deposited with a persistent identifier and explicit versioning. Where a repository distinguishes a concept identifier (resolving to the latest version) from version identifiers, an evaluation MUST cite exactly one version and state which, since results are not reproducible against "latest". [ANJUM-DATA] is a partial example: it meets the baseline-delimitation, metadata, and pipeline-documentation requirements above, and deposits the processed feature tables, per-fold results, and raw detector scores derived from the telemetry. It meets the ground-truth requirement only in part, because it records the injection schedule with fault type, target, and interval boundaries but not the signals in which each event is observable, that distinction not having been drawn when it was collected (Section 4.3.1 explains what turns on it). The raw telemetry itself (about 51 GB) is not in the deposit and is available from the author.¶
Results obtained on a single dataset, however carefully constructed, are results about that dataset's fault distribution. Benchmark artifacts also carry defects of their own that flatter detectors in ways a single-corpus evaluation cannot expose [WU2023]. In the evaluation underlying this document, the detector that ranked first on the study's primary benchmark performed at or near chance on [RCAEVAL], an independently collected third-party benchmark, while the failure modes described in Section 3 reproduced on both (Appendix A). An evaluation making claims beyond a single environment SHOULD therefore include at least one dataset collected by a different team with a different pipeline, and SHOULD present ranking claims as specific to the evaluated fault distribution.¶
An evaluation report that permits comparison MUST state:¶
the definition of the evaluated set, its prevalence, and the treatment of cooldown/recovery windows (Section 4);¶
for an evaluation over more than one signal, the labeling treatment of Section 4.3.1 and the per-signal prevalences;¶
the detection window size and, where checked, window sensitivity (Section 3);¶
the threshold selection procedure, including whether labeled anomalies were available to it, or, for a detector without a score, the statement that it emits binary decisions only (Section 3.1);¶
precision, recall, and F1 at the stated operating point, and, where the detector emits a score, AUC-ROC and PR-AUC, each with dispersion across folds or repetitions rather than a single point value;¶
the flag-everything F1 baseline for the evaluated set's prevalence, and, for a detector that emits binary decisions only, the flagged fraction (Section 3.2);¶
episode-level detection rate, and event-level detection rate where more than one signal is evaluated, together with the window-level metrics they qualify, and a statement that no point adjustment was applied to those window-level metrics (Section 3.3);¶
for an evaluation by replay against labeled history (Section 4.4), the size and date range of that history;¶
the exact dataset version used, by persistent identifier.¶
A report SHOULD additionally state the known limits of its claims: the fault distribution evaluated, the observation horizon (whether multi-day effects such as seasonality or drift are covered at all), and whether the rankings reported are claimed to transfer beyond the evaluated environment. Rankings SHOULD be presented as relative performance under the stated fault distribution, not as universal detector quality.¶
[RFC Editor: please remove this section before publication, per RFC 7942.]¶
This section records the status of known implementations of the methodology defined in this document at the time of posting, following the process in [RFC7942]. The description of implementations in this section is intended to assist the IETF in its decision processes. The information is not intended to be a catalog of available implementations or their features. Readers are advised to note that other implementations may exist. The listing of an implementation here does not imply endorsement by the IETF, and no effort has been spent to verify the information presented here that was supplied by IETF contributors.¶
Benchmark datasets derived from operational environments can reveal network topology, service names, traffic patterns, capacity limits, and failure behavior of the originating operator. Datasets intended for publication SHOULD be anonymized with respect to addresses, hostnames, service identifiers, and tenant identifiers, and SHOULD be reviewed for indirect disclosure through correlated fields before release. Replay-based evaluation environments reproduce failure conditions by design and MUST be isolated from production systems.¶
This document has no IANA actions.¶
This informative appendix summarizes the published evaluation [ANJUM-EVAL] as an end-to-end application of Section 4 and Section 3, including the observed failure modes that motivate the normative guidance.¶
Environment and campaign. Two Kubernetes-hosted microservice testbeds were instrumented and unified through an OpenTelemetry Collector exporting metrics, logs, and traces to a common store: one natively instrumented 14-service system and one 7-service system retrofitted through operator-based auto-instrumentation, deliberately covering both instrumentation patterns of Section 4. A 24-hour normal-operation baseline was captured, then 40 fault scenarios (graded network latency, injected request errors, CPU and memory saturation, and compound cascades, targeted by architectural position), 20 per testbed, were each run 10 times, with a 5-minute active-fault phase and 5-minute cooldown per run. The two testbeds ran in parallel, so the 400 runs completed as 200 sequential runs each, producing roughly 51 GB of telemetry over a 33.5-hour campaign. Labels derived from the injection schedule under the schedule labeling of Section 4.3.1: every signal was labeled anomalous for the active span of every run. The primary evaluated set excluded cooldown windows, yielding prevalences of roughly 0.27 to 0.41 depending on signal, all disclosed. Partitioning was by repetition, meaning independent experimental replicates separated in time, across five folds. The primary detection window was 60 seconds. A sensitivity check re-ran the pipeline at 10 and 30 seconds, on the metrics signal only, for four of the eight detectors, and under a reduced hyperparameter search budget at 10 seconds; the folds common to all three window sizes numbered two rather than five. Eight semi-supervised detection methods spanning four families were evaluated, with a supervised baseline for reference.¶
The F1-inflation failure mode. At the metrics signal's prevalence, the flag-everything F1 floor was 0.582. Three of the eight detectors, all autoencoder variants, converged to exactly that value (F1 0.582 +/- 0.001) while their AUC-ROC lay between 0.17 and 0.22, far below chance: seemingly respectable F1 produced by degenerate detectors. The study diagnosed the cause as a score-distribution inversion, referred to below as the inversion: these detectors assigned lower anomaly scores to anomalous windows than to normal ones, so a threshold that flags nearly everything was the best operating point available to them. Because the inversion yields high F1 while AUC-ROC exposes it, the study adopted AUC-ROC as the metric of record and F1 as an operating-point summary read against its prevalence baseline, the rule generalized in Section 3.¶
Window sensitivity. Qualitative conclusions were stable across the three window sizes. Within the two-fold sub-study described above, one detector (a one-class deep method) scored AUC-ROC 0.916 at 30 seconds and 0.509 and 0.553 at 10 and 60 seconds, despite F1 near 0.74 at both of the lower values. The same detector scored AUC-ROC 0.776 on the metrics signal at 60 seconds under the full five-fold protocol, and the study's primary results place it among the detectors that avoid the degenerate collapse. Two folds, one signal and a reduced search budget are not interchangeable with the primary protocol, which is why Section 3 requires a window-sensitivity check to state the protocol it ran under.¶
Prevalence extrapolation. Re-scoring at a synthetic 5% prevalence, closer to production conditions, collapsed the flag-everything F1 floor to about 0.10 and reduced the best detector's F1 from 0.906 to 0.670, quantifying how strongly experimental prevalence overstates practical utility, the reason Section 6 requires prevalence disclosure.¶
External validity. The strongest detector on the primary benchmark (a density-estimation deep method, F1 0.906 +/- 0.060 and AUC-ROC 0.960 +/- 0.043 on metrics) performed at or near chance (AUC-ROC 0.33 to 0.52) on an independently collected third-party benchmark [RCAEVAL], where a classical isolation-based method was strongest (AUC-ROC up to 0.95). The failure modes, by contrast, transferred: on the external benchmark the reconstruction-based methods again posted deceptively non-trivial F1 (0.65 to 0.71) at AUC-ROC of only 0.50 to 0.60. The study accordingly claims no universal ranking, and this document's Section 5 requirement for an independent second dataset follows directly from that observation.¶
The processed feature tables, per-fold results, raw detector scores, and analysis code are archived at [ANJUM-DATA]; the one-command reproduction package is described in the Implementation Status section.¶
What this study predates. The study was designed and run before this document, and it does not meet every requirement stated here. It reports no event-level detection rate, because the two-level scoring of Section 4.3.1 was not defined when it ran; its three signals are scored and reported per signal only. It reports no detection-latency distribution (Section 3.3), having recorded episode-level detection alone. Its ground truth records the injection schedule but not per-signal observability, the limit noted in Section 5. The published article [ANJUM-EVAL] cites the dataset by its concept identifier rather than by the single version that Section 5 requires; [ANJUM-DATA] in this document cites version 3.1.3.¶
[RFC Editor: please remove this appendix before publication.]¶
The following changes were prompted by the review of -01 posted to the NMOP list by Wanting Du on 26 August 2026 (https://mailarchive.ietf.org/arch/msg/nmop/ZemcwNghqX4gvtUnCblH-s7rlfU/):¶
Scope (review point 2). The title, the Abstract, Section 1.2, and the definitions in Section 2 now cover rule-based, statistical, and machine-learning-based detectors equally, and "detector" is used throughout in place of "model". The title dropped "Machine-Learning-Based", which had contradicted the widened scope on the cover page; the document name is unchanged. Detector, Signal, and Event are defined. Section 3.1 adds the rule for detectors that emit a binary decision without a score, Section 3.2 adds the flagged-fraction comparison that replaces the AUC-ROC check for them, and the requirements in Section 3.3, Section 4.3.1, and Section 6 that depend on a score are now conditional so that a scoreless detector can comply with all of them.¶
Multi-signal ground truth (review point 3). New Section 4.3.1 defines labeling and scoring at the individual-signal level and at the aggregated-event level, and states two ambiguous cases explicitly: an event visible in one signal and absent in another, and an event spanning dimensions such as interface and peer. Section 1.1 and Section 5 were extended to match.¶
Live networks and hidden false negatives (the review's context note). Section 4.4 now makes the replay dependency explicit: a detector is evaluated against labeled history before it is trusted live, and re-evaluated as the labeled set grows through postmortem labeling, with [I-D.ietf-nmop-network-anomaly-semantics] cited for the labeling vocabulary and [I-D.ietf-nmop-network-anomaly-lifecycle] for the label store. Detection latency is to be reported as a distribution, and the section states why that matters more in the live case.¶
Review point 1 (the combination of threshold-independent metrics, operating-point metrics, and prevalence baselines) required no change.¶
The following changes come from the author's own accuracy pass over -01:¶
Section 1: the two lifecycle citations now point to Section 1 and Section 5.3 of [I-D.ietf-nmop-network-anomaly-lifecycle]; -01 cited subsections that do not exist. All quotations from the three NMOP documents were re-verified against the datatracker texts current on 6 September 2026 (architecture-08, lifecycle-07, semantics-06).¶
Section 2: "System" was removed from the list of terms attributed to [RFC9940], which does not define it.¶
Section 3.4, Section 5, and Appendix A: "no better than chance" and "at or below chance" were replaced by "near chance" where the cited AUC-ROC values (0.509, 0.553, 0.33 to 0.52) are near rather than below 0.5; Appendix A now defines the inversion at first use.¶
Section 5: the sentence presenting [ANJUM-DATA] as an example now states which requirements the deposit meets and that the raw telemetry is not in it.¶
Implementation Status: "latency reporting" was removed from the description (it is not implemented); the reproduction claim was narrowed to the prevalence tables the archive verifies; "complete" was dropped; and the public one-command reproduction package was added as a new fact.¶
References: [ANJUM-DATA] now cites one pinned version (version 3.1.3, DOI 10.5281/zenodo.22078287) with the title as recorded, as Section 5 requires, instead of the concept identifier.¶
Editorial: "in order to" (Abstract) and "most robust basis" (Section 3.1) were reworded.¶
The following changes come from a second accuracy pass, made before this revision was posted:¶
Appendix A: the campaign description now records that the 40 scenarios were 20 per testbed run on two testbeds in parallel, without which the stated run count and duration contradicted each other. The window-sensitivity comparison is now identified as a two-fold, single-signal, four-detector sub-study with a reduced search budget at 10 seconds, and the AUC-ROC the same detector reached at 60 seconds under the full five-fold protocol (0.776) is given alongside the sub-study's 0.553. A closing paragraph records which requirements of this document the worked example predates and does not meet.¶
Section 3.1 and Section 2: -01 asserted both that metrics are not comparable across evaluated sets and that AUC-ROC values are. Comparability is now stated per metric family, and prevalence insensitivity is no longer presented as corpus independence. An AUC-ROC well below 0.5 is distinguished from one near 0.5, as a sign-reversed ranking rather than an absent one.¶
Section 3.3: point adjustment is named and forbidden, so an episode-level rate cannot be used to relabel window-level outcomes. Section 6 requires a statement that none was applied.¶
Section 4.4: the replay requirement now constrains acceptance evidence rather than an operator's decision to trust a detector, which is outside the scope of this document. Section 4.3.1: the prohibition on pooling signals relaxed to SHOULD NOT with a disclosure requirement, and the two ambiguous cases became numbered subsections so they can be cited.¶
Section 5 and Implementation Status: the coverage claims were narrowed. The archive meets the ground-truth requirement only in part, since it does not record per-signal observability, and the Implementation Status section no longer claims Section 5 or Section 6 in full. That section now uses the disclaimer wording suggested by [RFC7942] and is titled "Author's Implementation".¶
References: third-party work is now cited for the precision-recall argument [SAITO2015], the point-adjustment critique [KIM2022], range-aware measures [PAPARRIZOS2022], and benchmark defects [WU2023]. The external benchmark of Section 5 and Appendix A is named as [RCAEVAL], where -01 referred to it without naming it.¶
Section 2: terms imported but never used were dropped from the borrowed list, and Event is related to [RFC9940]'s Problem. Section 1 narrows "no metric" to evaluation metrics, since the lifecycle document does define a Confidence Score, and the deployment-report figures now carry the period they cover.¶
The author thanks Wanting Du for her review of -01 on the NMOP list. Her comments prompted the changes to Section 1.2, Section 4.3.1, and Section 4.4 recorded in Appendix B.¶