]> Phono Technologies Inc.

Kitchener Ontario Canada mateenali66@gmail.com

Operations and Management Network Management Operations anomaly detection evaluation benchmarking telemetry AIOps machine learning The Network Management Operations (NMOP) working group has adopted documents describing an architecture, an operational lifecycle, and a semantics for network anomaly detection. Those documents direct implementers to minimize false positives and false negatives, but do not define how the accuracy of an anomaly detection implementation is to be measured, compared, or tracked over time. This document describes an evaluation methodology for anomaly detectors operating on network and infrastructure telemetry, whether the detector is rule-based, statistical, or machine-learning-based: the metrics to report and their known failure modes, a benchmarking procedure based on controlled fault injection and replay, ground-truth labeling and scoring across multiple telemetry signals, and the properties a benchmark dataset needs to support reproducible, comparable evaluation. The methodology is informational and complements the adopted NMOP anomaly-detection documents.

Introduction The NMOP working group has adopted three documents that together describe how network anomaly detection is to be architected, operated, and annotated: an architecture for Service Disruption Detection , an operational lifecycle for iterative refinement of detection systems , and a semantics for annotating detected anomalies . All three documents treat detection accuracy as a central objective. The architecture requires that "the operator can trust that a targeted Service Disruption will be detected (no false negatives), while not spamming the operator with Alarms that do not reflect an actual issue within the network (false positives)" (Section 3.5 of ). The lifecycle states that "Providers of solutions for network anomaly detection should aim at increasing accuracy, by minimizing False Positives and False Negatives" (Section 1 of ), and its refinement stage concludes by "'replaying' the network anomaly and checking if there is any measurable improvement" (Section 5.3 of ). None of the three documents defines how that accuracy or that improvement is to be measured. No evaluation metric, no formula, and no comparison procedure is specified. The consequence is visible in the architecture document's own deployment report, which counts 20 monitored service disruptions against 52 false positives since December 2022 (Section 4.1 of ) but has no framework in which to express whether that ratio is acceptable, improving, or comparable to an alternative detector. This document describes such a companion, an evaluation methodology for anomaly detectors operating on network and infrastructure telemetry: which metrics to report and the known failure modes of each (), a benchmarking procedure based on controlled fault injection and replay (), including how ground truth is labeled and scored when a detector operates over more than one telemetry signal (), the properties a benchmark dataset needs for reproducible comparison (), and the minimum contents of an evaluation report so that results from different detectors can be compared at all (). The methodology is grounded in a published peer-reviewed evaluation of eight machine-learning detection methods on unified telemetry , summarized as a worked example in .

Relationship to the NMOP Anomaly Detection Documents This document is designed to be used inside the lifecycle defined by , not alongside it: The lifecycle's Validation stage decides, per detected Symptom, whether a real problem was signaled. Accumulated validation outcomes are exactly the labels this document's metrics consume (). The lifecycle's Refinement stage requires a check for "measurable improvement" after replaying an anomaly. turns that check into a defined procedure with defined measures. The architecture's Replaying capability (Section 3.8 of ), which preserves the data that led to an Alarm so that the detection algorithms can be rerun on the same input, is the natural execution environment for the replay-based evaluation in . The semantics document defines the vocabulary in which a validated anomaly is recorded: a Symptom bound to one network plane, with the topology context (for example interface and peer) it concerns. Labels expressed in that vocabulary are the multi-signal ground truth that scores against. One definitional difference must be reconciled explicitly. defines a False Positive as "a detected anomaly which has been identified during the postmortem to be not anomalous", an operational, postmortem judgment. Dataset-based evaluation instead defines false positives against ground-truth labels fixed before evaluation. Both definitions are useful and they converge as validated postmortem outcomes accumulate into a labeled corpus; defines the ground-truth variants used in this document, and describes how operational validation records become benchmark labels.

Scope This document specifies methodology, not protocol. It defines no new data models, YANG modules, or wire formats. It applies to any detector that consumes telemetry signals (metrics, logs, traces, flow records, routing events, or their combination) and emits, per detection window or per event, a decision or a score. How the detector was built does not matter: rule-based detectors such as static thresholds and expert rules, statistical detectors such as seasonal baselines and change-point tests, and machine-learning-based detectors are all in scope. The metrics of , the per-Symptom and per-episode scoring of , and the reporting requirements of are defined on detector output and ground truth, not on the detector's internals, so they apply equally across these classes. The adaptations needed for detectors that emit a binary decision without a score are given in , and the reporting requirements that depend on a score are marked as conditional in . The metric guidance in comes from failure modes observed when evaluating machine-learning-based detectors, but none of it is specific to that class. The F1 floor of is a property of the evaluated set, not of the detector: a poorly set static threshold reaches it as readily as a collapsed learned model.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses the following terms as defined elsewhere and does not redefine them: Problem, Symptom, and Alarm from ; Service Disruption Detection System (SDDS) from ; and False Positive and False Negative in their operational sense from (see for the reconciliation with the ground-truth sense used here). The following evaluation-specific terms are defined for this document:

Detector:

Any component that consumes one or more telemetry signals and emits, per detection window or per event, either a decision (anomalous or non-anomalous) or a score from which such a decision is derived by a threshold. The term covers rule-based, statistical, and machine-learning-based implementations alike; nothing in this document depends on which.

Signal:

One telemetry stream over which a detector produces its own sequence of decisions or scores, for example metrics, logs, or traces from a set of services, or flow records from the forwarding plane and routing events from the control plane of a network. A detector may operate over several signals; its output is then scored per signal and per event ().

Ground Truth:

The authoritative record of which intervals or events in a telemetry corpus are anomalous, fixed before an evaluation begins. Ground truth may originate from a controlled injection schedule () or from accumulated, validated postmortem outcomes.

Label:

The per-interval or per-event value assigned by the ground truth (anomalous or non-anomalous), against which detector output is scored.

Event:

One anomalous condition as recorded by the ground truth, with one onset and one end, together with the signals in which it is observable and the dimension values (for example interface or peer) it covers. Under injection, one fault-injection run is one event; under postmortem labeling, one validated annotation and its list of Symptoms is one event. An episode () is an event viewed on a single signal. An Event in this sense corresponds to a Problem in terms, qualified by the requirement that the ground truth, not a detector, is what attributes the Symptoms to it.

Evaluated Set:

The exact collection of intervals or events over which metrics are computed. Threshold-dependent metrics computed over different evaluated sets are not comparable, because they depend on the prevalence of the set (). Threshold-independent metrics are comparable across sets to the extent the sets are drawn from comparable fault distributions ().

Prevalence:

The fraction of the evaluated set that is labeled anomalous.

Detection Window:

The fixed time interval into which telemetry is aggregated before a detection decision is made for that interval.

Detection Latency:

The elapsed time between the onset of an anomalous condition per the ground truth and the first detection decision that flags it.

Evaluation Metrics Detection quality cannot be summarized in a single number. This section defines a minimum metric set and, for each threshold-dependent metric, the conditions under which it misleads. The guidance here is grounded in observed failure modes from a controlled evaluation of eight detection methods ; the worked example in shows each failure mode with real values.

Threshold-Independent Metrics An anomaly detector typically produces a continuous score per detection window, converted to a binary decision by a threshold. Metrics computed over the score distribution, independent of any threshold, are the soundest basis for comparing detectors: AUC-ROC (area under the receiver operating characteristic curve) SHOULD be the metric of record for ranking and acceptance decisions. It is threshold-independent and insensitive to prevalence, so a change in the anomaly rate of the evaluated set does not by itself move it. An AUC-ROC near 0.5 indicates a detector performing no better than chance, regardless of any other metric reported for it. PR-AUC (area under the precision-recall curve) SHOULD be reported alongside AUC-ROC. It focuses on the anomalous class and is the more informative of the two when prevalence is low, the common operational condition . The two answer different questions. AUC-ROC ranks, because prevalence insensitivity is what makes detectors scored on sets with different anomaly rates comparable at all. PR-AUC reflects the false-positive burden an operator carries at low prevalence. Prevalence insensitivity is not corpus independence: sets differ in fault distribution as well as in anomaly rate, and records one detector scoring far apart on two corpora. Comparison across corpora is bounded by . An AUC-ROC well below 0.5 is a different condition from one near it. Near 0.5 the score carries no ranking information; well below 0.5 it carries ranking information with the sign reversed, placing anomalous windows below normal ones. The second is a defect in the score convention, not an absence of signal. An evaluation that encounters it SHOULD report the observed value, state that the ordering is inverted, and re-evaluate with the sign corrected before judging the detector's discriminative power. A detector that emits only a binary decision, with no score, has a single operating point, and AUC-ROC and PR-AUC cannot be computed for it. Such a detector MUST be reported with the threshold-dependent metrics of against the prevalence baseline defined there, and SHOULD expose the quantity it thresholds internally (for example, the deviation from a static limit, or the test statistic of a change-point test) as a score, so that the threshold-independent metrics can be computed and the detector compared with scoring detectors on the same footing.

Threshold-Dependent Metrics and the Prevalence Problem Precision, recall, and their harmonic mean F1 describe a single operating point and remain the terms in which operators reason ("how many alarms are real", "how many disruptions are missed"). They MUST NOT, however, be used in isolation for acceptance or comparison, for two reasons. F1 has a nontrivial floor. A degenerate detector that flags every window achieves recall 1 and F1 = 2p/(1+p), where p is the prevalence of the evaluated set. At a prevalence of 0.4 this floor is approximately 0.57; a reported F1 of 0.6 at that prevalence is therefore nearly indistinguishable from a detector that discriminates nothing. The same degenerate detector falls to F1 of roughly 0.10 at 5% prevalence, which is why F1 values obtained at different prevalences are not comparable at all. An evaluation MUST report the prevalence of its evaluated set and the F1 of the trivial flag-everything detector on that set as a baseline. Where a score is available, an evaluation SHOULD verify AUC-ROC exceeds 0.5 before any F1 value is credited. For a detector that emits binary decisions only no such check is possible, and the comparison below takes its place. For a scoreless detector, compare the fraction of windows it flags with the prevalence of the evaluated set. A flagged fraction near 1 is the flag-everything detector in all but name, whatever its F1, and a flagged fraction well above prevalence buys recall with false positives that F1 at that prevalence hides. An evaluation of a scoreless detector MUST report its flagged fraction alongside precision, recall, and F1. Threshold selection can additionally mask detector quality in either direction. If the threshold is tuned on labeled validation data, reported F1 reflects an optimistic operating point unavailable in production, where thresholds are typically set from the score distribution of normal traffic alone (for example, a high percentile of normal-only scores). An evaluation MUST state how thresholds were selected, and SHOULD report the sensitivity of its threshold-dependent metrics to that choice.

Episode-Level Detection and Latency Window-level metrics measure classification quality; operators additionally need to know whether whole incidents are caught and how quickly. An episode is one contiguous anomalous condition per the ground truth on one signal (for a benchmark, one fault-injection run as seen on that signal). An episode counts as detected if at least one window within its labeled span is flagged. Episode-level detection rate is a recall-oriented measure and is trivially satisfied by the flag-everything detector, so it MUST be reported together with window-level precision, and with AUC-ROC where a score is available, never alone. The episode-level rate MUST NOT be used to modify the window-level metrics. Windows of a detected episode that the detector did not flag remain false negatives, and an evaluation MUST NOT relabel them as true positives before computing window-level precision, recall, or F1. That practice, known as point adjustment, inflates window-level scores far enough that a random score can outrank a discriminating detector . Range-aware threshold-independent measures are an alternative to two-level reporting where a detector is scored against anomalous ranges rather than isolated windows. This document does not require them; an evaluation MAY report them in addition. Where onset timestamps are available, detection latency () SHOULD be reported as a distribution, not only a mean; explains why the distribution, rather than its mean, is what predicts behavior on a live network. When a detector operates over several signals, the event-level counterpart of the episode-level rate is defined in .

Window Sensitivity The detection window is an experimental parameter with direct effect on results: aggregation over a longer window smooths transient faults, a shorter one raises noise. Conclusions that hold only at one window size are fragile: in a two-fold, single-signal sub-study of the evaluation underlying this document, one detector scored far apart across 10, 30 and 60-second windows ( records that sub-study's limits). An evaluation MUST state its primary window size and its rationale, and SHOULD re-run its headline comparison at one or more neighboring window sizes, reporting whether the qualitative conclusions are stable.

Benchmarking Procedure This section describes a repeatable procedure for producing the ground truth and the evaluated set that the metrics in require. The procedure is based on controlled fault injection into an instrumented environment; it is the offline complement to the operational Validation stage of .

Environment and Instrumentation The environment SHOULD be a representative multi-component system emitting the telemetry signal types the detector under test consumes, unified through a common collection pipeline so that metrics, logs, and traces share a timeline. Two instrumentation patterns exist in practice, and a benchmark gains external validity by covering both: components instrumented natively at development time, and components instrumented after the fact through operator- or agent-based retrofit. A baseline period of normal operation MUST be captured before any fault is injected; it supplies training data for detectors that learn from normal behavior only, the reference against which rule-based and statistical detectors set their limits, and the score distribution from which production-style thresholds are derived ().

Fault Injection Faults MUST be injected under an explicit, recorded schedule; the schedule is the ground truth. A useful minimum taxonomy, drawn from , covers: added network latency at graded severities, injected request errors, CPU saturation, memory saturation, and compound cascades that combine a fault on one component with resource stress on a dependent component. Injection targets SHOULD be chosen by architectural position (entry point, mid-chain dependency, backend) rather than convenience, and each scenario SHOULD be repeated multiple times so that dispersion across repetitions can be reported. Each run SHOULD include a defined active-fault phase followed by a cooldown phase long enough for the system to return to nominal behavior.

Ground-Truth Labeling Labels derive from the injection schedule: windows within an active-fault phase are anomalous. Cooldown windows require an explicit decision. The system is recovering but no fault is active; labeling them anomalous inflates prevalence with ambiguous intervals, while silently including them as normal penalizes detectors that correctly flag residual disturbance. The evaluation MUST state how cooldown windows were treated, and SHOULD report its headline results with cooldown windows excluded from the evaluated set, with the resulting prevalence disclosed. Where a detector is fitted, and its evaluation data is therefore partitioned for training, validation, and testing, the partition MUST be by episode (whole runs), never by random sampling of windows; windows from the same run are temporally correlated, and random partitions leak that correlation into inflated results. Operational deployments accumulate an equivalent ground truth over time: each postmortem-validated outcome under the lifecycle of is a label. describes how such records become benchmark datasets.

Multi-Signal Ground Truth and Scoring A detector, or a set of detectors, commonly operates over more than one signal. In the environment of the signals are metrics, logs, and traces from the same services. In an operator network they may be flow records from the forwarding plane (for example IPFIX) and routing events from the control plane (for example BGP), each carrying dimensions such as interface and peer, and an event on one plane may or may not show on another. The labels of are per signal and per window. This section defines how labels and scores are assigned at two levels, the individual signal and the aggregated event, and states the ambiguous cases explicitly. An evaluation over more than one signal MUST report both levels. Labeling at the signal level. Each signal carries its own label sequence over its own windows. Under injection, the schedule fixes the active span of each event; under postmortem labeling, the recorded Symptoms fix it. In the vocabulary of , each Symptom is bound to one network plane (Section 3 of that document) and to the topology context it concerns (Section 4.3 of that document), so a Symptom labels the signal of its plane and the dimension values it names, and nothing else. Labeling at the event level. An event () is the set of Symptoms, across signals and dimension values, that the ground truth attributes to one cause, with one onset and one end. Under injection, one run is one event. Under postmortem labeling, one validated annotation in the label store of (Section 6 of that document), with the list of Symptoms it names, is one event. The ground truth MUST record, for every event, the signals in which it is observable and the dimension values it covers. Scoring at the signal level. Window-level metrics (, ) are computed per signal, on that signal's windows against that signal's labels, and reported per signal. They SHOULD NOT be pooled across signals into one evaluated set: prevalence and window counts differ by signal (in , prevalence ranged from 0.27 to 0.41 by signal), and pooling changes the evaluated set in the way warns against. An evaluation that pools anyway MUST report the per-signal prevalences and window counts alongside the pooled figure, so that a reader can see what the pooled set is composed of. Scoring at the event level. An event counts as detected if at least one signal in which the event is labeled observable flags at least one window inside the event's span on a dimension value the event covers. The event-level detection rate is the multi-signal counterpart of the episode-level rate of and inherits its rules: it is recall-oriented, and MUST be reported together with the signal-level precision, and the signal-level AUC-ROC where a score is available, that it summarizes, never alone. It MUST NOT be used to relabel signal-level windows, for the reason given in . A flagged window that falls inside no event's span on its signal is a false positive at both levels.

Ambiguous Case 1: An Event Visible in One Signal and Absent in Another Two labeling treatments are admissible, and an evaluation MUST state which it used and MUST NOT mix them within one evaluated set. Schedule labeling: every signal is labeled anomalous for every window in the event's span, whether or not the event is observable in that signal. A miss on a signal the event does not reach is then a signal-level false negative. This treatment measures whether each signal's detector catches each class of event, and its informative output is per-event-class recall reported per signal. The evaluation in used it. Visibility labeling: only the signals in which the ground truth records the event as observable are labeled anomalous for its span. A flag on an unlabeled signal during the event is a signal-level false positive, and a miss there is a true negative. Postmortem labeling under yields this treatment by construction, since a Symptom is recorded only where something was observed. Under either treatment the event is detected at the event level if any signal in which it is labeled observable flags it. The two treatments give different signal-level prevalences for the same corpus, so results obtained under one are not comparable with results obtained under the other.

Ambiguous Case 2: An Event Spanning Dimensions An event may cover several values of one dimension (several interfaces on one node, several peers of one router) or several dimensions at once (an interface and a BGP peer). The ground truth MUST record every dimension value the event covers, in the terms of the semantics document's service topology model where it applies. A flagged window during the event on a dimension value the event does not cover (an unaffected interface, an unrelated peer) is a false positive at both levels, not a detection of the event. When the detector reports at a coarser dimension than the label (per node when the label is per interface), the evaluation MUST state its matching rule; the default is that a coarse detection counts as detecting every labeled fine-grained value it contains, and the report SHOULD then also give the fraction of events whose labeled dimension values were named exactly, since localization, not only detection, is what the operator acts on. The rules in this section are fitted to the two ground-truth shapes available to the author: the injection schedule of , and the Symptom-and-topology vocabulary of . Input from operators holding labeled multi-plane events, for example flow records and routing events across the forwarding and control planes with interface and peer dimensions, is sought so that a future revision can include a worked instance in that vocabulary.

Replay-Based Evaluation Section 3.8 of calls for an SDDS to preserve the data that led to an Alarm so that the detection algorithms can be rerun on the same input. That capability turns the lifecycle's "measurable improvement" check into a defined measurement: hold the evaluated set fixed, rerun the modified detector on the identical preserved input, and compare the metrics of before and after the change. A comparison in which the evaluated set itself changed between runs is not a measurement of the detector, and reporting it as an improvement misstates what was measured. Live networks have no injection schedule. Operator triage records true positives and false positives as Alarms are validated, but a false negative leaves no record until its effect is reported by a user, so the labeled set of a live network is incomplete on exactly the side that matters most. This document therefore treats evaluation for a live network as replay against labeled history, in two steps that depend on the lifecycle and semantics documents. Acceptance evidence for a detector intended for live traffic MUST include metrics obtained by replay against the labeled history available: a benchmark dataset (), the operator's own preserved Alarm data with validated outcomes, or both. Metrics accumulated from live triage alone do not constitute such evidence, because their false-negative count is a lower bound. The labeled set grows as postmortem labeling proceeds. defines the vocabulary in which that labeling is expressed, a Symptom per network plane with its topology context (Sections 3 and 4 of that document), and the label store of (Section 6 of that document) persists each annotation and its versions. Every validated annotation, including a user-reported disruption that no detector flagged, adds labels to the history. A detector SHOULD be re-evaluated by replay whenever the labeled set has grown materially, and the report MUST state the size and date range of the labeled history used, so that two evaluations of the same detector are read against the labeled sets they were computed on. A re-evaluation on a grown labeled set is a new measurement, not a measurement of improvement; to measure the effect of a detector change, both versions are run on the same, current labeled set. Detection latency matters more in the live case than in a benchmark. In a benchmark a late true positive still counts as detected within its episode. In a live network a true positive that arrives after a user has reported the disruption is, in operational terms, a false negative, because the user report and not the detector triggered triage. An evaluation intended to predict live behavior SHOULD therefore report detection latency as a distribution (), and SHOULD state the fraction of episodes detected within a time bound chosen and stated by the operator, since that fraction, and not the mean latency, separates a detector that leads triage from one that trails it.

Benchmark Dataset Requirements A benchmark dataset supports reproducible, comparable evaluation only if it contains, at minimum: the telemetry itself, covering every signal type the benchmark claims, on a common timeline; the ground truth: the full injection schedule (or validated postmortem record), including fault type, target, severity, the signals in which each event is observable, the dimension values it covers, and the active and cooldown interval boundaries for every episode; the baseline period of normal operation, clearly delimited; enough topology and deployment metadata to interpret the faults (component inventory, dependency relationships, instrumentation method per component); documentation of the collection pipeline, since preprocessing choices (aggregation windows, feature construction) are part of what is being reproduced. Datasets intended for comparison across publications SHOULD be deposited with a persistent identifier and explicit versioning. Where a repository distinguishes a concept identifier (resolving to the latest version) from version identifiers, an evaluation MUST cite exactly one version and state which, since results are not reproducible against "latest". is a partial example: it meets the baseline-delimitation, metadata, and pipeline-documentation requirements above, and deposits the processed feature tables, per-fold results, and raw detector scores derived from the telemetry. It meets the ground-truth requirement only in part, because it records the injection schedule with fault type, target, and interval boundaries but not the signals in which each event is observable, that distinction not having been drawn when it was collected ( explains what turns on it). The raw telemetry itself (about 51 GB) is not in the deposit and is available from the author. Results obtained on a single dataset, however carefully constructed, are results about that dataset's fault distribution. Benchmark artifacts also carry defects of their own that flatter detectors in ways a single-corpus evaluation cannot expose . In the evaluation underlying this document, the detector that ranked first on the study's primary benchmark performed at or near chance on , an independently collected third-party benchmark, while the failure modes described in reproduced on both (). An evaluation making claims beyond a single environment SHOULD therefore include at least one dataset collected by a different team with a different pipeline, and SHOULD present ranking claims as specific to the evaluated fault distribution.

Reporting Requirements An evaluation report that permits comparison MUST state: the definition of the evaluated set, its prevalence, and the treatment of cooldown/recovery windows (); for an evaluation over more than one signal, the labeling treatment of and the per-signal prevalences; the detection window size and, where checked, window sensitivity (); the threshold selection procedure, including whether labeled anomalies were available to it, or, for a detector without a score, the statement that it emits binary decisions only (); precision, recall, and F1 at the stated operating point, and, where the detector emits a score, AUC-ROC and PR-AUC, each with dispersion across folds or repetitions rather than a single point value; the flag-everything F1 baseline for the evaluated set's prevalence, and, for a detector that emits binary decisions only, the flagged fraction (); episode-level detection rate, and event-level detection rate where more than one signal is evaluated, together with the window-level metrics they qualify, and a statement that no point adjustment was applied to those window-level metrics (); for an evaluation by replay against labeled history (), the size and date range of that history; the exact dataset version used, by persistent identifier. A report SHOULD additionally state the known limits of its claims: the fault distribution evaluated, the observation horizon (whether multi-day effects such as seasonality or drift are covered at all), and whether the rankings reported are claimed to transfer beyond the evaluated environment. Rankings SHOULD be presented as relative performance under the stated fault distribution, not as universal detector quality.

Implementation Status [RFC Editor: please remove this section before publication, per RFC 7942.] This section records the status of known implementations of the methodology defined in this document at the time of posting, following the process in . The description of implementations in this section is intended to assist the IETF in its decision processes. The information is not intended to be a catalog of available implementations or their features. Readers are advised to note that other implementations may exist. The listing of an implementation here does not imply endorsement by the IETF, and no effort has been spent to verify the information presented here that was supplied by IETF contributors.

Author's Implementation Organization: Phono Technologies Inc. (document author) Description: An implementation of the metric computation and reporting requirements of this document: the threshold-independent and prevalence-baselined metrics of , the prevalence re-scoring analysis, episode-level detection rate, and the reporting requirements of , applied to eight machine-learning detection methods over unified OpenTelemetry traces, metrics, and logs. It does not implement the detection-latency distribution of (the underlying study recorded episode-level detection only), and it represents the fault-injection procedure of by the recorded injection configurations and archived outputs rather than by an executable environment. Maturity: Research-grade. The analysis code was executed against the archived benchmark in a clean environment on 2026-08-24 and regenerated the archived prevalence-sensitivity tables exactly, including the values reported in the associated peer-reviewed study (DOI 10.1109/ACCESS.2026.3705430). A one-command reproduction package is public at https://github.com/mateenali66/otel-aiops-reproduction. It fetches the archived record and checks its checksum, runs in a container image pinned by digest with fixed seeds, provides a smoke mode that runs one signal and one fold in a few minutes on a laptop CPU (about two and a half minutes in the recorded run), and includes a verify step that compares the smoke output against the archived per-fold results and regenerates the archived tables from the archived raw scores. The full eight-method regeneration through the package had not been run at the time of this revision; the package documents its estimated cost. Coverage: in full except for the detection-latency distribution of , which is not implemented. except for the event-level detection rate and the multi-signal labeling statement, both introduced in this revision and not yet implemented. in part: the archive meets the requirements recorded in , including the ground-truth requirement only to the extent described there, and does not carry the raw telemetry. by recorded configurations and archived outputs. The eight detection methods implemented are all machine-learning methods, so the rule-based and statistical classes admitted by are exercised only through the package's path for externally supplied binary decisions. Full retraining of the detectors additionally requires the environment described in the study or the package's CPU path. Licensing: The archived record is openly available under CC-BY 4.0 with a versioned persistent identifier (DOI 10.5281/zenodo.22078287, version 3.1.3), and includes code, processed feature tables with versioned ground truth, per-fold results, raw detector scores, and reproduction instructions. The reproduction package is public on GitHub at the address above; its wrapper code is Apache-2.0 and the artifacts it fetches and the tables it derives remain CC-BY 4.0. Contact: mateenali66@gmail.com The processed feature tables and ground truth in the same archive are usable by other implementations of this methodology as a common evaluation baseline, within the limits recorded in ; the author is aware of no independent implementation at the time of this revision and will record any that appear in future revisions.

Security Considerations Benchmark datasets derived from operational environments can reveal network topology, service names, traffic patterns, capacity limits, and failure behavior of the originating operator. Datasets intended for publication SHOULD be anonymized with respect to addresses, hostnames, service identifiers, and tenant identifiers, and SHOULD be reviewed for indirect disclosure through correlated fields before release. Replay-based evaluation environments reproduce failure conditions by design and MUST be isolated from production systems.

IANA Considerations This document has no IANA actions.

In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Swisscom Swisscom INSA-Lyon Deutsche Telekom This document describes the motivation and architecture of a Network Anomaly Detection Framework and the relationship to other documents describing network Symptom semantics and network incident lifecycle. The described architecture for detecting IP network service interruption is designed to be generic applicable and extensible. Different applications are described and examples are referenced with open-source running code. Huawei Swisscom Swisscom Deutsche Telekom This document defines a structured, iterative lifecycle for network anomaly detection systems to enable "human-in-the-loop" refinements. Key contributions include defining three lifecycle stages, a state machine for anomaly annotations, and YANG data models for standardized labeling and exchange. Swisscom Swisscom Deutsche Telekom Huawei The document proposes a unified symptoms vocabulary for network anomaly metadata to improve data sharing and analysis among human network operators, network analytics implementers, and AI systems to improve accuracy of Service Disruption Detection. This document sets out some terms that are fundamental to a common understanding of network fault and problem management within the IETF. The purpose of this document is to bring clarity to discussions and other work related to network fault and problem management -- in particular, to YANG data models and management protocols that report, make visible, or manage network faults and problems. This document describes a simple process that allows authors of Internet-Drafts to record the status of known implementations by including an Implementation Status section. This will allow reviewers and working groups to assign due consideration to documents that have the benefit of running code, which may serve as evidence of valuable experimentation and feedback that have made the implemented protocols more mature. This process is not mandatory. Authors of Internet-Drafts are encouraged to consider using the process for their documents, and working groups are invited to think about applying the process to all of their protocol specifications. This document obsoletes RFC 6982, advancing it to a Best Current Practice.

Worked Example: An Application of This Methodology This informative appendix summarizes the published evaluation as an end-to-end application of and , including the observed failure modes that motivate the normative guidance. Environment and campaign. Two Kubernetes-hosted microservice testbeds were instrumented and unified through an OpenTelemetry Collector exporting metrics, logs, and traces to a common store: one natively instrumented 14-service system and one 7-service system retrofitted through operator-based auto-instrumentation, deliberately covering both instrumentation patterns of . A 24-hour normal-operation baseline was captured, then 40 fault scenarios (graded network latency, injected request errors, CPU and memory saturation, and compound cascades, targeted by architectural position), 20 per testbed, were each run 10 times, with a 5-minute active-fault phase and 5-minute cooldown per run. The two testbeds ran in parallel, so the 400 runs completed as 200 sequential runs each, producing roughly 51 GB of telemetry over a 33.5-hour campaign. Labels derived from the injection schedule under the schedule labeling of : every signal was labeled anomalous for the active span of every run. The primary evaluated set excluded cooldown windows, yielding prevalences of roughly 0.27 to 0.41 depending on signal, all disclosed. Partitioning was by repetition, meaning independent experimental replicates separated in time, across five folds. The primary detection window was 60 seconds. A sensitivity check re-ran the pipeline at 10 and 30 seconds, on the metrics signal only, for four of the eight detectors, and under a reduced hyperparameter search budget at 10 seconds; the folds common to all three window sizes numbered two rather than five. Eight semi-supervised detection methods spanning four families were evaluated, with a supervised baseline for reference. The F1-inflation failure mode. At the metrics signal's prevalence, the flag-everything F1 floor was 0.582. Three of the eight detectors, all autoencoder variants, converged to exactly that value (F1 0.582 +/- 0.001) while their AUC-ROC lay between 0.17 and 0.22, far below chance: seemingly respectable F1 produced by degenerate detectors. The study diagnosed the cause as a score-distribution inversion, referred to below as the inversion: these detectors assigned lower anomaly scores to anomalous windows than to normal ones, so a threshold that flags nearly everything was the best operating point available to them. Because the inversion yields high F1 while AUC-ROC exposes it, the study adopted AUC-ROC as the metric of record and F1 as an operating-point summary read against its prevalence baseline, the rule generalized in . Window sensitivity. Qualitative conclusions were stable across the three window sizes. Within the two-fold sub-study described above, one detector (a one-class deep method) scored AUC-ROC 0.916 at 30 seconds and 0.509 and 0.553 at 10 and 60 seconds, despite F1 near 0.74 at both of the lower values. The same detector scored AUC-ROC 0.776 on the metrics signal at 60 seconds under the full five-fold protocol, and the study's primary results place it among the detectors that avoid the degenerate collapse. Two folds, one signal and a reduced search budget are not interchangeable with the primary protocol, which is why requires a window-sensitivity check to state the protocol it ran under. Prevalence extrapolation. Re-scoring at a synthetic 5% prevalence, closer to production conditions, collapsed the flag-everything F1 floor to about 0.10 and reduced the best detector's F1 from 0.906 to 0.670, quantifying how strongly experimental prevalence overstates practical utility, the reason requires prevalence disclosure. External validity. The strongest detector on the primary benchmark (a density-estimation deep method, F1 0.906 +/- 0.060 and AUC-ROC 0.960 +/- 0.043 on metrics) performed at or near chance (AUC-ROC 0.33 to 0.52) on an independently collected third-party benchmark , where a classical isolation-based method was strongest (AUC-ROC up to 0.95). The failure modes, by contrast, transferred: on the external benchmark the reconstruction-based methods again posted deceptively non-trivial F1 (0.65 to 0.71) at AUC-ROC of only 0.50 to 0.60. The study accordingly claims no universal ranking, and this document's requirement for an independent second dataset follows directly from that observation. The processed feature tables, per-fold results, raw detector scores, and analysis code are archived at ; the one-command reproduction package is described in the Implementation Status section. What this study predates. The study was designed and run before this document, and it does not meet every requirement stated here. It reports no event-level detection rate, because the two-level scoring of was not defined when it ran; its three signals are scored and reported per signal only. It reports no detection-latency distribution (), having recorded episode-level detection alone. Its ground truth records the injection schedule but not per-signal observability, the limit noted in . The published article cites the dataset by its concept identifier rather than by the single version that requires; in this document cites version 3.1.3.

Changes from -01 to -02 [RFC Editor: please remove this appendix before publication.] The following changes were prompted by the review of -01 posted to the NMOP list by Wanting Du on 26 August 2026 (https://mailarchive.ietf.org/arch/msg/nmop/ZemcwNghqX4gvtUnCblH-s7rlfU/): Scope (review point 2). The title, the Abstract, , and the definitions in now cover rule-based, statistical, and machine-learning-based detectors equally, and "detector" is used throughout in place of "model". The title dropped "Machine-Learning-Based", which had contradicted the widened scope on the cover page; the document name is unchanged. Detector, Signal, and Event are defined. adds the rule for detectors that emit a binary decision without a score, adds the flagged-fraction comparison that replaces the AUC-ROC check for them, and the requirements in , , and that depend on a score are now conditional so that a scoreless detector can comply with all of them. Multi-signal ground truth (review point 3). New defines labeling and scoring at the individual-signal level and at the aggregated-event level, and states two ambiguous cases explicitly: an event visible in one signal and absent in another, and an event spanning dimensions such as interface and peer. and were extended to match. Live networks and hidden false negatives (the review's context note). now makes the replay dependency explicit: a detector is evaluated against labeled history before it is trusted live, and re-evaluated as the labeled set grows through postmortem labeling, with cited for the labeling vocabulary and for the label store. Detection latency is to be reported as a distribution, and the section states why that matters more in the live case. Review point 1 (the combination of threshold-independent metrics, operating-point metrics, and prevalence baselines) required no change. The following changes come from the author's own accuracy pass over -01: Section 1: the two lifecycle citations now point to Section 1 and Section 5.3 of ; -01 cited subsections that do not exist. All quotations from the three NMOP documents were re-verified against the datatracker texts current on 6 September 2026 (architecture-08, lifecycle-07, semantics-06). : "System" was removed from the list of terms attributed to , which does not define it. Section 3.4, , and : "no better than chance" and "at or below chance" were replaced by "near chance" where the cited AUC-ROC values (0.509, 0.553, 0.33 to 0.52) are near rather than below 0.5; now defines the inversion at first use. : the sentence presenting as an example now states which requirements the deposit meets and that the raw telemetry is not in it. Implementation Status: "latency reporting" was removed from the description (it is not implemented); the reproduction claim was narrowed to the prevalence tables the archive verifies; "complete" was dropped; and the public one-command reproduction package was added as a new fact. References: now cites one pinned version (version 3.1.3, DOI 10.5281/zenodo.22078287) with the title as recorded, as requires, instead of the concept identifier. Editorial: "in order to" (Abstract) and "most robust basis" (Section 3.1) were reworded. The following changes come from a second accuracy pass, made before this revision was posted: : the campaign description now records that the 40 scenarios were 20 per testbed run on two testbeds in parallel, without which the stated run count and duration contradicted each other. The window-sensitivity comparison is now identified as a two-fold, single-signal, four-detector sub-study with a reduced search budget at 10 seconds, and the AUC-ROC the same detector reached at 60 seconds under the full five-fold protocol (0.776) is given alongside the sub-study's 0.553. A closing paragraph records which requirements of this document the worked example predates and does not meet. and : -01 asserted both that metrics are not comparable across evaluated sets and that AUC-ROC values are. Comparability is now stated per metric family, and prevalence insensitivity is no longer presented as corpus independence. An AUC-ROC well below 0.5 is distinguished from one near 0.5, as a sign-reversed ranking rather than an absent one. : point adjustment is named and forbidden, so an episode-level rate cannot be used to relabel window-level outcomes. requires a statement that none was applied. : the replay requirement now constrains acceptance evidence rather than an operator's decision to trust a detector, which is outside the scope of this document. : the prohibition on pooling signals relaxed to SHOULD NOT with a disclosure requirement, and the two ambiguous cases became numbered subsections so they can be cited. and Implementation Status: the coverage claims were narrowed. The archive meets the ground-truth requirement only in part, since it does not record per-signal observability, and the Implementation Status section no longer claims or in full. That section now uses the disclaimer wording suggested by and is titled "Author's Implementation". References: third-party work is now cited for the precision-recall argument , the point-adjustment critique , range-aware measures , and benchmark defects . The external benchmark of and is named as , where -01 referred to it without naming it. : terms imported but never used were dropped from the borrowed list, and Event is related to 's Problem. Section 1 narrows "no metric" to evaluation metrics, since the lifecycle document does define a Confidence Score, and the deployment-report figures now carry the period they cover.

Acknowledgments The author thanks Wanting Du for her review of -01 on the NMOP list. Her comments prompted the changes to , , and recorded in .