Benchmarking Methodology M. Gaikwad Internet-Draft Independent Intended status: Informational 11 August 2026 Expires: 12 February 2027 Benchmarking Methodology for Output Behavior Fault Detection in Large Language Model Serving Systems draft-gaikwad-llm-fault-detection-methodology-00 Abstract This document defines test procedures for characterizing the fault detection capability of observability systems that monitor Large Language Model (LLM) serving deployments. Procedures are given for Detection Latency, Detection Coverage, Detection Threshold Magnitude, False Detection Rate, and Boundary Masking. The Detector Under Test is the observability system. Output Behavior Faults are injected at a known time under controlled conditions, which makes detection latency directly measurable. This document is a companion to "Benchmarking Terminology for Output Behavior Fault Detection in Large Language Model Serving Systems" and is to be read alongside it. This document specifies no acceptance thresholds. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 12 February 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. Gaikwad Expires 12 February 2027 [Page 1] Internet-Draft Fault Detection Methodology August 2026 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2. Applicability . . . . . . . . . . . . . . . . . . . . . . 3 2. Conventions and Definitions . . . . . . . . . . . . . . . . . 4 3. Test Setup . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.1. Topology . . . . . . . . . . . . . . . . . . . . . . . . 4 3.2. Detector Under Test Configuration . . . . . . . . . . . . 5 3.3. Traffic Generation . . . . . . . . . . . . . . . . . . . 5 3.4. Fault Injection Apparatus . . . . . . . . . . . . . . . . 5 3.5. Clock Synchronization . . . . . . . . . . . . . . . . . . 6 4. Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . 6 4.1. Detection Signal Declaration . . . . . . . . . . . . . . 6 4.2. Baseline Characterization . . . . . . . . . . . . . . . . 6 4.3. Inter-Trial Quiet Period . . . . . . . . . . . . . . . . 7 4.4. Single Fault Application . . . . . . . . . . . . . . . . 7 5. Test Procedures . . . . . . . . . . . . . . . . . . . . . . . 7 5.1. Detection Latency . . . . . . . . . . . . . . . . . . . . 7 5.2. Detection Coverage . . . . . . . . . . . . . . . . . . . 8 5.3. Detection Threshold Magnitude . . . . . . . . . . . . . . 8 5.4. False Detection Rate . . . . . . . . . . . . . . . . . . 9 5.5. Boundary Masking . . . . . . . . . . . . . . . . . . . . 9 6. Reporting Format . . . . . . . . . . . . . . . . . . . . . . 10 7. Security Considerations . . . . . . . . . . . . . . . . . . . 11 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11 9. Normative References . . . . . . . . . . . . . . . . . . . . 11 10. Informative References . . . . . . . . . . . . . . . . . . . 12 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 12 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 12 1. Introduction [FAULT-TERMS] defines terminology for benchmarking the fault detection capability of observability systems monitoring LLM serving deployments. This document defines the procedures that produce those measurements. Gaikwad Expires 12 February 2027 [Page 2] Internet-Draft Fault Detection Methodology August 2026 The procedures follow the pattern of [RFC2544] and [RFC8239]. A parameter is varied across trials, the Detector Under Test is observed, and a characteristic of the detector is reported with the conditions under which it was obtained. 1.1. Scope This document specifies test procedures, required test conditions, and a reporting format. It does not specify acceptance thresholds, target values, or service-level objectives. A result obtained by these procedures characterizes a detector. It does not establish whether that detector is adequate for any purpose. The mechanism by which a fault is produced is out of scope. Any mechanism that shifts the declared indicator by the declared magnitude at the declared time is acceptable. Section 3.4 states the properties an injection apparatus must have. Benchmarking described by this document is confined to a test environment. Section 7 states this requirement. 1.2. Applicability A result obtained by these procedures characterizes one Detector Under Test in one declared environment. The serving deployment includes a language model whose output is stochastic, so the baseline against which fault magnitude is expressed is a property of that environment. Section 4.2 requires the baseline to be established for each environment before any fault trial. Two results are comparable when the Observation Point, the DUT configuration, the serving deployment configuration, the request population, and the offered load are the same. Results MUST NOT be compared when any of these differ. Section 6 requires each to be reported so that a reader can determine whether a comparison is valid. The intended uses follow from this. A detector configuration can be compared against another configuration on the same environment. A detector can be tracked across changes to itself or to the deployment it observes. A detector can be characterized before it is relied on. These procedures are not intended to produce values that rank detectors across organizations, and a value obtained here carries no meaning outside the environment in which it was obtained. A full series under these procedures is long. Twenty trials at each combination of Fault Class, magnitude, profile, and load, each followed by a quiet period, together with the search of Section 5.3, Gaikwad Expires 12 February 2027 [Page 3] Internet-Draft Fault Detection Methodology August 2026 occupies substantial test time. A reduced series MAY be run for screening, using fewer trials, fewer load points, or a coarser search resolution. A reduced series MUST be reported as such, with the reductions stated, and MUST NOT be reported as a result under this methodology. 2. Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. Terms defined in [FAULT-TERMS] are used here with their definitions from that document. These include Detector Under Test, Observation Point, Fault Injection Point, Output Behavior Fault, Fault Class, Fault Profile, Fault Onset, Detection Event, False Detection, Baseline Interval, Detection Latency, Detection Coverage, Detection Threshold Magnitude, and Boundary Masking. 3. Test Setup 3.1. Topology The test environment comprises four components. Traffic generator: Offers requests to the serving deployment at a controlled rate from a declared request population. Serving deployment: The LLM serving stack under observation, configured to one of the System Under Test boundaries of [LLM-TERMS]. Fault injection apparatus: Applies Output Behavior Faults at the Fault Injection Point and records Fault Onset. Detector Under Test: The observability system, sampling at the Observation Point and emitting the declared detection signal. The relative position of the Fault Injection Point and the Observation Point MUST be recorded and reported. A fault injected downstream of the Observation Point is not observable by the detector, and trials in that configuration measure only Boundary Masking. Gaikwad Expires 12 February 2027 [Page 4] Internet-Draft Fault Detection Methodology August 2026 3.2. Detector Under Test Configuration The DUT configuration MUST be frozen before the first trial and MUST NOT be changed during a test series. Changing thresholds, sampling rates, judge models, or alerting rules begins a new test series. The following MUST be recorded: the indicators the DUT computes, the method used to compute each, the sampling rate for any sampled indicator, the aggregation window, the alerting thresholds, and the version of any model used as a judge. 3.3. Traffic Generation Offered load MUST be constant within a trial. The request population MUST be constant across all trials in a series. Detection Latency for a rate-valued indicator depends on how quickly evidence accumulates, and evidence accumulates with requests rather than with time. Results are therefore reported against offered load. A test series SHOULD cover at least four load points spanning at least two orders of magnitude, and SHOULD include a load point representative of the intended deployment. Response caching in the serving deployment suppresses the indicator shift a fault is intended to produce. Caching MUST be disabled, or the request population MUST be constructed so that cache hits do not occur. Which approach was used MUST be reported. 3.4. Fault Injection Apparatus The apparatus MUST record Fault Onset with a resolution of one millisecond or finer, and MUST record it independently of any DUT output. The apparatus MUST be able to remove an applied fault and return the deployment to its baseline configuration. The apparatus MUST NOT emit any signal to the DUT other than through the behavior of the serving deployment. An apparatus that notifies the DUT of injection invalidates the measurement. The magnitude actually achieved MUST be verified against the magnitude intended, by measuring the affected indicator at the Fault Injection Point during application. A trial in which the achieved magnitude differs from the intended magnitude by more than ten percent MUST be discarded and repeated. Gaikwad Expires 12 February 2027 [Page 5] Internet-Draft Fault Detection Methodology August 2026 3.5. Clock Synchronization Fault Onset and Detection Event may be recorded by different hosts. Detection Latency is the difference between them, and clock offset between those hosts enters the result directly as error. The maximum clock offset between the injection apparatus and the DUT MUST be measured and reported. The offset MUST be smaller than one percent of the smallest Detection Latency reported in the series. Where this cannot be achieved, the offset MUST be reported as an error bound on every latency value in the series. Where the injection apparatus and the DUT record timestamps on the same host, this requirement does not apply and the arrangement MUST be reported. 4. Prerequisites 4.1. Detection Signal Declaration The signal that constitutes a Detection Event MUST be declared before the first trial. The declaration MUST state the signal type, the component that emits it, and the criterion by which a signal is attributed to an applied fault. The declaration MUST NOT be changed after any trial in the series has been run. Selecting the detection signal after observing results invalidates the series. 4.2. Baseline Characterization Before the first fault trial, a Baseline Interval MUST be run at each load point to be used in the series. The Baseline Interval MUST be at least ten times the DUT aggregation window, and MUST be at least three hundred seconds. For each indicator to be faulted, the mean and standard deviation over the Baseline Interval MUST be recorded. Fault magnitude is reported as an absolute change from this mean. A fault magnitude smaller than twice the baseline standard deviation is not distinguishable from baseline variation. Trials at such magnitudes MAY be run, and results from them MUST be reported with the baseline standard deviation alongside. Gaikwad Expires 12 February 2027 [Page 6] Internet-Draft Fault Detection Methodology August 2026 4.3. Inter-Trial Quiet Period Observability systems commonly implement alert deduplication, flap suppression, and state that persists after a condition clears. A trial begun while the DUT retains state from a previous trial does not measure detection from a resting state. Between trials, the fault MUST be removed and a quiet period MUST be observed. The quiet period MUST be long enough for the DUT to return every affected indicator to within one standard deviation of its baseline mean, and to clear any active alert state. The quiet period used MUST be reported. 4.4. Single Fault Application One Output Behavior Fault MUST be applied at a time, unless the declared attribution criterion of Section 4.1 is able to attribute a signal to a specific fault in the presence of others. Where multiple faults are applied, the attribution criterion MUST be reported. 5. Test Procedures 5.1. Detection Latency *Objective.* Determine the interval from Fault Onset to the Detection Event, for a stated Fault Class, magnitude, Fault Profile, and offered load. *Procedure.* 1. Offer traffic at the selected load. Wait for the quiet period of Section 4.3. 2. Apply the fault. Record Fault Onset. 3. Observe the DUT until a Detection Event attributable to the fault occurs, or until the trial timeout expires. 4. Record the Detection Event time, or record the trial as undetected. 5. Remove the fault. Observe the quiet period. 6. Repeat from step 2 until the required number of trials is complete. *Trial count.* At least twenty trials MUST be run at each combination of Fault Class, magnitude, profile, and load. Gaikwad Expires 12 February 2027 [Page 7] Internet-Draft Fault Detection Methodology August 2026 *Trial timeout.* The timeout MUST be declared before the series and MUST be at least twenty times the DUT aggregation window. A trial reaching the timeout is recorded as undetected and contributes to Detection Coverage. *Reported value.* The median and the ninety-fifth percentile of Detection Latency across detected trials MUST be reported, together with the number of trials and the number undetected. The mean MAY be reported. Values from undetected trials MUST NOT be imputed. 5.2. Detection Coverage *Objective.* Determine the fraction of applied faults for which a Detection Event occurred, for a stated Fault Class, magnitude, profile, and load. *Procedure.* Detection Coverage is computed from the same trials as Section 5.1. No separate trials are required. *Reported value.* The number of trials in which a Detection Event occurred, divided by the number of trials run. The numerator and denominator MUST both be reported. A coverage value MUST NOT be reported without the fault magnitude at which it was obtained. 5.3. Detection Threshold Magnitude *Objective.* Determine the smallest fault magnitude at which the DUT achieves a stated Detection Coverage, for a given Fault Class, profile, and load. *Procedure.* The procedure is a binary search on magnitude, following the pattern of the throughput search in [RFC2544]. 1. Select a target coverage level. Declare it before the search begins. 2. Establish an upper bound magnitude at which coverage meets or exceeds the target, and a lower bound at which it does not. Where the lower bound falls below twice the baseline standard deviation, record that the search floor was reached and stop. 3. Run the trials of Section 5.1 at the midpoint magnitude. 4. If coverage at the midpoint meets or exceeds the target, the midpoint becomes the new upper bound. Otherwise it becomes the new lower bound. Gaikwad Expires 12 February 2027 [Page 8] Internet-Draft Fault Detection Methodology August 2026 5. Repeat from step 3 until the interval between bounds is smaller than the declared resolution. *Reported value.* The upper bound at termination, the target coverage level, the search resolution, and the baseline standard deviation of the affected indicator. Where the search floor was reached, the result MUST be reported as below the floor and MUST NOT be reported as a magnitude. 5.4. False Detection Rate *Objective.* Determine the rate at which the DUT emits the declared detection signal when no fault is applied. *Procedure.* 1. Offer traffic at the selected load with no fault applied. 2. Observe the DUT for the declared observation period. Record every emission of the declared detection signal. 3. Repeat at each load point in the series. *Observation period.* The period MUST be at least the total time spent under fault application across the whole series at that load point. A False Detection Rate measured over a shorter period than the latency trials it accompanies understates the rate. *Reported value.* The count of false detections and the observation period. A Detection Latency or Detection Threshold Magnitude result MUST NOT be published without an accompanying False Detection Rate obtained under the same DUT configuration and load. 5.5. Boundary Masking *Objective.* Determine the reduction in observed fault magnitude between the Fault Injection Point and the Observation Point. *Procedure.* 1. Apply a fault of known magnitude at the Fault Injection Point. 2. Measure the affected indicator at the Fault Injection Point and at the Observation Point over the same interval, using the same method at both points. 3. Remove the fault. Observe the quiet period. Repeat for the required number of trials. Gaikwad Expires 12 February 2027 [Page 9] Internet-Draft Fault Detection Methodology August 2026 *Trial count.* At least twenty trials MUST be run. *Reported value.* One minus the ratio of the magnitude observed at the Observation Point to the magnitude observed at the Fault Injection Point, reported as a median across trials. The measurement method used at both points MUST be reported. Where the two points are the same, Boundary Masking is zero by construction and MUST be reported as not applicable. 6. Reporting Format A result reported under this methodology MUST include the following fields. +=================+=================================+ | Field | Content | +=================+=================================+ | Observation | Model Engine, Application | | Point | Gateway, or Compound System | +-----------------+---------------------------------+ | Fault Injection | Location, and position relative | | Point | to the Observation Point | +-----------------+---------------------------------+ | DUT | Indicators, methods, sampling | | configuration | rates, aggregation window, | | | thresholds, judge model version | +-----------------+---------------------------------+ | Fault Class | As defined in the terminology | | | document | +-----------------+---------------------------------+ | Fault magnitude | Absolute change from baseline | | | mean, in indicator units | +-----------------+---------------------------------+ | Fault Profile | Step, Ramp, or Intermittent, | | | with parameters | +-----------------+---------------------------------+ | Baseline | Mean and standard deviation of | | statistics | the affected indicator | +-----------------+---------------------------------+ | Offered load | Requests per second, per load | | | point | +-----------------+---------------------------------+ | Caching | Disabled, or request population | | | constructed to avoid hits | +-----------------+---------------------------------+ | Trial count | Trials run, and trials | | | undetected | +-----------------+---------------------------------+ Gaikwad Expires 12 February 2027 [Page 10] Internet-Draft Fault Detection Methodology August 2026 | Trial timeout | Seconds | +-----------------+---------------------------------+ | Quiet period | Seconds | +-----------------+---------------------------------+ | Clock offset | Maximum measured offset, or | | | same-host declaration | +-----------------+---------------------------------+ | Detection | Median and 95th percentile, | | Latency | seconds | +-----------------+---------------------------------+ | Detection | Numerator and denominator | | Coverage | | +-----------------+---------------------------------+ | False Detection | Count and observation period | | Rate | | +-----------------+---------------------------------+ | Boundary | Median fraction, or not | | Masking | applicable | +-----------------+---------------------------------+ Table 1: Required reporting fields Results obtained at different Observation Points, under different DUT configurations, or at different offered loads MUST NOT be combined into a single reported value. 7. Security Considerations The procedures in this document deliberately degrade the output behavior of a serving deployment. Applying them to a deployment serving production traffic exposes users to ungrounded, unsafe, or incorrectly refused responses. The test environment MUST be isolated from production traffic. The fault injection apparatus MUST NOT be reachable from a production control plane. The reasoning of [RFC6815] applies to this document. A published Detection Threshold Magnitude states the degradation an operator would not observe. A published False Detection Rate indicates how much noise an adversary would need to generate to exhaust operator attention. Publishers SHOULD consider the disclosure audience for both. 8. IANA Considerations This document has no IANA actions. 9. Normative References Gaikwad Expires 12 February 2027 [Page 11] Internet-Draft Fault Detection Methodology August 2026 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, May 2017, . [FAULT-TERMS] Gaikwad, M., "Benchmarking Terminology for Output Behavior Fault Detection in Large Language Model Serving Systems", Work in Progress, Internet-Draft, draft-gaikwad-llm-fault- detection-terminology, 2026, . [LLM-TERMS] Gaikwad, M., "Benchmarking Terminology for Large Language Model Serving", Work in Progress, Internet-Draft, draft- gaikwad-llm-benchmarking-terminology, 2026, . 10. Informative References [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, March 1999, . [RFC6815] Bradner, S., Dubray, K., McQuaid, J., and A. Morton, "Applicability Statement for RFC 2544: Use on Production Networks Considered Harmful", RFC 6815, November 2012, . [RFC8239] Avramov, L. and J. Rapp, "Data Center Benchmarking Methodology", RFC 8239, August 2017, . Acknowledgements This document follows the methodology structure established by [RFC2544] and applied to data center benchmarking in [RFC8239]. The binary search of Section 5.3 follows the throughput search of [RFC2544]. The author thanks the Benchmarking Methodology Working Group for that foundation. Author's Address Gaikwad Expires 12 February 2027 [Page 12] Internet-Draft Fault Detection Methodology August 2026 Madhava Gaikwad Independent Email: gaikwad.madhav@gmail.com Gaikwad Expires 12 February 2027 [Page 13]