Internet-Draft cats-req-service-segmentation October 2026
Bui, et al. Expires 16 April 2027 [Page]
Workgroup:
cats
Internet-Draft:
draft-dcn-cats-req-service-segmentation-05
Published:
Intended Status:
Informational
Expires:
Authors:
D. K. Bui
Soongsil University
N. Tran
Soongsil University
K. Nguyen-Trung
Soongsil University
Y. Kim
Soongsil University

Additional CATS requirements consideration for Service Segmentation-related use cases

Abstract

This document discusses possible additional CATS requirements when considering service segmentation in related CATS use cases such as AR-VR and Distributed AI Inference

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 16 April 2027.

▲

Table of Contents

1. Introduction

Service segmentation is a service deployment option that splits the service into smaller subtasks which can be executed in parallel or in sequence before the subtasks execution results are aggregated to serve the service request [draft-li-cats-task-segmentation-framework]. It is an interesting service deployment option that is widely considered to improve the performance of several services such as AR-VR or Distributed AI Inference which are also key CATS use cases [draft-ietf-cats-usecases-requirements].

For example, a recent 3GPP Technical Report on 6G use cases and services [TR-22870-3GPP] describes an XR rendering service that can be implemented as a sequential pipeline of subtasks, including a render engine, engine adaptation, and rendering acceleration. In contrast, an example of parallel service segmentation is parallel Machine Learning (ML) model partitioning for inference [SplitPlace], [Gillis]. Specifically, a ML model layer can be divided into multiple smaller partitions, which are executed in parallel. ML model inference, described as Distributed AI Inference in Section 4.6.1 of [draft-ietf-cats-usecases-requirements], can also be segmented by inference phase or by model role. In prefill-decode (PD) disaggregation, the prefilling and decoding phases of an autoregressive model, such as a Large Language Model (LLM), are executed in sequence by different service instances [DistServe] [Splitwise]. In speculative decoding, a small draft model and a large target model exchange proposals and verification results iteratively until the response is complete [Leviathan] [Chen]. In all these segmentation cases, subtask may have multiple instances which are deployed across different computing sites.

This document analyzes these CATS service segmentation use case examples to discuss the impact of service segmentation deployment method on CATS system design.

2. Terminology used in this draft

This document re-uses the CATS component terminologies which has been defined in [draft-ietf-cats-framework]. Additional definitions related to service segmentation are:

Service subtask: An offering that performs only a partial functionality of the original service. The complete functionality of the original service is achieved by aggregating the results of all its divided service subtasks. Subtask result aggregation may be performed either in parallel or sequentially.

Service subtask instance: When a service is segmented into multiple service subtasks, each service subtask might have multiple instances that performs the same partial functionality of the original service.

Iterative subtask execution: An execution of a segmented service in which two service subtasks exchange intermediate results repeatedly before a result is returned to the client. The number of exchanges depends on the processed data and is not known when the service subtask instances are selected.

Remote service subtask: A service subtask whose instance is located outside the CATS domain, at a location that is not selected by the CATS system. The network path between a service site in the CATS domain and a remote service subtask can nevertheless be measured.

In addition to service subtasks that each perform a part of the functionality of the original service, this document also considers service subtasks that cooperate to produce the result of the original service. An example is the draft model in speculative decoding (Section 4.3), which proposes results that another service subtask verifies.

3. Example 1: AR-VR (XR) Rendering Sequential Subtask Segmentation 

                      XR Rendering request
                          +--------+
                          | Client |
                          +---|----+
                              |
                      +-------|-------+
                      |   AR-VR(XR)   |
                      |  App Platform |
                      +-------|-------+
                              |  Supposed Optimal combination:
                              |  RE Site 1, EA Site 3, RA site 4
                              |
                              |  Forwards packet in ORDER:
                              |  Site 1 -> 3 -> 4
                        +-----|-----+------+
+-----------------------|   CATS**  |C-PS  |---------------------+
|       Underlay        | Forwarder |------+          +-------+  |
|    Infrastructure     +-----|-----+                 |C-NMA  |  |
|                             |                       +-------+  |
|       +---------------+-----+---------+---------------+        |
|        Various network latency between different links         |
|       |               |               |               |        |
|       | /-----------\ | /-----------\ | /-----------\ |        |
+-+-----|/----+---+----\|/----+---+----\|/----+---+----\|-----+--+
  |   CATS    |   |  CATS     |   |   CATS    |   |   CATS    |
  | Forwarder |   | Forwarder |   | Forwarder |   | Forwarder |
  +-----|-----+   +-----|-----+   +-----|-----+   +-----|-----+
        |               |               |               |
  +-----|------+   +----|------+   +----|-------+   +---|--------+
  |+----------+|   |+---------+|   |+----------+|   |+----------+|
  ||  Render  ||   || Render  ||   || Engine   ||   ||  Render  ||
  ||  Engine  ||   || Engine  ||   ||Adaptation||   ||Accelerate||
  |+----------+|   |+---------+|   |+----------+|   |+----------+|  +---+---+
  |  Optimal   |   |           |   |  Optimal   |   |  Optimal   |  |C-SMA* |
  |            |   |           |   |            |   |            |  +---+---+
  |+----------+|   |           |   |+----------+|   |            |      |
  || Engine   ||   |           |   || Render   ||   |            |      |
  ||Adaptation||   |           |   ||Accelerate||   |            |      |
  |+----------+|   |           |   |+----------+|   |            |      |
  |            |   |           |   |            |   |            |      |
  +-----|------+   +-----|-----+   +-----|------+   +-----|------+      |
        +----------------+---------------+----------------+-------------+
     Service         Service         Service        Service
      Site 1          Site 2          Site 3         Site 4
Figure 1: Example of a CATS system in Sequential Service Segmentation case

Figure 1 illustrates how a CATS system should perform optimal traffic steering for an XR rendering service deployed as a sequential pipeline of subtasks, including the render engine, engine adaptation, and rendering acceleration. This example is derived from the corresponding use case in [TR-22870-3GPP]. To return the rendered XR object to the client, the XR rendering request must be processed sequentially in the specified order by the three rendering subtasks.

3.1. Expected CATS system flow

  • The client sends an XR rendering request via its connected network to the XR application platform.
  • The XR application platform determines that the request should be processed by the XR rendering pipeline and forwards the packet via its attached CATS Forwarder.
  • The CATS Path Selector (CATS-PS) determines the optimal subtask composition of XR rendering pipeline and selects the most suitable instance for each subtask to steer the request. This selection is based on the current status of computing and network resources at the sites hosting the XR rendering subtask instances. For example, in Figure 1, the sequential pipeline consists of three subtasks: the optimal Render Engine instance is located at Site 1, the optimal Engine Adaptation instance at Site 3, and the optimal Rendering Acceleration instance at Site 4.
  • The CATS-PS configures the CATS Forwarder with routing information that specifies the required processing order: from Site 1 to Site 3 to Site 4.
  • The packet is steered through the CATS underlay infrastructure following the specified routing order and is sequentially processed at the designated service sites.
  • The XR application platform returns the final processed XR rendering result to the client.

3.2. Impacts on CATS system design

  • A CATS system should provide a method to distinguish different CATS candidate paths corresponding to different service subtask instance combinations (different subtask composition or same subtask composition but different subtask instance location)
  • A CATS system should provide a method to deliver the service request to the determined optimal service subtask instance combination in correct order and correct composition.

4. Example 2: Distributed AI Inference

Distributed AI Inference is one of the CATS use cases (Section 4.6.1 of [draft-ietf-cats-usecases-requirements]). A single ML model inference service can be segmented along different dimensions, which lead to different kinds of subtask execution. This section considers three of them: segmentation by model structure, in which the model is partitioned into slices that are executed in parallel (Section 4.1); segmentation by inference phase, in which the prefilling and decoding phases of an autoregressive model are executed in sequence at different service sites (Section 4.2); and segmentation by model role, in which a draft model and a target model cooperate iteratively (Section 4.3).

In all three examples, the client sends an ML inference request to the ML application platform, which forwards it via its attached CATS Forwarder. The C-PS selects the service subtask instances as described below, and the result is returned to the client via the platform.

4.1. ML Model Vertical Partitioning Inference Parallel Subtask Segmentation 

             +-----+             +----------+
 +-----+     |     |  +-------+  |          |
 |Input|---> |Layer|--| Layer |--|   Layer  |   Original ML Model
 |     |     |  1  |  | 2 (L2)|  |   3 (L3) |
 +-----+     |(L1) |  +-------+  +----------+
             +-----+



             +-----+             +----------+
 +-----+     |Split|  +-------+  |          |
 |Input|---> |  L1 |--|SplitL2|--| Split L3 |   ML Model Slice 1
 |     |\    +-----+  +-------+  +----------+
 +-----+ \
  Split   \  +-----+
           > |Split|  +-------+  +----------+
             |  L1 |--|SplitL2|--| Split L3 |   ML Model Slice 2
             +-----+  +-------+  +----------+
Figure 2: ML model Vertical Partitioning Illustration 
                      ML Inference request
                          +--------+
                          | Client |
                          +---|----+
                              |
                      +-------|-------+
   *Merges output from|       ML      | *Divides input corresponding
    Slice 1 and 2     |  App Platform |  to Slice 1, 2 input sizes
    before responding +-------|-------+
    to Client                 |
                              |
                              |  Supposed Optimal combination:
                              |  Slice 1 Site 1, Slice 2 Site 3
                              |
                              |  Forwards packet in PARALLEL:
                              |  Site 1 & 3
                        +-----|-----+------+
+-----------------------|   CATS**  |C-PS  |---------------------+
|       Underlay        | Forwarder |------+          +-------+  |
|    Infrastructure     +-----|-----+                 |C-NMA  |  |
|                             |                       +-------+  |
|       +---------------+-----+---------+---------------+        |
|        Various network latency between different links         |
|       |               |               |               |        |
|       | /-----------\ | /-----------\ | /-----------\ |        |
+-+-----|/----+---+----\|/----+---+----\|/----+---+----\|-----+--+
  |   CATS    |   |  CATS     |   |   CATS    |   |   CATS    |
  | Forwarder |   | Forwarder |   | Forwarder |   | Forwarder |
  +-----|-----+   +-----|-----+   +-----|-----+   +-----|-----+
        |               |               |               |
  +-----|------+   +----|------+   +----|-------+   +---|--------+
  |+----------+|   |+---------+|   |+----------+|   |+----------+|
  ||  Model   ||   || Model   ||   ||  Model   ||   ||  Model   ||
  ||  Slice1  ||   || Slice 1 ||   ||  Slice 2 ||   ||  Slice 2 ||
  |+----------+|   |+---------+|   |+----------+|   |+----------+|  +---+---+
  |  Optimal   |   |           |   |  Optimal   |   |            |  |C-SMA* |
  |            |   |           |   |            |   |            |  +---+---+
  |            |   |           |   |            |   |            |      |
  +-----|------+   +-----|-----+   +-----|------+   +-----|------+      |
        +----------------+---------------+----------------+-------------+
     Service         Service         Service        Service
      Site 1          Site 2          Site 3         Site 4
Figure 3: Example of a CATS system in Parallel Service Segmentation case

Figure 3 illustrates how a CATS system can perform optimal traffic steering for a machine learning (ML) inference service deployed as a parallel pipeline of subtasks, where each subtask corresponds to a vertically partitioned slice of the original ML model. Based on the ML model splitting use cases described in [SplitPlace] and [Gillis], Figure 2 shows how an ML model can be vertically partitioned into slices that are executed in parallel to reduce inference response time. The input inference data from the client should be partitioned according to the input dimensions expected by each model slice. These slices then process their respective inputs in parallel, and the resulting outputs are merged to produce the final inference result, which is returned to the client.

  • The C-PS selects the optimal combination of model slice instances (Slice 1 at Site 1 and Slice 2 at Site 3 in Figure 3) and communicates it to the ML application platform.
  • The platform divides the input according to the selected model slices, forwards the input slices in parallel, and merges the outputs before responding to the client.

4.2. ML Model Inference Phase Splitting Sequential Subtask Segmentation 

                         ML inference request
                               +--------+
                               | Client |
                               +--------+
                                   |
                           +---------------+
                           |      ML       |
                           | App Platform  |
                           +---------------+
                                   |
                                   |  Supposed optimal combination:
                                   |  Prefill Site 1, Decode Site 3
                                   |
                             +-----------+------+
    +------------------------|   CATS    | C-PS |------------------+
    |    Underlay            | Forwarder |------+    +-------+     |
    | Infrastructure         +-----------+           | C-NMA |     |
    |                              |                 +-------+     |
    |                              |                               |
    |      +---------------+-------+-------+---------------+       |
    +------+---------------+---------------+---------------+-------+
           |               |               |               |
     +-----------+   +-----------+   +-----------+   +-----------+
     |   CATS    |   |   CATS    |   |   CATS    |   |   CATS    |
     | Forwarder |   | Forwarder |   | Forwarder |   | Forwarder |
     +-----------+   +-----------+   +-----------+   +-----------+
           |               |               |               |
     +-----------+   +-----------+   +-----------+   +-----------+
     |+---------+|   |+---------+|   |+---------+|   |+---------+|
     || Prefill ||   || Prefill ||   || Decode  ||   || Decode  ||
     |+---------+|   |+---------+|   |+---------+|   |+---------+|
     |  Optimal  |   |           |   |  Optimal  |   |           |
     +-----------+   +-----------+   +-----------+   +-----------+

        Service         Service         Service         Service
        Site 1          Site 2          Site 3          Site 4
       Compute-        Compute-         Memory-         Memory-
      advantaged      advantaged      advantaged      advantaged
Figure 4: Example of a CATS system in Prefill-Decode Disaggregation case

Figure 4 illustrates how a CATS system can perform optimal traffic steering for an inference service of an autoregressive ML model, such as an LLM, deployed with prefill-decode (PD) disaggregation. Prefilling is compute-intensive, since the whole prompt is processed in one step, while decoding is memory-intensive, since the intermediate state of the inference (the KV cache) is held and read for every generated token [DistServe] [Splitwise]. Executing the two phases at different service sites, for example a compute-advantaged site for prefilling and a memory-advantaged site for decoding, allows each phase to be placed where the corresponding resource is available, so that the service is segmented into a prefill subtask followed by a decode subtask.

  • The C-PS selects a prefill subtask instance at a compute-advantaged site and a decode subtask instance at a memory-advantaged site, considering their computing and memory status and the network status between the sites (Site 1 and Site 3 in Figure 4).
  • The prefill subtask instance processes the prompt and produces the first token; the decode subtask instance continues from its state and generates the subsequent tokens.

4.3. ML Model Speculative Decoding Iterative Subtask Segmentation 

                         ML inference request
                               +--------+
                               | Client |
                               +--------+
                                   |
                           +---------------+
                           |      ML       |
                           | App Platform  |
                           +---------------+
                                   |
                                   |  Supposed optimal instance:
                                   |  Draft model at Site 2
                                   |
                             +-----------+------+
    +------------------------|   CATS    | C-PS |------------------+
    |    Underlay            | Forwarder |------+    +-------+     |
    | Infrastructure         +-----------+           | C-NMA |     |
    |                              |                 +-------+     |
    |                              |                               |
    |              +---------------+---------------+               |
    +--------------+---------------+---------------+---------------+
                   |               |               |
             +-----------+   +-----------+   +-----------+
             |   CATS    |   |   CATS    |   |   CATS    |
             | Forwarder |   | Forwarder |   | Forwarder |
             +-----------+   +-----------+   +-----------+
                   |               |               |
             +-----------+   +-----------+   +-----------+
             |  Site 1   |   |  Site 2   |   |  Site 3   |
             |+---------+|   |+---------+|   |+---------+|
             ||  Draft  ||   ||  Draft  ||   ||  Draft  ||
             ||  Model  ||   ||  Model  ||   ||  Model  ||
             |+---------+|   |+---------+|   |+---------+|
             |           |   |  Optimal  |   |           |
             +-----------+   +-----------+   +-----------+
                                  ^ |
                                  | | proposals (draft -> target)
                                  | | verdicts  (target -> draft)
       CATS domain                | | repeated once per iteration
     - - - - - - - - - - - - - - -|-|- - - - - - - - - - - - - - -
                                  | v
                           +---------------+
                           | Target Model  |
                           +---------------+
                             Remote Site (outside the CATS domain)
Figure 5: Example of a CATS system in Iterative Service Segmentation case

Figure 5 illustrates how a CATS system can perform optimal traffic steering for an inference service of an autoregressive ML model, such as an LLM, that uses speculative decoding. A small draft model proposes a number of candidate tokens in sequence. A large target model verifies all of them in a single step, accepts the longest valid prefix, and provides the token at the first rejected position [Leviathan] [Chen]. The result is the same as the result the target model would produce alone, and the draft model reduces the number of sequential steps of the target model. The draft model is small enough to be instantiated at edge service sites in the CATS domain, whereas the target model requires a well-provisioned site. In this example, the target model is located at a remote site outside the CATS domain, and is therefore a remote service subtask whose placement is out of the scope of the CATS system. The two service subtasks exchange a proposal and a verification result once per iteration, and the number of iterations of a request depends on the generated content: a response of a few hundred tokens may require tens to hundreds of iterations.

  • The C-PS selects the draft model instance, considering also the path from each candidate site to the remote site hosting the target model, weighted by the expected number of iterations (Site 2 in Figure 5).
  • The draft model and the target model exchange proposals and verification results repeatedly over a path outside the CATS domain, which is not steered by the CATS system.

4.4. Impacts on CATS system design

  • A CATS system should provide a method to distinguish different CATS candidate paths corresponding to different combinations of service subtask instances.
  • The service subtasks of one service may be characterized by different types of metrics (e.g., computing metrics for prefilling and memory metrics for decoding in Section 4.2). A CATS system should be able to evaluate each service subtask with the metrics relevant to it.
  • A CATS system should consider network metrics between service subtask instances and toward remote service subtasks, not only between the client and a service instance. The bandwidth between service sites matters when intermediate results are large (Section 4.2), and a path traversed repeatedly should be weighted by the expected number of iterations, which can only be estimated (Section 4.3). Such repeated traversal is an intended property of the service and is distinct from the routing loops that R10 of [draft-ietf-cats-usecases-requirements] requires to be avoided.
  • A CATS system should coordinate with the segmented service platform entity to pre-process the original request data into the input formats required by the selected parallel subtasks (Section 4.1).
  • Service subtasks may differ in state. Instance affinity (R14) and service continuity (R16) apply to stateful service subtasks, such as the decode subtask (Section 4.2) and the draft model (Section 4.3), for the duration of a request, while stateless service subtasks can be reselected.
  • Where a subtask modeled as remote, such as the target model of Section 4.3, is instead deployed at several service sites in the CATS domain, the C-PS selects the combination of instances of all subtasks, and the considerations above apply to it as well.

5. Differences comparison between Normal and Service Segmentation CATS scenarios

In the normal CATS scenario:

In the Service Segmentation CATS scenario:

The following table summarizes how the examples of this document are segmented and what each adds to the CATS system design.

Table 1: Summary of the Service Segmentation Examples
Example Segmented by Subtask execution Traffic between subtasks Added consideration
XR rendering (Section 3) Rendering stage Sequential Forwarded in order Ordered steering
Vertical partitioning (Section 4.1) Model structure Parallel Split and merged Input partitioning
PD disaggregation (Section 4.2) Inference phase Sequential Large, once per request Subtask-specific metrics, inter-site bandwidth
Speculative decoding (Section 4.3) Model role Iterative Repeated, may leave the CATS domain Remote subtask, iteration weighting

6. CATS system design Consideration points to support Service Segmentation

As AR/VR and Distributed AI Inference are among the CATS-supported use cases listed in [draft-ietf-cats-usecases-requirements], the CATS system should also fully support scenarios where service segmentation is applied to these use cases.

This section outlines five CATS system design considerations that are not yet addressed in existing CATS WG documents, including the Problem and Requirement document ([draft-ietf-cats-usecases-requirements]), the Framework document ([draft-ietf-cats-framework]), and the Metric Definition document ([draft-ietf-cats-usecases-requirements]):

- Traffic Steering Objective:

- Traffic Steering Mechanism:

- CATS Metrics Aggregation:

- Metrics toward Remote Service Subtasks:

- Subtask-specific Affinity and Service Continuity:

7. Normative References

[Chen]
Chen, C., et al., "Accelerating Large Language Model Decoding with Speculative Sampling", arXiv 2302.01318, .
[DistServe]
Zhong, Y., et al., "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving", USENIX OSDI 2024, .
[draft-ietf-cats-framework]
Li, C., et al., "A Framework for Computing-Aware Traffic Steering (CATS)", draft-ietf-cats-framework, .
[draft-ietf-cats-metric-definition]
Yao, K., et al., "CATS Metrics Definition", draft-ietf-cats-metric-definition, .
[draft-ietf-cats-usecases-requirements]
Yao, K., et al., "Computing-Aware Traffic Steering (CATS) Problem Statement, Use Cases, and Requirements", draft-ietf-cats-usecases-requirements, .
[draft-ietf-spring-sr-service-programming]
Abdelsalam, A. , et al., "Service Programming with Segment Routing", draft-ietf-spring-sr-service-programming, .
[draft-lbdd-cats-dp-sr]
Li, C., et al., "Computing-Aware Traffic Steering (CATS) Using Segment Routing", draft-lbdd-cats-dp-sr, .
[draft-li-cats-task-segmentation-framework]
Li, C., et al., "A Task Segmentation Framework for Computing-Aware Traffic Steering", draft-li-cats-task-segmentation-framework, .
[Gillis]
Yu, M., Jiang, Z., Chun Ng, H., Wang, W., Chen, R., and B. Li, "Gillis: Serving Large Neural Networks in Serverless Functions with Automatic Model Partitioning", , <https://doi.org/10.1109/ICDCS51616.2021.00022>.
[Leviathan]
Leviathan, Y., Kalman, M., and Y. Matias, "Fast Inference from Transformers via Speculative Decoding", ICML 2023, .
[SplitPlace]
Tuli, S., Casale, G., and N. Jennings, "SplitPlace: AI Augmented Splitting and Placement of Large-Scale Neural Networks in Mobile Edge Environments", , <https://doi.org/10.1109/TMC.2022.3177569>.
[Splitwise]
Patel, P., et al., "Splitwise: Efficient Generative LLM Inference Using Phase Splitting", ACM/IEEE ISCA 2024, .
[TR-22870-3GPP]
"Study on 6G Use Cases and Service Requirements", , <https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=4374>.

Authors' Addresses

Duong Kham Bui
Soongsil University
369, Sangdo-ro, Dongjak-gu
Seoul
06978
Republic of Korea
Phone: +82 10 5812 2230
Minh-Ngoc Tran
Soongsil University
369, Sangdo-ro, Dongjak-gu
Seoul
06978
Republic of Korea
Kiem Nguyen Trung
Soongsil University
369, Sangdo-ro, Dongjak-gu
Seoul
06978
Republic of Korea
Younghan Kim
Soongsil University
369, Sangdo-ro, Dongjak-gu
Seoul
06978
Republic of Korea
Phone: +82 10 2691 0904