Additional CATS requirements consideration for Service Segmentation-related use cases

Introduction Service segmentation is a service deployment option that splits the service into smaller subtasks which can be executed in parallel or in sequence before the subtasks execution results are aggregated to serve the service request . It is an interesting service deployment option that is widely considered to improve the performance of several services such as AR-VR or Distributed AI Inference which are also key CATS use cases . For example, a recent 3GPP Technical Report on 6G use cases and services describes an XR rendering service that can be implemented as a sequential pipeline of subtasks, including a render engine, engine adaptation, and rendering acceleration. In contrast, an example of parallel service segmentation is parallel Machine Learning (ML) model partitioning for inference , . Specifically, a ML model layer can be divided into multiple smaller partitions, which are executed in parallel. ML model inference, described as Distributed AI Inference in Section 4.6.1 of , can also be segmented by inference phase or by model role. In prefill-decode (PD) disaggregation, the prefilling and decoding phases of an autoregressive model, such as a Large Language Model (LLM), are executed in sequence by different service instances . In speculative decoding, a small draft model and a large target model exchange proposals and verification results iteratively until the response is complete . In all these segmentation cases, subtask may have multiple instances which are deployed across different computing sites. This document analyzes these CATS service segmentation use case examples to discuss the impact of service segmentation deployment method on CATS system design.

Terminology used in this draft This document re-uses the CATS component terminologies which has been defined in . Additional definitions related to service segmentation are: Service subtask: An offering that performs only a partial functionality of the original service. The complete functionality of the original service is achieved by aggregating the results of all its divided service subtasks. Subtask result aggregation may be performed either in parallel or sequentially. Service subtask instance: When a service is segmented into multiple service subtasks, each service subtask might have multiple instances that performs the same partial functionality of the original service. Iterative subtask execution: An execution of a segmented service in which two service subtasks exchange intermediate results repeatedly before a result is returned to the client. The number of exchanges depends on the processed data and is not known when the service subtask instances are selected. Remote service subtask: A service subtask whose instance is located outside the CATS domain, at a location that is not selected by the CATS system. The network path between a service site in the CATS domain and a remote service subtask can nevertheless be measured. In addition to service subtasks that each perform a part of the functionality of the original service, this document also considers service subtasks that cooperate to produce the result of the original service. An example is the draft model in speculative decoding (), which proposes results that another service subtask verifies.

Example 1: AR-VR (XR) Rendering Sequential Subtask Segmentation

3 -> 4 +-----|-----+------+ +-----------------------| CATS** |C-PS |---------------------+ | Underlay | Forwarder |------+ +-------+ | | Infrastructure +-----|-----+ |C-NMA | | | | +-------+ | | +---------------+-----+---------+---------------+ | | Various network latency between different links | | | | | | | | | /-----------\ | /-----------\ | /-----------\ | | +-+-----|/----+---+----\|/----+---+----\|/----+---+----\|-----+--+ | CATS | | CATS | | CATS | | CATS | | Forwarder | | Forwarder | | Forwarder | | Forwarder | +-----|-----+ +-----|-----+ +-----|-----+ +-----|-----+ | | | | +-----|------+ +----|------+ +----|-------+ +---|--------+ |+----------+| |+---------+| |+----------+| |+----------+| || Render || || Render || || Engine || || Render || || Engine || || Engine || ||Adaptation|| ||Accelerate|| |+----------+| |+---------+| |+----------+| |+----------+| +---+---+ | Optimal | | | | Optimal | | Optimal | |C-SMA* | | | | | | | | | +---+---+ |+----------+| | | |+----------+| | | | || Engine || | | || Render || | | | ||Adaptation|| | | ||Accelerate|| | | | |+----------+| | | |+----------+| | | | | | | | | | | | | +-----|------+ +-----|-----+ +-----|------+ +-----|------+ | +----------------+---------------+----------------+-------------+ Service Service Service Service Site 1 Site 2 Site 3 Site 4 ]]> illustrates how a CATS system should perform optimal traffic steering for an XR rendering service deployed as a sequential pipeline of subtasks, including the render engine, engine adaptation, and rendering acceleration. This example is derived from the corresponding use case in . To return the rendered XR object to the client, the XR rendering request must be processed sequentially in the specified order by the three rendering subtasks.

Expected CATS system flow

The client sends an XR rendering request via its connected network to the XR application platform.
The XR application platform determines that the request should be processed by the XR rendering pipeline and forwards the packet via its attached CATS Forwarder.
The CATS Path Selector (CATS-PS) determines the optimal subtask composition of XR rendering pipeline and selects the most suitable instance for each subtask to steer the request. This selection is based on the current status of computing and network resources at the sites hosting the XR rendering subtask instances. For example, in , the sequential pipeline consists of three subtasks: the optimal Render Engine instance is located at Site 1, the optimal Engine Adaptation instance at Site 3, and the optimal Rendering Acceleration instance at Site 4.
The CATS-PS configures the CATS Forwarder with routing information that specifies the required processing order: from Site 1 to Site 3 to Site 4.
The packet is steered through the CATS underlay infrastructure following the specified routing order and is sequentially processed at the designated service sites.
The XR application platform returns the final processed XR rendering result to the client.

Impacts on CATS system design

A CATS system should provide a method to distinguish different CATS candidate paths corresponding to different service subtask instance combinations (different subtask composition or same subtask composition but different subtask instance location)
A CATS system should provide a method to deliver the service request to the determined optimal service subtask instance combination in correct order and correct composition.

Example 2: Distributed AI Inference Distributed AI Inference is one of the CATS use cases (Section 4.6.1 of ). A single ML model inference service can be segmented along different dimensions, which lead to different kinds of subtask execution. This section considers three of them: segmentation by model structure, in which the model is partitioned into slices that are executed in parallel (); segmentation by inference phase, in which the prefilling and decoding phases of an autoregressive model are executed in sequence at different service sites (); and segmentation by model role, in which a draft model and a target model cooperate iteratively (). In all three examples, the client sends an ML inference request to the ML application platform, which forwards it via its attached CATS Forwarder. The C-PS selects the service subtask instances as described below, and the result is returned to the client via the platform.

ML Model Vertical Partitioning Inference Parallel Subtask Segmentation

|Layer|--| Layer |--| Layer | Original ML Model | | | 1 | | 2 (L2)| | 3 (L3) | +-----+ |(L1) | +-------+ +----------+ +-----+ +-----+ +----------+ +-----+ |Split| +-------+ | | |Input|---> | L1 |--|SplitL2|--| Split L3 | ML Model Slice 1 | |\ +-----+ +-------+ +----------+ +-----+ \ Split \ +-----+ > |Split| +-------+ +----------+ | L1 |--|SplitL2|--| Split L3 | ML Model Slice 2 +-----+ +-------+ +----------+ ]]>

illustrates how a CATS system can perform optimal traffic steering for a machine learning (ML) inference service deployed as a parallel pipeline of subtasks, where each subtask corresponds to a vertically partitioned slice of the original ML model. Based on the ML model splitting use cases described in and , shows how an ML model can be vertically partitioned into slices that are executed in parallel to reduce inference response time. The input inference data from the client should be partitioned according to the input dimensions expected by each model slice. These slices then process their respective inputs in parallel, and the resulting outputs are merged to produce the final inference result, which is returned to the client.

The C-PS selects the optimal combination of model slice instances (Slice 1 at Site 1 and Slice 2 at Site 3 in ) and communicates it to the ML application platform.
The platform divides the input according to the selected model slices, forwards the input slices in parallel, and merges the outputs before responding to the client.

ML Model Inference Phase Splitting Sequential Subtask Segmentation

Example of a CATS system in Prefill-Decode Disaggregation case illustrates how a CATS system can perform optimal traffic steering for an inference service of an autoregressive ML model, such as an LLM, deployed with prefill-decode (PD) disaggregation. Prefilling is compute-intensive, since the whole prompt is processed in one step, while decoding is memory-intensive, since the intermediate state of the inference (the KV cache) is held and read for every generated token . Executing the two phases at different service sites, for example a compute-advantaged site for prefilling and a memory-advantaged site for decoding, allows each phase to be placed where the corresponding resource is available, so that the service is segmented into a prefill subtask followed by a decode subtask.

The C-PS selects a prefill subtask instance at a compute-advantaged site and a decode subtask instance at a memory-advantaged site, considering their computing and memory status and the network status between the sites (Site 1 and Site 3 in ).
The prefill subtask instance processes the prompt and produces the first token; the decode subtask instance continues from its state and generates the subsequent tokens.

ML Model Speculative Decoding Iterative Subtask Segmentation

Example of a CATS system in Iterative Service Segmentation case target) | | verdicts (target -> draft) CATS domain | | repeated once per iteration - - - - - - - - - - - - - - -|-|- - - - - - - - - - - - - - - | v +---------------+ | Target Model | +---------------+ Remote Site (outside the CATS domain) ]]> illustrates how a CATS system can perform optimal traffic steering for an inference service of an autoregressive ML model, such as an LLM, that uses speculative decoding. A small draft model proposes a number of candidate tokens in sequence. A large target model verifies all of them in a single step, accepts the longest valid prefix, and provides the token at the first rejected position . The result is the same as the result the target model would produce alone, and the draft model reduces the number of sequential steps of the target model. The draft model is small enough to be instantiated at edge service sites in the CATS domain, whereas the target model requires a well-provisioned site. In this example, the target model is located at a remote site outside the CATS domain, and is therefore a remote service subtask whose placement is out of the scope of the CATS system. The two service subtasks exchange a proposal and a verification result once per iteration, and the number of iterations of a request depends on the generated content: a response of a few hundred tokens may require tens to hundreds of iterations.

The C-PS selects the draft model instance, considering also the path from each candidate site to the remote site hosting the target model, weighted by the expected number of iterations (Site 2 in ).
The draft model and the target model exchange proposals and verification results repeatedly over a path outside the CATS domain, which is not steered by the CATS system.

Impacts on CATS system design

A CATS system should provide a method to distinguish different CATS candidate paths corresponding to different combinations of service subtask instances.
The service subtasks of one service may be characterized by different types of metrics (e.g., computing metrics for prefilling and memory metrics for decoding in ). A CATS system should be able to evaluate each service subtask with the metrics relevant to it.
A CATS system should consider network metrics between service subtask instances and toward remote service subtasks, not only between the client and a service instance. The bandwidth between service sites matters when intermediate results are large (), and a path traversed repeatedly should be weighted by the expected number of iterations, which can only be estimated (). Such repeated traversal is an intended property of the service and is distinct from the routing loops that R10 of requires to be avoided.
A CATS system should coordinate with the segmented service platform entity to pre-process the original request data into the input formats required by the selected parallel subtasks ().
Service subtasks may differ in state. Instance affinity (R14) and service continuity (R16) apply to stateful service subtasks, such as the decode subtask () and the draft model (), for the duration of a request, while stateless service subtasks can be reselected.
Where a subtask modeled as remote, such as the target model of , is instead deployed at several service sites in the CATS domain, the C-PS selects the combination of instances of all subtasks, and the considerations above apply to it as well.

Differences comparison between Normal and Service Segmentation CATS scenarios In the normal CATS scenario:

The CATS system objective is selecting an optimal service instance to serve a service request
Different candidate CATS paths are caused by: service instances' computing and network resources status.
The CATS system delivers the service request to the determined optimal service instance

In the Service Segmentation CATS scenario:

The CATS system objective is selecting an optimal service subtask combination. An optimal combination is composed of the optimal instances of each service subtask. A combination may also include remote service subtasks, whose instances are not selected by the CATS system but have to be taken into account.
Different candidate CATS paths are caused by: service subtask instances' computing and network resources status, and possible different service segmentation variations (e.g. a service is segmented into different number of subtasks). In addition, they are caused by the network status between service subtask instances or toward remote service subtasks and, for iterative subtask execution, by the expected number of iterations.
The CATS system delivers the service request to the determined optimal combination of service subtask instances in correct order (sequential, parallel, or iterative) and subtask composition. Exchanges between service subtasks may be repeated and may leave the CATS domain.

The following table summarizes how the examples of this document are segmented and what each adds to the CATS system design. Summary of the Service Segmentation Examples

Example	Segmented by	Subtask execution	Traffic between subtasks	Added consideration
XR rendering ()	Rendering stage	Sequential	Forwarded in order	Ordered steering
Vertical partitioning ()	Model structure	Parallel	Split and merged	Input partitioning
PD disaggregation ()	Inference phase	Sequential	Large, once per request	Subtask-specific metrics, inter-site bandwidth
Speculative decoding ()	Model role	Iterative	Repeated, may leave the CATS domain	Remote subtask, iteration weighting

CATS system design Consideration points to support Service Segmentation As AR/VR and Distributed AI Inference are among the CATS-supported use cases listed in , the CATS system should also fully support scenarios where service segmentation is applied to these use cases. This section outlines five CATS system design considerations that are not yet addressed in existing CATS WG documents, including the Problem and Requirement document (), the Framework document (), and the Metric Definition document (): - Traffic Steering Objective:

The optimal service instance can be a sequence/parallel pipeline that consists of the optimal instances of each subtask composing the service, instead of a single entity providing the complete service functionality, as assumed in the conventional CATS scenario.
The optimal combination may include a remote service subtask whose instance is not selected by the CATS system. The selection of the service subtask instances in the CATS domain should account for it ().

- Traffic Steering Mechanism:

The CATS system may be required to provide a mechanism to steer service requests in a predetermined sequence, as in the case of sequential service segmentation.
Besides the traffic between the client and a service subtask instance, traffic between service subtask instances can be large () or repeated (). The CATS system may be required to steer such traffic, or at least to account for its path when selecting service subtask instances.

- CATS Metrics Aggregation:

CATS metrics can be aggregated not only by metric category (e.g., computing, networking) but also by individual service subtasks. For instance, the CATS metric representing a candidate combination of subtasks may be derived by aggregating the metrics of its component subtasks.
One possible realization of such metric aggregation is Service Pipeline Metrics.

The model organizes CATS-related measurements into multiple levels of abstraction. Level 0 (L0) metrics capture primitive measurements (e.g., resource, traffic, or system observations). Level 1 (L1) metrics are derived from L0 metrics to represent service-relevant aspects at a finer granularity. Level 2 (L2) metrics summarize the overall capability or suitability of each subtask (or subtask instance) by aggregating its L1 metrics. Finally, a pipeline-level metric (denoted as M3) is computed by aggregating the L2 metrics across all subtasks in a candidate pipeline, yielding a single score that can be used to compare candidate pipelines.
Aggregating metrics at the pipeline level can also reduce control-plane signaling overhead by avoiding the need to disseminate fine-grained metrics for each individual service replica. This consideration becomes increasingly important in large-scale 6G deployments, where the number of service replicas within a metro-area cell may grow to the order of hundreds.
The L1 metrics that characterize a service subtask can differ between the service subtasks of one service, for example computing metrics for a prefill subtask and memory metrics for a decode subtask. L2 metrics allow such service subtasks to be compared within one pipeline.
A pipeline-level metric also depends on the network metrics between service subtask instances, including bandwidth when large amounts of data are transferred, and on the network metrics toward remote service subtasks. Where a path is traversed repeatedly, its contribution should reflect the expected number of traversals; a simple sum of per-subtask metrics does not capture this.

- Metrics toward Remote Service Subtasks:

Network metrics between service sites in the CATS domain and a remote service subtask are needed, although the remote service subtask is not a CATS service instance. Such metrics could be collected by the C-NMA and are subject to the same staleness considerations as other CATS metrics (R7 of ).

- Subtask-specific Affinity and Service Continuity:

Instance affinity (R14) and service continuity (R16) can apply to individual service subtasks of a pipeline rather than to the pipeline as a whole. A stateful service subtask, such as the decode subtask of or the draft model of , requires affinity for the duration of a request, while a stateless service subtask can be reselected. This should be achieved without maintaining per-flow state in network nodes (R15).