YANG Message Keys for Message Broker Integration

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 22 July 2026.¶

Copyright Notice

Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

Nowadays network operators are using machine and human readable YANG [RFC7950] to model their configurations and monitor YANG operational data from their networks according to [Mar24].¶

Most network analytic use cases require real-time data and the delivery of near real-time analytical and actionable insights. This imposes high scalability, resilience and low overhead in the data processing pipeline. Accessing the right data for the right use case with minimal overhead and in the shortest period of time is therefore crucial.¶

Network operators organize their data in a Data Mesh [Deh22] according to [Bod24] where a Message Broker, such as Apache Kafka [Kaf11] or Apache Pulsar [Pul16], facilitates the exchange of Messages among data processing components in topics and subjects. Typically, data is being stored in Message Broker topics for several hours or days to facilitate resilience in the data processing chain and addressed in Subjects depending on Schema, enabling a data consumer to address and re-consume previously consumed data again if previously lost.¶

Dimensional data is structured information in a data store. It uses a model of dimension tables to organize business metrics and their descriptive context. This model, developed by Ralph Kimball [Kim96], simplifies data analysis and reporting by creating denormalized, easy-to-understand structures for quick querying. It is optimized for online analytical processing (OLAP) and data warehouses by using the data taxonomy to scale in partitions and shards. YANG [RFC7950] as a data modelling language facilitates the modelling of dimensional data.¶

An Architecture for YANG-Push to Message Broker Integration [I-D.ietf-nmop-yang-message-broker-integration] specifies an architecture for integrating YANG-Push with Message Brokers for a Data Mesh architecture. Section 4.5 of [I-D.ietf-nmop-yang-message-broker-integration] describes how the notification messages at a YANG-Push Receiver are being transformed to the Message Broker while Section 3 of [I-D.ietf-nmop-message-broker-telemetry-message] specifies to a Message Schema to contextualize telemetry data. However, neither of these documents address how these messages should be indexed in a Message Broker, nor define how topics, partitioning and sharding must be used.¶

Due to this missing dimensional indexing for Message Broker stored YANG data, all YANG data is stored in one single Topic. This leads to a round robin distribution across multiple Partitions where each YANG Schema id is defined as a subject within that topic. Therefore, the entire Topic from all Partitions needs to be consumed first before data selection can be applied. This leads to avoidable data processing overhead which in turn impairs scalability and real-time capabilities, required for certain Network Analytics use cases.¶

YANG telemetry data can be used for several network analytic use cases. Importantly, depending on the use case, only a subset of the subscribed YANG data might be necessary (in time or space). For example, for specific use cases, it is more important to know the current network state, as opposed to have the full series of the state changes over time. In other use cases, instead of consuming data for all network nodes, only a specific network node or network node component requires the YANG monitoring and hence subscription.¶

This document defines how YANG Messages [I-D.ietf-nmop-message-broker-telemetry-message] should be indexed and organized in Message Broker topics by leveraging the network node hostname, the YANG datastore name and a YANG Item Identifier for indexing. Then, a YANG-Push subscription type and YANG Schema name for a Message Broker topic naming scheme is defined to better organize YANG data.¶

Network node hostname, YANG datastore name and subtree and xpath filters are part of "ietf-yang-push-telemetry-message" structured YANG data defined in Section 3 of [I-D.ietf-nmop-message-broker-telemetry-message]. YANG item identifier are derived from subtree and xpath filters respectively from their YANG Schema tree.¶

2. Conventions and Definitions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

3. Solution Design

To identify which network node produced which YANG data into which Message Broker Topic, Partition and Subject, YANG Message Keys and Indexes (Section 3.1.1) are being introduced. These keys enable a deterministic distribution of YANG messages accross Topics and Partitions enabling applications to consume only the needed data from specific topics and partitions.¶

In order to facilitate Message Broker Topic Compaction, a YANG-Push subscription type based topic naming scheme (Section 3.2) is defined. This segregates statistical (Value), State and State change YANG metrics and facilitates a YANG Message Broker Consumer to use the Topic wild card consumption method to select based on YANG-Push subscription type.¶

ietf-interface:interfaces/interface[type='ianaift:ethernetCsmacd']

<get>
  <filter type="subtree">
    <hardware xmlns="urn:ietf:params:xml:ns:yang:ietf-hardware">
      <component>
        <state/>
      </component>
    </hardware>
  </filter>
</get>

project.environment.statistics.yang.ietf-interfaces.interfaces_interface

project.environment.states.yang.*

4. Message Broker Implementations

Topic, Partitioning and Message Keys are generic concepts of Message Brokers. There are two known Message Broker implementations supporting all features described in this document.¶

Properties props = new Properties();
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

try (Admin admin = Admin.create(props)) {
 String topicName = "my-topic";
 int partitions = 12;
 short replicationFactor = 3;
 // Create a compacted topic
 CreateTopicsResult result = admin.createTopics(Collections.singleton(
  new NewTopic(topicName, partitions, replicationFactor)
   .configs(Collections.singletonMap(TopicConfig.CLEANUP_POLICY_CONFIG,/
   TopicConfig.CLEANUP_POLICY_COMPACT))));

 // Call values() to get the result for a specific topic
 // KafkaFuture<Void> future = result.values().get(topicName);

 // Call get() to block until the topic creation is complete or has
 // failed if creation failed the ExecutionException wraps the
 // underlying cause. future.get();
}

5. Time Series Database Implementations

Tables, partition and keys are generic concepts of time series databases. With ClickHouse, this document examples how YANG Message Keys can be obtained from Message Broker and how it can used for indexing.¶

          CREATE TABLE queue (
              timestamp UInt64,
              level String,
              message String
          )
          ENGINE = Kafka
          SETTINGS kafka_broker_list = 'localhost:9092',
              kafka_topic_list = 'topic',
              kafka_group_name = 'group1',
              kafka_format = 'JSONEachRow',
              kafka_num_consumers = 4;

          CREATE TABLE messages (
              key String,
              timestamp UInt64,
              level String,
              message String
          )
          ENGINE = MergeTree
          ORDER BY (key, timestamp);

          CREATE MATERIALIZED VIEW mv_messages TO messages AS
          SELECT
              _key AS key,
              timestamp,
              level,
              message
          FROM queue;

6. IANA Considerations

This document includes no request to IANA.¶

7. Security Considerations

This document should not affect the security of the Internet.¶

8. Operational Considerations

The YANG Message Broker Producer of a YANG-Push receiver should have three config knobs facilitate the features described in this document as optional:¶

• Topic Distribution: Select between "topic" and "subject" distribution. Default is subject to remain backward compatibility to [I-D.ietf-nmop-yang-message-broker-integration].¶
• Distribution Type: Select between "none" and "YANG-Push subscription type".¶
• YANG Message Key: Select between "enable" and "disable".¶

Subject distribution enables message ordering for a set of YANG Message Keys on each partition. Where in topic distribution messages are randomly being distributed among partitions.¶

To accommodate for potential date loss throughout the data processing pipeline, periodical update of the current State for State metrics is RECOMMENDED. This can be accommodated with YANG-Push as defined in [RFC8641] by complementing "on-change sync on start" subscriptions with periodical subscriptions. Alternatively, in YANG-Push Lite defined in Section 7.6 of [I-D.wilton-netconf-yang-push-lite] this simplified in one subscription.¶

Acknowledgements

Thanks to Camilo Cardona, Rob Wilton, Holger Keller, Reshad Rahman and Nigel Davis for their comments and reviews.¶

We also like to thank Victor Lopez for the initial idea on the network controller use case. Ashley Woods, Sivakumar Sundaravadivel and Rafael Julio for the idea of grouping topics by YANG-Push subscription type and insisting that Topic Compaction is a key enabler for inventory metrics and YANG data consumer integration and should be supported day 1. Nigel Davis for confirming that Topic Compaction simplifies indeed data processing system architecture and Loïc Monney for the operational configuration and monitoring details on Apache Kafka.¶

Contributors

Many thanks goes to Hellmar Becker who contributed Section 3.1.3 and Section 5 on how YANG Message Keys can be obtained from Message Broker, how time series databases can use it for indexing YANG data and example implementation in ClickHouse.¶

Authors' Addresses