-
Notifications
You must be signed in to change notification settings - Fork 0
/
draft-zhou-netconf-multi-stream-originators-00.txt
560 lines (369 loc) · 21.5 KB
/
draft-zhou-netconf-multi-stream-originators-00.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
Network Working Group T. Zhou
Internet-Draft G. Zheng
Intended status: Standards Track Huawei
Expires: April 14, 2018 E. Voit
Cisco Systems
A. Clemm
Huawei
A. Bierman
YumaWorks
October 11, 2017
Subscription to Multiple Stream Originators
draft-zhou-netconf-multi-stream-originators-00
Abstract
This document describes the distributed data collection mechanism
that allows multiple data streams to be managed using a single
subscription. Specifically, multiple data streams are pushed
directly to the collector without passing through a broker for
internal consolidation.
Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 14, 2018.
Zhou, et al. Expires April 14, 2018 [Page 1]
Internet-Draft Multiple Steam Originators October 2017
Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1. Use Case 1: Data Collection from Devices with Main-board
and Line-cards . . . . . . . . . . . . . . . . . . . . . 3
2.2. Use Case 2: IoT Data Collection . . . . . . . . . . . . . 4
3. Solution Overview . . . . . . . . . . . . . . . . . . . . . . 5
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7
5. Security Considerations . . . . . . . . . . . . . . . . . . . 7
6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 7
7.1. Normative References . . . . . . . . . . . . . . . . . . 7
7.2. Informative References . . . . . . . . . . . . . . . . . 8
Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 8
Appendix B. Subscription Decomposition . . . . . . . . . . . . . 8
Appendix C. Publication Composition . . . . . . . . . . . . . . 9
Appendix D. Examples . . . . . . . . . . . . . . . . . . . . . . 10
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10
1. Introduction
Streaming telemetry refers to sending a continuous stream of
operational data from a device to a remote receiver. This provides
an ability to monitor a network from remote and to provide network
analytics. Devices generate telemetry data and push that data to a
collector for further analysis. By streaming the data, much better
performance, finer-grained sampling, monitoring accuracy, and
bandwidth utilization can be achieved than with polling-based
alternatives.
YANG-Push [I-D.ietf-netconf-yang-push] defines a transport-
independent subscription mechanism for datastore updates, in which a
Zhou, et al. Expires April 14, 2018 [Page 2]
Internet-Draft Multiple Steam Originators October 2017
subscriber can subscribe to a stream of datastore updates from a
server, or update provider. The current design involves subscription
to a single push server. This conceptually centralized model
encounters efficiency limitations in cases where the data sources are
themselves distributed, such as line cards in a piece of network
equipment. In such cases, it will be a lot more efficient to have
each data source (e.g., each line card) originate its own stream of
updates, rather than requiring updates to be tunneled through a
central server where they are combined. What is needed is a
distributed mechanism that allows to directly push multiple
individual data substreams, without needing to first pass them
through an additional processing stage for internal consolidation,
but still allowing those substreams to be managed and controlled via
a single subscription.
This document will describe such distributed data collection
mechanism and how it can work by extending existing YANG-Push
mechanism. The proposal is general enough to fit many scenarios.
2. Use Cases
2.1. Use Case 1: Data Collection from Devices with Main-board and Line-
cards
For data collection from devices with main-board and line-cards,
existing YANG-Push solutions consider only one push server typically
reside in the main board. As shown in the following figure, data are
collected from line cards and aggregate to the main board as one
consolidated stream. So the main board can easily become the
performance bottle-neck. The optimization is to apply the
distributed data collection mechanism which can directly push data
from line cards to a collector. On one hand, this will reduce the
cost of scarce compute and memory resources on the main board for
data processing and assembling. On the other hand, distributed data
push can off-load the streaming traffic to multiple interfaces.
Zhou, et al. Expires April 14, 2018 [Page 3]
Internet-Draft Multiple Steam Originators October 2017
+-------------------------------------+
| collector |
+------^-----------^-----------^------+
| | |
| | |
+-------------------------------------+
| | | | |
| | +-----+------+ | |
| | | main board | | |
| | +--^-----^---+ | |
| | | | | |
| | +---+ +---+ | |
| | | | | |
| +----+----+---+ +---+----+----+ |
| | line card 1 | | line card 2 | |
| +-------------+ +-------------+ |
| device |
+-------------------------------------+
Fig. 1 Data Collection from Devices with Main-board and Line-cards
2.2. Use Case 2: IoT Data Collection
In the IoT data collection scenario, as shown in the following
figure, collector usually cannot access to IoT nodes directly, but is
isolated by the border router. So the collector subscribes data from
the border router, and let the border router to disassemble the
subscription to corresponding IoT nodes. The border router is
typically the traffic convergence point. It's intuitive to treat the
border router as a broker assembling the data collected from the IoT
nodes and forwarding to the collector[I-D.ietf-core-coap-pubsub].
However, the border router is not so powerful on data assembling as a
network device. It's more efficient for the collector, which may be
a server or even a cluster, to assemble the subscribed data if
possible. In this case, push servers that reside in IoT nodes can
stream data to the collector directly while traffic only passes
through the border router.
Zhou, et al. Expires April 14, 2018 [Page 4]
Internet-Draft Multiple Steam Originators October 2017
+-------------------------------+
| collector |
+---^-----------^------------^--+
| | |
| | |
| | |
| +-------+--------+ |
| | border router | |
| +----^------^----+ |
| | | |
| | | |
| +---+ +---+ |
| | | |
+---+----+---+ +---+----+---+
| IoT node 1 | | IoT node 2 |
+------------+ +------------+
Fig. 2 IoT Data Collection
3. Solution Overview
All the use cases described in the previous section are very similar
on the data subscription and publication mode, hence can be
abstracted to the following generic distributed data collection
framework, as shown in the following figure.
A Collector usually includes two components,
o the Subscriber generates the subscription instructions to express
what and how the collector want to receive the data;
o the Receiver is the target for the data publication.
For one subscription, there may be one to many receivers. And the
subscriber does not necessarily share the same address with
receivers.
In this framework, the stream originators have the Master role and
the Agent role. Both the Master and the Agent include two
components,
o the Subscription Server manages capabilities that it can provide
to the subscriber.
o the Publication Server pushes data to the receiver according to
the subscription information.
Zhou, et al. Expires April 14, 2018 [Page 5]
Internet-Draft Multiple Steam Originators October 2017
The Master knows all the capabilities that the attached Agents and
itself can provide, and exposes the global capability to the
Collector. The Collector cannot see the Agents directly, so it will
only send the subscription information to the Master. The Master
disassembles the subscription to multiple component subscriptions,
each involving data from a separate telemetry source. The component
subscriptions are then distributed to the corresponding Agents.
When data streaming, the Publication Server located in each stream
originator collects and encapsulates the packets per the component
subscription, and pushes the piece of data which it can serve
directly to the designated data Collector. The Collector is able to
assemble many pieces of data associated with one subscription, and
can also deduce the missing pieces of data.
+-------------------------------------+
| Collector |
| +------------+ +------------+ |
| | Subscriber | | Receiver <-------+
| +-^----+-----+ +------^-----+ | |
| | | | | |
+-------------------------------------+ |
capability | |subscription | push |
| | | |
+-------------------------------------+ |
| | | Master | | |
| +--+----v------+ +------+------+ | |
| | Subscription | | Publication | | |
| | Server | | Server | | |
| +--^----+------+ +-------------+ | |
| | | | |
+-------------------------------------+ |
component | | component push |
capability | | subscription |
+-------------------------------------+ |
| | | Agent | |
| +--+----v------+ +-------------+ | |
| | Component | | Publication | | |
| | Subscription | | Server +------+
| | Server | +-------------+ |
| +--------------+ |
+-------------------------------------+
Fig. 3 The Generic Distributed Data Collection Framework
Master and Agents may interact with each other in several ways:
Zhou, et al. Expires April 14, 2018 [Page 6]
Internet-Draft Multiple Steam Originators October 2017
o Agents need to have a registration or announcement handshake with
the Master Node, so the Master Node is aware of them and of
lifecycle events (such as Agent appearing and disappearing).
o The Master relays the component subscriptions to the Agents.
o The Agents indicate status of component subscriptions to the
Master. The status of the overall subscription is maintained by
the Master. The Master is also responsible for notifying the
subscriber in case of any problems of component subscriptions.
The details of the interaction between the Master and the Agent is
out of scope of this document. It may be described by a dedicated
protocol specification.
During the subscription and the associated publication process, this
document assumes all the Agents registered to the Mater can always
provide the announced capabilities.
Note: Some preliminary considerations on the solution details are now
listed in the appendix for reference. The detailed solution need to
be discussed and will be added if the WG accepts the problem
statement.
4. IANA Considerations
This document makes no request of IANA.
Note to RFC Editor: this section may be removed on publication as an
RFC.
5. Security Considerations
6. Acknowledgements
7. References
7.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed.,
and A. Bierman, Ed., "Network Configuration Protocol
(NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011,
<https://www.rfc-editor.org/info/rfc6241>.
Zhou, et al. Expires April 14, 2018 [Page 7]
Internet-Draft Multiple Steam Originators October 2017
[RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language",
RFC 7950, DOI 10.17487/RFC7950, August 2016,
<https://www.rfc-editor.org/info/rfc7950>.
[W3C.REC-xpath-19991116]
Clark, J. and S. DeRose, "XML Path Language (XPath)
Version 1.0", World Wide Web Consortium Recommendation
REC-xpath-19991116, November 1999,
<http://www.w3.org/TR/1999/REC-xpath-19991116>.
7.2. Informative References
[I-D.ietf-core-coap-pubsub]
Koster, M., Keranen, A., and J. Jimenez, "Publish-
Subscribe Broker for the Constrained Application Protocol
(CoAP)", draft-ietf-core-coap-pubsub-02 (work in
progress), July 2017.
[I-D.ietf-netconf-yang-push]
Clemm, A., Voit, E., Prieto, A., Tripathy, A., Nilsen-
Nygaard, E., Bierman, A., and B. Lengyel, "Subscribing to
YANG datastore push updates", draft-ietf-netconf-yang-
push-10 (work in progress), October 2017.
Appendix A. Change Log
(To be removed by RFC editor prior to publication)
Appendix B. Subscription Decomposition
Since Agents are invisible to the Collector, the Collector can only
subscribe to the Master. This requires the Master to:
1. expose the global capability that can be served by multiple
stream originators;
2. disassemble the subscription to multiple component subscriptions,
and distribute them to the corresponding telemetry sources.
To achieve the above requirement, the Master need a global capability
description which is typically the YANG [RFC7950] data model. This
global YANG model is provided as the contract between the Master and
the Collector.
The Master also need a data structure, typically a table as shown
below, to keep track the mapping between the resource and the
corresponding location identifier of the node that commits to serve
Zhou, et al. Expires April 14, 2018 [Page 8]
Internet-Draft Multiple Steam Originators October 2017
the data. For the YANG defined capabilities, the resource is
described using the XPath [W3C.REC-xpath-19991116] expression.
+------------+---------------+
| resource | location ID |
+------------+---------------+
Table 1
Each Agent associating to the Master owns a local YANG model to
describe the capabilities which it can serve as part of the global
capability. All the Agents need to know the namespace associate with
the Master.
YANG-Push supports two filtering syntaxes which are XPath and Subtree
[RFC6241]. When a YANG-Push subscription request arrives, the Master
Node will firstly extract the filter information. Consequently,
according to the resource-locationID table, the master subscription
can be disassembled into multiple component subscriptions, and the
corresponding location ID can be associated. The component
subscriptions share the same Subscription ID as the master
subscription.
Appendix C. Publication Composition
The Publication Server collects and encapsulates the packets per the
component subscription. There are several potential encodings,
including XML, JSON, CBOR and GPB. The encoding of the data records
follows the YANG schema, so that the composition at the Receiver can
benefit from the structured and hierarchical data instance. The
Collector may be able to assemble many pieces of data associated with
one subscription, and can also deduce the missing pieces of data.
The Collector recognizes data records associated with one
subscription according the Subscription ID. Data records generated
per one subscription are assigned with the same Subscription ID.
For the time series data stream, records are produced periodically
from each stream originator. The message arrival time varies because
of the distributed nature of the publication. The receiver assembles
data generated at the same time period based on the recording time
consisted in each data record. In this case, time synchronization is
required for all the steam originators.
Zhou, et al. Expires April 14, 2018 [Page 9]
Internet-Draft Multiple Steam Originators October 2017
Appendix D. Examples
TBD
Authors' Addresses
Tianran Zhou
Huawei
156 Beiqing Rd., Haidian District
Beijing
China
Email: [email protected]
Guangying Zheng
Huawei
101 Yu-Hua-Tai Software Road
Nanjing, Jiangsu
China
Email: [email protected]
Eric Voit
Cisco Systems
USA
Email: [email protected]
Alexander Clemm
Huawei
2330 Central Expressway
Santa Clara, California
USA
Email: [email protected]
Andy Bierman
YumaWorks
Email: [email protected]
Zhou, et al. Expires April 14, 2018 [Page 10]