Title: Issues about merging network data collected in a redundant environment Authors: Katsuhisa Abe (abekatsu@cysols.com) Kohei Ohta (kohei@cysols.com) Glenn Mansfield Keeni (glenn@cysols.com) Date: 2005/1/22 Table of Contents: 1. Introduction 2. Environment 3. Issues 4. Plans for solutions 5. Conclusion 1. Introduction Network Traffic Monitoring is an important aspect of network management and security. For example, we might observe the effects on the network traffic when an event, such as a network failure, an operational failure or a security incident has occurred. And we would know the predictive information about quality of service, throughputs and also. We, netman WG, have been collecting traffic statistics for the JGN II network to study the results. In this monitoring, we put two collecting agents to secure verbosity. But we adopt SNMP framework with IPv6 network to collect traffic statistics, so we cannot assure of the identity of these collected statistics. The purpose of this document is to point out issues of multiple collecting agents with SNMP and we will suggest the solutions. 2. Environment Before discussing problems, we start to introduce our monitoring environment in JGN II. JGN II is an open test-bed network environment for research and development and provides nationwide IPv6 network and optical wavelength networks in Japan. We project a network traffic monitoring in JGN II network. We aim at providing user network traffic information and analysis techniques to use for research and experiment in JGN II network. We have been placing probes at Miyagi, Tokyo, Gifu, Kyoto, Hiroshima and Saga, and polling trafic statistics from agents on Sendai and Kyoto. Table 1 shows our monitoring environment as on 26 July 2004. We place probes monitoring lines passively at all points in JGN II. We use SNMP framework to collect traffic statistics. two Polling Agents at Sendai and Kyoto periodically, every 60 seconds, access probes by SNMP over IPv6 and obtains traffic statistics which is obtained in the form of Managed Objects. We open these traffic information to the public at http://www.cysol.co.jp/research/jgn2mon/. Table 1 Monitoring Environment in JGN II +----------------------------------------+ | Items | Number | +----------------------------------------+ | Sites where probe is placed | 9 | +----------------------------------------+ | Placed probes | 10 | +----------------------------------------+ | Monitoring points | 11 | +----------------------------------------+ | Monitoring links | 26 | +----------------------------------------+ | (with VLAN) | (19) | +----------------------------------------+ | Polling Agents | 2 | +----------------------------------------+ We show our monitoring traffic statistics on table 2. Here ``other protocols'' means that it is an IPv6 packet but its next header field is not ICMPv6, TCP or UDP. We also have been collecting Round Trip Time measured by traceroute6. Table 2 Measuring statistics +----------------------------------------+ | IPv6 packets/traffic volume | +----------------------------------------+ | ICMPv6 packets/traffic volume | +----------------------------------------+ | TCP over IPv6 packets/traffic volume | +----------------------------------------+ | UDP over IPv6 packets/traffic volume | +----------------------------------------+ | Other protocols packets/traffic volume | +----------------------------------------+ | SNMP Polling Interval | +----------------------------------------+ | Elapsed time by Traceroute6 | +----------------------------------------+ 3. Issues >From previous section, we show that we places two polling agents and they collect network traffic periodically, every 60 seconds, The reason that we adopt two different polling agents is for data redundancy. But this matter causes other isseus. We show an example table to point out what kind of matter this data redundancy causes here. Table 3. example issue caused by redundance of polling agents Polling Agent A Polling Agent B ------------------- ------------------- timestamp counter timestamp counter ------------------- ------------------- 1101826839 23699750 1101826811 23697998 1101826901 23699954 1101826880 23699902 1101826962 23700101 1101826940 23700008 1101827021 23700119 1101827000 23700119 1101827081 23700263 1101827060 23700219 1101827141 23700840 1101827120 23700458 1101827200 23700942 1101827180 23700894 1101827260 23701050 1101827240 23700996 1101827320 23701163 1101827380 23701271 1101827360 23701209 >From table 3, we can clarify three issues: A. Timestamp issue Timestamp of every polling is generated at each polling agent, so, those are fluctuated due to: - network distance from each poling agent is different - no synchronization mechanism between two polling agent B. Polling interval issue Polling interval is also fluctuated by configuration and load of each polling agent. So, it is not easy to compare them. C. Data lack issue We use SNMP, async protocol, to collect traffic statistics from probes with UDP over IPv6 protocol which does not have the responsibility of delivering packets. Those are reasons that we cannot guarantee the sameness of the contents in two different polling agents. So it is necessary to solve these issues. 4. Plan for solutions We discuss here how to solve issues which we point out at Sec. 3. Fig.1 shows out plans to solve these issues. Fig.1 Image abount plans for solutions +----------------+ | Data collected | | by Agent A |\ [new normalize polling data] +----------------+ \+------------+ timestamp counter | Merge Data |---> ------------------- +------------+ 1101826840 23699751 +----------------+ / 1101826900 23699950 | Data collected |/ 1101826960 23700100 | by Agent B | +----------------+ First we decide the master traffic statistics by comparing the data polling agent has. We consider the parameters to decide the master are number of successful polling times, mean of polling interval and variance of polling interval. Next step is that we complement the traffic statistics which master does not have. Finally, if there is a fluctuation in polling intervals, then we normalize the interval with the traffic statistics allocating by pro rate. Again, we enumerate our plan to merge traffic statistics below: a. Decision of the master polling agent for merging traffic statistics. b. Complementation from other polling agent c. Timestamp normalization by pro rate allocation To solve these issues, we are studying on merging traffic statistics which two different polling agents collect. 5. Conclusion Actually our network monitoring project faces with an issue such that there is no guarantee the sameness of the contents in two different polling agents. In this document, we point out an issue caused by the multiple polling agents. and make a plan to solve. As future work, we are studying on and we are implementing merging traffic statistics which two different polling agents collect. Copyright Notice Copyright (C) Cyber Solutions Inc. (2005). All Rights Reserved. Copyright (C) WIDE Project (2005). All Rights Reserved.