Skip to main content
Log in

Collection, Analysis and Interactive Visualization of NetFlow Data: Experience with Big Data on the Base of the National Research Computer Network of Russia

  • Published:
Lobachevskii Journal of Mathematics Aims and scope Submit manuscript

Abstract

The implementation of a set of measures aimed to regular monitoring and analyzing the activity of users of the National Research Computer Network of Russia in the inter-network interaction, evaluation of the level of its involvement in joint research projects, the intensity of using the technological infrastructure of Russian and the world’s national research and education networks are discussed in the paper. Information on the developed methods, hardware and software solutions based on NetFlow data and provided collecting, analysis and visualization of network traffic is given; some results of work in identifying and visualizing of data exchange and the main directions of network connectivity are presented and discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

REFERENCES

  1. P. Haag, Nfdump—NetFlow Processing Tools. https://github.com/phaag/nfdump. Accessed May 14, 2020.

  2. V. T. Guimarães, C. M. D. S. Freitas, R. Sadre, L. M. R. Tarouco, and L. Z. Granville, ‘‘A survey on information visualization for network and service management,’’ IEEE Comm. Surv. Tutor. 18, 285–323 (2016). https://doi.org/10.1109/COMST.2015.2450538

    Article  Google Scholar 

  3. C. Allocchio, L. Balint, V. Berkhout, J. Bersee, Y. Izhvanov, et al., A History of International Research Networking: The People who Made it Happen (Wiley-VCH, New York, 2010).

    Google Scholar 

  4. J. Ryan, A history of the Internet and the Digital Future (Breaktion Books, London, 2015).

    Google Scholar 

  5. A. G. Abramov and A. V. Evseev, ‘‘RUNNet: Infrastructural and service basis of the national research and education network of the Russian Federation,’’ in Proceedings of the 8th International Conference on Distributed Computing and Grid-technologies in Science and Education GRID 2018, Sept. 10–14, 2018 (Dubna, 2018), pp. 52–57. Urn:nbn:de:0074-2267-5. http://ceur-ws.org/Vol-2267/52-57-paper-8.pdf.

  6. A. G. Abramov and A. V. Evseev, ‘‘Network RUNNet: Towards the state-of-the-art challenges in the field of telecommunications in science and education,’’ Inform. Educat. Sci. 4 (40), 3–15 (2018).

    Google Scholar 

  7. A. G. Abramov and A. V. Evseev, ‘‘Conceptual aspects of creating a new generation national research computer network in the Russian Federation,’’ Inform. Technol. 25, 724–733 (2019).

    Google Scholar 

  8. G. I. Savin, B. M. Shabanov, A. V. Baranov, A. P. Ovsyannikov, and A. A. Gonchar, ‘‘On the use of the federal research telecommunications infrastructure for supercomputer computations,’’ Vestn. YuUrGU, Ser.: Vychisl. Mat. Inform. 9 (1), 20–35 (2020). https://doi.org/10.14529/cmse200102

    Article  Google Scholar 

  9. Cisco IOS NetFlow. https://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-netflow/. Accessed May 14, 2020.

  10. R. Bejtlich, The Practice of Network Security Monitoring: Understanding Incident Detection and Response (No Starch Press, San Francisco, 2013).

    Google Scholar 

  11. C. Sanders and J. Smith, Applied Network Security Monitoring: Collection, Detection, and Analysis (Syngress, Waltham, MA, USA, 2014).

    Book  Google Scholar 

  12. R. Hofstede, P. Čeleda, B. Trammell, I. Drago, R. Sadre, A. Sperotto, and A. Pras, ‘‘Flow monitoring explained: From packet capture to data analysis with NetFlow and IPFIX,’’ IEEE Comm. Surv. Tutor. 16, 2037–2064 (2014). https://doi.org/10.1109/COMST.2014.2321898

    Article  Google Scholar 

  13. I. Ghafir, J. Svoboda, and V. Prenosil, ‘‘A survey on network security monitoring implementations,’’ in Proceedings of the 2016 IEEE 4th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW), Aug. 22–24, 2016, Vienna, Austria (2016), pp. 77–82. https://doi.org/10.1109/W-FiCloud.2016.30

  14. Z. Tian, ‘‘Management of large scale NetFlow data by distributed systems’’ Master’s Thesis (Norwegian Univ. Sci. Technol., Trondheim, 2016). http://hdl.handle.net/11250/2410246. Accessed May 14, 2020.

  15. M. Čermák, D. Tovarňák, M. Laštovička, and P. Čeleda, ‘‘A performance benchmark for NetFlow data analysis on distributed stream processing systems,’’ in Proceedings of the IEEE/IFIP Network Operations and Management Symposium (NOMS), Apr. 25–29, 2016, Istanbul, Turkey (2016), pp. 919–924. https://doi.org/10.1109/NOMS.2016.7502926

  16. A. Raulot and S. Zaidi, Large-Scale NetFlow Information Management (University of Amsterdam, Amsterdam, 2018). https://delaat.net/rp/2017-2018/p83/report.pdf. Accessed May 14, 2020.

  17. F. Fischer, F. Mansmann, D. A. Keim, S. Pietzko, and M. Waldvogel, ‘‘Large-scale network monitoring for visual analysis of attacks,’’ in Proceedings of the 5th International Workshop on Visualization for Computer Security (VizSec 2008), Sept. 15, 2008, Cambridge, USA (2008), pp. 111–118. https://doi.org/10.1007/978-3-540-85933-8_11

  18. B. Li, J. Springer, G. Bebis, and M. H. Gunes, ‘‘A survey of network flow applications,’’ J. Network Comput. Appl. 36, 567–581 (2013).

    Article  Google Scholar 

  19. A. I. Getman, E. F. Evstropov, and Yu. V. Markin, ’’Real-time network traffic analysis: A review of applied tasks, approaches and solutions,’’ Preprint ISP RAS No. 28 (ISP RAS, Moscow, 2015). https://www.ispras.ru/preprints/docs/prep_28_2015.pdf. Accessed May 14, 2020.

  20. NfSen—Netflow Sensor: A graphical web based front end for the nfdump netflow tools. http://nfsen.sourceforge.net. Accessed May 14, 2020.

  21. K. Lakkaraju, W. Yurci, and A. J. Lee, ‘‘NVisionIP: Netflow visualizations of system state for security situational awareness,’’ in Proceedings of the 2004 ACM Workshop on Visualization and Data Mining for Computer Security (VizSec/DMSec 2004), Oct. 29, 2004, Washington DC, USA (2004), pp. 65–72.

  22. P. Minařík and T. Dymáček, ‘‘NetFlow data visualization based on graphs,’’ in Proceedings of the 5th International Workshop on Visualization for Computer Security (VizSec 2008), Sept. 15, 2008, Cambridge, USA (2008), pp. 144–151. https://doi.org/10.1007/978-3-540-85933-8_14

  23. A. G. Abramov and A. V. Evseev, ‘‘Monitoring of user activity of the Russian research and education network RUNNet in inter-network interaction: Methods, tools and results,’’ Inform. Educat. Sci. 3 (39), 34–49 (2018).

    Google Scholar 

  24. Cytoscape.js—Graph Theory (network) Library for Visualisation and Analysis. https://js.cytoscape.org. Accessed May 14, 2020.

Download references

ACKNOWLEDGMENTS

The author is grateful to the employees of the SPb branch of JSCC RAS V.A. Porkhachev and I.V. Vasilyev, who took an active part in the development and implementation of the system and applied software solutions.

Funding

The paper was prepared within the framework of the state assignment of Federal State Institution ‘‘Scientific Research Institute for System Analysis of the Russian Academy of Sciences’’ on the topic ‘‘Research and development of methods and means of organizing high-performance computing, creating, processing, storing and distributing big data and digital content in distributed information and computing environments’’ (project no. 0065-2019-0014).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to A. G. Abramov.

Additional information

(Submitted by A. M. Elizarov)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abramov, A.G. Collection, Analysis and Interactive Visualization of NetFlow Data: Experience with Big Data on the Base of the National Research Computer Network of Russia. Lobachevskii J Math 41, 2525–2534 (2020). https://doi.org/10.1134/S1995080220120021

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S1995080220120021

Keywords:

Navigation