BGPGuard
BGP anomaly detection tool that integrates various stages of the anomaly detection process
About BGPGuard
BGPGuard is used to detect Border Gateway Protocol (BGP) anomalous events based on routing records collected from major Internet exchange points worldwide. It also facilitates creating new machine learning models based on historical BGP anomalous events. BGPGuard integrates various stages of the anomaly detection process. The tool consists of: data download, feature extraction, label refinement, data partition, data processing, machine learning algorithms, parameter selection, machine learning models, and classification modules. BGPGuard is based on Python and JavaScript and has been developed with both terminal-based and web-based applications for Linux platforms. The web-based version of BGPGuard offers an interactive interface with a better view for monitoring and performing experiments. Its front-end is built based on HTML, CSS (Bootstrap: an open-source CSS framework), and Socket.IO (a transport protocol written in a JavaScript for real-time web applications). Its back-end is developed using Flask (a micro web framework written in Python).
BGPGuard is used for real-time anomaly detection or off-line data classification based on messages collected from RIPE or Route Views. For real-time anomaly detection, BGP update messages are retrieved, processed, and analyzed using available pre-trained models. The off-line classification is based on the specified start and end dates, times of the anomalous event, partitioning of the training and test datasets, and machine learning algorithms (GBDT, CNN, RNN, or VFBLS).
Structure:
BGPGaurd
├── LICENSE
├── README.md
├── app.py
├── app_offline.py
├── app_realtime.py
├── config.py
├── requirements.txt
├── database
│ ├── database.py
│ └── db_files
├── src
│ ├── __init__.py
│ ├── check_versions.py
│ ├── dataDownload.py
│ ├── data_partition.py
│ ├── data_process.py
│ ├── featureExtraction.py
│ ├── input_exp.txt
│ ├── label_generation.py
│ ├── progress.py
│ ├── progress_bar.py
│ ├── subprocess_cmd.py
│ ├── time_tracker.py
│ ├── CSharp_Tool_BGP
│ ├── STAT
│ ├── VFBLS_v110
│ ├── data_historical
│ ├── data_ripe
│ ├── data_routeviews
│ ├── data_split
│ ├── data_test
│ └── parmSel
│ └── playground
├── static
│ ├── css
│ │ └── style.css
│ ├── imgs
│ └── js
└── templates
├── bgp_ad_offline.html
├── bgp_ad_realtime.html
├── contact.html
├── index.html
└── layout.html
The src directory contains the source code for the real-time detection and off-line classification tasks.
Various Python functions have been developed to implement and incorporate the anomaly detection steps.
🏗️ Getting Started with BGPGuard
Python 3.7 or Python 3.8
External Libraries
The web-based application relies on additional external libraries.
The external CSS and JavaScript libraries provided by jsDelivr have been
included in layout.html.
The Python libraries installed by pip are:
- NumPy: used to perform mathematical operations on multi-dimensional arrays and on matrices generated during the process.
- SciPy: dependency of the scikit-learn library. SciPy’s zscore: function used to perform normalization.
- scikit-learn: employed for processing data and calculating performance metrics.
-
PyTorch: used for developing deep learning models.
- Flask: web framework based on Werkzeug and Jinja. (The Flask’s functions are used to transfer variables and render web pages. Flask also processes the GET/POST requests from the front-end.)
- Werkzeug: web application library used to create a web server gateway interface (WSGI).
- Jinja: web template engine. Variables, statements, and expressions allowed to include in HTML files.
- Flask-SocketIO: offers bi-directional communications with low latency between the clients (front-end) and the server (back-end) for Flask applications.
- python-engineio: implementation of the Engine.IO in Python.
- python-socketio: library for real-time communication between client and server based on WebSocket.
- Eventlet: networking library for executing asynchronous tasks.
Install the external Python libraries by running:
pip install -r requirements.txt
The C# compiler should be installed prior to executing the BGP C# tool.
- Mono: an open source version of Microsoft .NET framework. Mono includes a C# compiler for several operating systems such as macOS, Linux, and Windows.
🚀 Launching BGPGuard
The Python file app.py is used to execute the application.
The following command is used to start the application:
export FLASK_APP=app.py
export FLASK_ENV=development
flask run
Running on:
http://127.0.0.1:5000/
🎡 Playground
Sample code for gradient boosting machines may be run without launching the app.py:
./src/playground/gbdt_offline_sample
Please see details in README.md.
🗃️ Our generated BGP Datasets
We have download the raw data and processed the BGP routing records. Five well-known BGP anomalous events are publicly available from IEEE DataPort: WannaCrypt, Moscow blackout, Slammer, Nimda, Code Red I, occurred in May 2017, May 2005, January 2003, September 2001, and July 2001, respectively. The Reseaux IP Europeens (RIPE) BGP update messages are publicly available from the Network Coordination Centre (NCC) and contain: WannaCrypt, Moscow blackout, Slammer, Nimda, Code Red I, and regular data: https://www.ripe.net/analyse. Regular data are also collected from BCNET.
BGP datasets with routing records collected from Reseaux IP Europeens (RIPE), BCNET, and Route Views: