100g-passive-tools

This repository contains code recipes to demonstrate how to access CAIDA's 100G passive pcap data using various methods.

Requirements

• Python <= 3.13 (tested on 3.11, incompatible with 3.13+)
  • boto3==1.23.10
  • botocore==1.26.10
  • configparser==3.5.3

---

Clone the repository into an environment that has a minimum of 2TB of disk space

git clone https://github.com/CAIDA/100g-passive-tools.git

(Recommended) Create a virtual environment

python -m venv .venv

Configuration

---

In order to access CAIDA's swift server, a swift_config.ini file will have to be configured in the repository directory, using the AWS S3 credentials provided in the confirmation email (aws_access_key_id and aws_secret_access_key).

Note: One can use the following config file template to get started: swift_config-example.ini

Considerations

---

Each one-directional anonymized pcap file, captured monthly, can reach up to approximately 1TB in size, so users will need more than 2TB of space to download the entire one-hour capture.

Troubleshooting

---

Refer to the following Wiki Page to troubleshoot commonly found errors.

Access Methods

Note: Refer to the requirements.txt file to install the dependencies needed for the recipes.

AWS SDK (boto3)

---

Note: This method requires a swift_config.ini file to be configured, and assumes you've installed boto3 locally.

Refer to the following recipes to download or list out files for a given capture, 100g-anon_download-objects.py and 100g-anon_list-objects.py

AWS CLI

---

Note: This method requires your ~/.aws/credentials file to be configured, and assumes you've set up AWS CLI locally.
For installing instructions, reference AWS Documentation.

[default]
aws_access_key_id =
aws_secret_access_key =

List out files in the 100g-anon-pcap bucket/container

aws s3api list-objects --bucket 100g-anon-pcap-{year} --endpoint-url https://hermes.caida.org --output text

Note: --output {text|json|table}

Download a file in the 100g-anon-pcap-{year} bucket/container

aws s3api get-object --bucket 100g-anon-pcap-{year} --key monitor=100g-01/mon=05/date=20240523-210000.UTC/20240523-210000.dira.stats --endpoint-url https://hermes.caida.org --output text 20240523-210000.dira.stats