CICFlowMeter2018下载

researchgate看到这位哥们遇到了跟我一样的问题: CICIDS官方 给的资源超过 200 GiB,但是我只想获取其经过处理后的 .csv 文件。

下面来自网友的解决方案!

一、安装 aws cli

1 macOS

官方:https://docs.aws.amazon.com/zh_cn/cli/latest/userguide/install-cliv2-mac.html

1
2
$ curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg" 
$ sudo installer -pkg AWSCLIV2.pkg -target /
  • curl 方式下载可能会很慢,可以拷贝里面的 url 到 bt 工具进行下载。

安装后,通过$ aws --version 确认安装。

2 Ubuntu

$ apt-get install -Y aws

二 、列出 aws 资源

总共 452.8 GiB,找到我们需要下载的资源,给出指定的资源路径。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
$ aws s3 ls --no-sign-request "s3://cse-cic-ids2018" --recursive --human-readable --summarize

2018-10-10 19:52:09 0 Bytes Original Network Traffic and Log data/
2018-10-10 19:52:23 0 Bytes Original Network Traffic and Log data/Friday-02-03-2018/
2018-10-10 20:00:39 225.8 MiB Original Network Traffic and Log data/Friday-02-03-2018/logs.zip
2018-10-10 20:00:51 41.7 GiB Original Network Traffic and Log data/Friday-02-03-2018/pcap.zip
2018-10-10 19:52:34 0 Bytes Original Network Traffic and Log data/Friday-16-02-2018/
2018-10-10 20:45:49 148.1 MiB Original Network Traffic and Log data/Friday-16-02-2018/logs.zip
2018-10-10 20:46:01 35.9 GiB Original Network Traffic and Log data/Friday-16-02-2018/pcap.zip
2018-10-10 19:52:41 0 Bytes Original Network Traffic and Log data/Friday-23-02-2018/
2018-10-10 20:46:10 199.8 MiB Original Network Traffic and Log data/Friday-23-02-2018/logs.zip
2018-10-10 20:46:31 55.0 GiB Original Network Traffic and Log data/Friday-23-02-2018/pcap.zip
2018-10-10 19:52:47 0 Bytes Original Network Traffic and Log data/Thursday-01-03-2018/
2018-10-10 21:41:13 217.1 MiB Original Network Traffic and Log data/Thursday-01-03-2018/logs.zip
2018-10-10 21:41:45 48.8 GiB Original Network Traffic and Log data/Thursday-01-03-2018/pcap.zip
2018-10-10 19:52:54 0 Bytes Original Network Traffic and Log data/Thursday-15-02-2018/
2018-10-10 21:41:28 142.6 MiB Original Network Traffic and Log data/Thursday-15-02-2018/logs.zip
2018-10-10 21:41:55 38.4 GiB Original Network Traffic and Log data/Thursday-15-02-2018/pcap.zip
2018-10-10 19:53:01 0 Bytes Original Network Traffic and Log data/Thursday-22-02-2018/
2018-10-10 21:41:42 195.3 MiB Original Network Traffic and Log data/Thursday-22-02-2018/logs.zip
2018-10-10 21:42:27 46.8 GiB Original Network Traffic and Log data/Thursday-22-02-2018/pcap.zip
2018-10-10 19:53:07 0 Bytes Original Network Traffic and Log data/Tuesday-20-02-2018/
2018-10-10 22:39:45 178.9 MiB Original Network Traffic and Log data/Tuesday-20-02-2018/logs.zip
2018-10-10 22:40:40 41.3 GiB Original Network Traffic and Log data/Tuesday-20-02-2018/pcap.rar
2018-10-10 19:53:14 0 Bytes Original Network Traffic and Log data/Wednesday-14-02-2018/
2018-10-11 00:44:20 133.7 MiB Original Network Traffic and Log data/Wednesday-14-02-2018/logs.zip
2018-10-11 20:22:03 37.2 GiB Original Network Traffic and Log data/Wednesday-14-02-2018/pcap.zip
2018-10-10 19:53:21 0 Bytes Original Network Traffic and Log data/Wednesday-21-02-2018/
2018-10-11 00:44:34 185.6 MiB Original Network Traffic and Log data/Wednesday-21-02-2018/logs.zip
2018-10-11 21:35:15 49.8 GiB Original Network Traffic and Log data/Wednesday-21-02-2018/pcap.zip
2018-10-10 19:53:28 0 Bytes Original Network Traffic and Log data/Wednesday-28-02-2018/
2018-10-11 00:44:47 216.1 MiB Original Network Traffic and Log data/Wednesday-28-02-2018/logs.zip
2018-10-11 22:21:03 49.6 GiB Original Network Traffic and Log data/Wednesday-28-02-2018/pcap.zip
2018-10-12 00:02:25 0 Bytes Processed Traffic Data for ML Algorithms/
2018-10-12 00:02:49 336.0 MiB Processed Traffic Data for ML Algorithms/Friday-02-03-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:03:10 318.3 MiB Processed Traffic Data for ML Algorithms/Friday-16-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:03:33 365.1 MiB Processed Traffic Data for ML Algorithms/Friday-23-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:03:59 3.8 GiB Processed Traffic Data for ML Algorithms/Thuesday-20-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:08:38 102.8 MiB Processed Traffic Data for ML Algorithms/Thursday-01-03-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:08:48 358.5 MiB Processed Traffic Data for ML Algorithms/Thursday-15-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:09:20 364.9 MiB Processed Traffic Data for ML Algorithms/Thursday-22-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:09:44 341.6 MiB Processed Traffic Data for ML Algorithms/Wednesday-14-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:10:12 313.7 MiB Processed Traffic Data for ML Algorithms/Wednesday-21-02-2018_TrafficForML_CICFlowMeter.csv
2018-10-12 00:10:33 199.6 MiB Processed Traffic Data for ML Algorithms/Wednesday-28-02-2018_TrafficForML_CICFlowMeter.csv

Total Objects: 42
Total Size: 452.8 GiB

三、下载 CSV 数据集

aws zones : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html

1
$ aws s3 cp --no-sign-request "s3://cse-cic-ids2018/Processed Traffic Data for ML Algorithms/" cicids2018 --recursive

四、Tips

aws 下载可能很慢,可以使用下面的方式获取

  • 使用 VPS

~

~

  • Colab 保存至 Google Drive

~