00x0 前言
安装服务器环境是CentOS 7,安装当前最新的版本arkime-3.1.1
00x1 安装
1. 下载rpm包
# wget https://s3.amazonaws.com/files.molo.ch/builds/centos-7/arkime-3.1.1-1.x86_64.rpm
2. 安装依赖
# yum -y install perl-libwww-perl perl-JSON libyaml-devel perl-LWP-Protocol-https
3. 安装rpm包
# rpm -ivh arkime-3.1.1-1.x86_64.rpm
4. 安装回显
注意:提示需要按照README的来运行配置文件
00x2 配置
[root@localhost arkime]# cat README.txt
Instructions for using the prebuilt Arkime packages.
Please report any bugs or feature requests by opening an issue at https://github.com/arkime/arkime/issues
Basic Arkime Installation steps:
1) Download an Arkime build for 64bit Ubuntu 14.04, Ubuntu 16.04, Centos 6, or Centos 7 from http://arkime.com/index.html#downloads
2) Install package
3) Configure basic arkime items by running the Configure script (this needs to be done only once)
/opt/arkime/bin/Configure
4) The Configure script can install elasticsearch for you or you can install yourself
systemctl start elasticsearch.service
5) Initialize/Upgrade Elasticsearch Arkime configuration
a) If this is the first install, or want to delete all data
/opt/arkime/db/db.pl http://ESHOST:9200 init
b) If this is an update to a moloch/arkime package
/opt/arkime/db/db.pl http://ESHOST:9200 upgrade
6) Add an admin user if a new install or after an init
/opt/arkime/bin/arkime_add_user.sh admin "Admin User" THEPASSWORD --admin
7) Start everything
systemctl start arkimecapture.service
systemctl start arkimeviewer.service
8) Look at log files for errors
/opt/arkime/logs/viewer.log
/opt/arkime/logs/capture.log
9) Visit http://arkimeHOST:8005 with your favorite browser.
user: admin
password: THEPASSWORD from step #6
If you want IP -> Geo/ASN to work, you need to setup a maxmind account and the geoipupdate program.
See https://arkime.com/faq#maxmind
Any configuration changes can be made to /opt/arkime/etc/config.ini
See https://arkime.com/faq#moloch-is-not-working for issues
Additional information can be found at:
* https://arkime.com/faq
* https://arkime.com/settings
1. 运行配置向导脚本
1.1 安装jdk环境
# yum install java-11-openjdk -y //yum安装arkime依赖jDK环境
# java -version
openjdk version "11.0.12" 2021-07-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7-LTS, mixed mode, sharing)
1.2 运行配置脚本
主要做几件事:
- 确定抓包接口
- 下载并安装默认的es版本,用于快速演示部署(本文采用),生产需要指向生产es
- 检查jdk环境,如果没有会提示,安装后再次运行脚本
- 下载MaxMind的GEO地理位置数据,这个需要去注册一个账号,这样数据显示可以带上地址位置(本文没有选择)
# cd /opt/arkime/bin/
#./Configure
Found interfaces: eth0;lo
Semicolon ';' seperated list of interfaces to monitor [eth1] eth0 //选择抓包接口
Install Elasticsearch server locally for demo, must have at least 3G of memory, NOT recommended for production use (yes or no) [no] yes //是否要本地安装es,至少要3G内存,这边主要是演示环境,不推荐生产
/usr/bin/java //这边提示有java环境,如果不是则需要安装jdk环境,再运行配置脚步
Password to encrypt S2S and other things [no-default] yourpassword
Arkime - Creating configuration files
Not overwriting /opt/arkime/etc/config.ini, delete and run again if update required (usually not), or edit by hand
Installing systemd start files, use systemctl
Arkime - Downloading and installing demo OSS version of Elasticsearch
Loaded plugins: fastestmirror
elasticsearch-oss-7.10.2-x86_64.rpm | 220 MB 00:00:33
Examining /var/tmp/yum-root-Wzs9zH/elasticsearch-oss-7.10.2-x86_64.rpm: elasticsearch-oss-7.10.2-1.x86_64
/var/tmp/yum-root-Wzs9zH/elasticsearch-oss-7.10.2-x86_64.rpm: does not update installed package.
Error: Nothing to do
Download GEO files? You'll need a MaxMind account https://arkime.com/faq#maxmind (yes or no) [yes] no
Arkime - NOT downloading GEO files
Arkime - Configured - Now continue with step 4 in /opt/arkime/README.txt
4) The Configure script can install elasticsearch for you or you can install yourself
systemctl start elasticsearch.service
5) Initialize/Upgrade Elasticsearch Arkime configuration
a) If this is the first install, or want to delete all data
/opt/arkime/db/db.pl http://ESHOST:9200 init
b) If this is an update to a moloch/arkime package
/opt/arkime/db/db.pl http://ESHOST:9200 upgrade
6) Add an admin user if a new install or after an init
/opt/arkime/bin/arkime_add_user.sh admin "Admin User" THEPASSWORD --admin
7) Start everything
systemctl start arkimecapture.service
systemctl start arkimeviewer.service
8) Look at log files for errors
/opt/arkime/logs/viewer.log
/opt/arkime/logs/capture.log
9) Visit http://arkimeHOST:8005 with your favorite browser.
user: admin
password: THEPASSWORD from step #6
If you want IP -> Geo/ASN to work, you need to setup a maxmind account and the geoipupdate program.
See https://arkime.com/faq#maxmind
Any configuration changes can be made to /opt/arkime/etc/config.ini
See https://arkime.com/faq#moloch-is-not-working for issues
Additional information can be found at:
* https://arkime.com/faq
* https://arkime.com/settings
2. 开启服务
2.1 开启本地安装的elasticsearch
# systemctl start elasticsearch.service
3. 创建Web访问账号
1qaz2wsx替换为你的密码
# /opt/arkime/bin/arkime_add_user.sh admin "Admin User" 1qaz2wsx --admin
Added
4. 初始化elasticsearch
需要使用arkime提供的db.pl对es进行初始化
# /opt/arkime/db/db.pl http://localhost:9200 init
It is STRONGLY recommended that you stop ALL Arkime captures and viewers before proceeding. Use 'db.pl http://localhost:9200 backup' to backup db first.
There is 1 elastic search data node, if you expect more please fix first before proceeding.
This is a fresh Arkime install
Erasing
Creating
Finished
5. 启动web服务
5.1 启动arkimeviewer(web服务)
# systemctl start arkimeviewer.service
# systemctl status arkimeviewer.service //查看arkimeviewer服务是否运行正常
5.2 启动arkimecapture(抓包服务)
由于arkimecapture依赖GEO位置数据,运行配置脚本的时候我并没有选择安装GEO,所以也就无法启动了,下面提供了一个办法,通过分析arkime_update_geo.sh脚本内容来达到支持GEO位置信息包更新的目的。
注意:由于国内网络环境有的时候无法正常下载GEO数据,所以我上面就没有选择安装,而是直接手动运行脚本安装,但发现运行脚本也不行,于是我们打开这个shell脚本,看看到底需要下载什么数据
# cd /opt/arkime/bin
# vim arkime_update_geo.sh
从脚本上主要就是要把2个文件拷贝复制到/opt/arkime/conf目录下,并且用chmod修改权限
#!/bin/bash
# Variables
DEST_DIR="${MOLOCH_DIR:-/opt/arkime}/etc"
TIMEOUT=${WGET_TIMEOUT:-30}
# Check we have a number for timeout
if ! [[ $TIMEOUT =~ ^[0-9]+$ ]] ; then
echo "WGET_TIMEOUT isn't a number '$TIMEOUT'"
exit 1;
fi
# Try and download ipv4-address-space.csv, only copy if it works
FILENAME=$(mktemp)
wget -nv --timeout=${TIMEOUT} --no-check-certificate -O "$FILENAME" https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv
if (( $? == 0 )) ; then
chmod a+r "$FILENAME"
mv "$FILENAME" "${DEST_DIR}/ipv4-address-space.csv"
fi
# Try and download manuf, only copy if it works
FILENAME=$(mktemp)
wget -nv --timeout=${TIMEOUT} -O "$FILENAME" https://raw.githubusercontent.com/wireshark/wireshark/master/manuf
if (( $? == 0 )) ; then
chmod a+r "$FILENAME"
mv "$FILENAME" "${DEST_DIR}/oui.txt"
fi
# Call the maxind geoipupdate program if available. See
# https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
# https://dev.maxmind.com/geoip/geoipupdate/#For_Free_GeoLite2_Databases
if [ -x "/usr/bin/geoipupdate" ]; then
/usr/bin/geoipupdate
chmod a+r /usr/share/GeoIP/*.mmdb
fi
文件我下载放到优快云上了,点击下载,文件解压到/opt/arkime/etc,并修改下权限
# chmod a+r oui.txt
# chmod a+r ipv4-address-space.csv
这个时候就可以启动服务了
# systemctl start arkimecapture.service
# systemctl status arkimecapture.service //观察arkimecapture服务是否启动成功
6. 访问界面
ipaddress替换为你安装的服务器IP
http://ipaddress:8005
7. config.ini
有朋友留言要配置文件的内容,贴一下到这里,不过就是按照上面配置,应该就可以跑起来了
[root@localhost etc]# cat config.ini
# Latest settings documentation: https://arkime.com/settings
#
# Arkime uses a tiered system for configuration variables. This allows Arkime
# to share one config file for many machines. The ordering of sections in this
# file doesn't matter.
#
# Order of config variables:
# 1st) [optional] The section titled with the node name is used first.
# 2nd) [optional] If a node has a nodeClass variable, the section titled with
# the nodeClass name is used next. Sessions will be tagged with
# class:<node class name> which may be useful if watching different networks.
# 3rd) The section titled "default" is used last.
[default]
# Comma seperated list of elasticsearch host:port combinations. If not using a
# Elasticsearch load balancer, a different elasticsearch node in the cluster can be specified
# for each Arkime node to help spread load on high volume clusters. For user/password
# use http://user:pass@host:port
elasticsearch=http://localhost:9200
# How often to create a new elasticsearch index. hourly,hourly6,daily,weekly,monthly
# Changing the value will cause previous sessions to be unreachable
rotateIndex=daily
# Cert file to use, comment out to use http instead
# certFile=/opt/arkime/etc/arkime.cert
# File with trusted roots/certs. WARNING! this replaces default roots
# Useful with self signed certs and can be set per node.
# caTrustFile=/opt/arkime/etc/roots.cert
# Private key file to use, comment out to use http instead
# keyFile=/opt/arkime/etc/arkime.key
# Password Hash and S2S secret - Must be in default section. Since elasticsearch
# is wide open by default, we encrypt the stored password hashes with this
# so a malicous person can't insert a working new account. It is also used
# for secure S2S communication. Comment out for no user authentication.
# Changing the value will make all previously stored passwords no longer work.
# Make this RANDOM, you never need to type in
passwordSecret = yourpassword
# Use a different password for S2S communication then passwordSecret.
# Must be in default section. Make this RANDOM, you never need to type in
#serverSecret=
# HTTP Digest Realm - Must be in default section. Changing the value
# will make all previously stored passwords no longer work
httpRealm = Moloch
# The base path for Arkime web access. Must end with a / or bad things will happen
# Default: "/"
# webBasePath = /arkime/
# Semicolon ';' seperated list of interfaces to listen on for traffic
interface=eth0
# The bpf filter of traffic to ignore
bpf=not port 8005
# The yara file name
#yara=
# Host to connect to for wiseService
#wiseHost=127.0.0.1
# Log viewer access requests to a different log file
#accessLogFile = /opt/arkime/logs/access.log
# Control the log format for access requests. This uses URI % encoding.
#accessLogFormat = :date :username %1b[1m:method%1b[0m %1b[33m:url%1b[0m :status :res[content-length] bytes :response-time ms
# The directory to save raw pcap files to
pcapDir = /opt/arkime/raw
# The max raw pcap file size in gigabytes, with a max value of 36G.
# The disk should have room for at least 10*maxFileSizeG
maxFileSizeG = 12
# The max time in minutes between rotating pcap files. Default is 0, which means
# only rotate based on current file size and the maxFileSizeG variable
#maxFileTimeM = 60
# TCP timeout value. Arkime writes a session record after this many seconds
# of inactivity.
tcpTimeout = 600
# Arkime writes a session record after this many seconds, no matter if
# active or inactive
tcpSaveTimeout = 720
# UDP timeout value. Arkime assumes the UDP session is ended after this
# many seconds of inactivity.
udpTimeout = 30
# ICMP timeout value. Arkime assumes the ICMP session is ended after this
# many seconds of inactivity.
icmpTimeout = 10
# An aproximiate maximum number of active sessions Arkime/libnids will try
# and monitor
maxStreams = 1000000
# Arkime writes a session record after this many packets
maxPackets = 10000
# Delete pcap files when free space is lower then this in gigabytes OR it can be
# expressed as a percentage (ex: 5%). This does NOT delete the session records in
# the database. It is recommended this value is between 5% and 10% of the disk.
# Database deletes are done by the db.pl expire script
freeSpaceG = 5%
# The port to listen on, by default 8005
viewPort = 8005
# The host/ip to listen on, by default 0.0.0.0 which is ALL
#viewHost = localhost
# By default the viewer process is https://hostname:<viewPort> for each node.
#viewUrl = https://HOSTNAME:8005
# A MaxMind account is now required, Arkime checks several install locations, or
# will work without Geo files installed. See https://arkime.com/faq#maxmind
#geoLite2Country = /var/lib/GeoIP/GeoLite2-Country.mmdb;/usr/share/GeoIP/GeoLite2-Country.mmdb;/opt/arkime/etc/GeoLite2-Country.mmdb
#geoLite2ASN = /var/lib/GeoIP/GeoLite2-ASN.mmdb;/usr/share/GeoIP/GeoLite2-ASN.mmdb;/opt/arkime/etc/GeoLite2-ASN.mmdb
# Path of the rir assignments file
# https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv
rirFile = /opt/arkime/etc/ipv4-address-space.csv
# Path of the OUI file from whareshark
# https://raw.githubusercontent.com/wireshark/wireshark/master/manuf
ouiFile = /opt/arkime/etc/oui.txt
# Arkime rules to allow you specify actions to perform when criteria are met with certain fields or state.
# See https://arkime.com/rulesformat
#rulesFiles=/opt/arkime/etc/arkime.rules
# User to drop privileges to. The pcapDir must be writable by this user or group below
dropUser=root
# Group to drop privileges to. The pcapDir must be writable by this group or user above
dropGroup=daemon
# Semicolon ';' seperated list of tags which once capture sets for a session causes the
# remaining pcap from being saved for the session. It is likely that the initial packets
# WILL be saved for the session since tags usually aren't set until after several packets
# Each tag can optionally be followed by a :<num> which specifies how many total packets to save
#dontSaveTags=
# Header to use for determining the username to check in the database for instead of
# using http digest. Use this if apache or something else is doing the auth.
# Set viewHost to localhost or use iptables
# Might need something like this in the httpd.conf
# RewriteRule .* - [E=ENV_RU:%{REMOTE_USER}]
# RequestHeader set ARKIME_USER %{ENV_RU}e
#userNameHeader=arkime_user
#
# Headers to use to determine if user from `userNameHeader` is
# authorized to use the system, and if so create a new user
# in the Arkime user database. This implementation expects that
# the users LDAP/AD groups (or similar) are populated into an
# HTTP header by the Apache (or similar) referenced above.
# The JSON in userAutoCreateTmpl is used to insert the new
# user into the arkime database (if not already present)
# and additional HTTP headers can be sourced from the request
# to populate various fields.
#
# The example below pulls verifies that an HTTP header called `UserGroup`
# is present, and contains the value "ARKIME_ACCESS". If this authorization
# check passes, the user database is inspected for the user in `userNameHeader`
# and if it is not present it is created. The system uses the
# `arkime_user` and `http_auth_mail` headers from the
# request and uses them to populate `userId` and `userName`
# fields for the new user record.
#
# Once the user record is created, this functionaity
# neither updates nor deletes the data, though if the user is no longer
# reported to be in the group, access is denied regardless of the status
# in the arkime database.
#
#requiredAuthHeader="UserGroup"
#requiredAuthHeaderVal="ARKIME_ACCESS"
#userAutoCreateTmpl={"userId": "${this.arkime_user}", "userName": "${this.http_auth_mail}", "enabled": true, "webEnabled": true, "headerAuthEnabled": true, "emailSearch": true, "createEnabled": false, "removeEnabled": false, "packetSearch": true }
# Should we parse extra smtp traffic info
parseSMTP=true
# Should we parse extra smb traffic info
parseSMB=true
# Should we parse HTTP QS Values
parseQSValue=false
# Should we calculate sha256 for bodies
supportSha256=false
# Only index HTTP request bodies less than this number of bytes */
maxReqBody=64
# Only store request bodies that Utf-8?
config.reqBodyOnlyUtf8 = true
# Semicolon ';' seperated list of SMTP Headers that have ips, need to have the terminating colon ':'
smtpIpHeaders=X-Originating-IP:;X-Barracuda-Apparent-Source-IP:
# Semicolon ';' seperated list of directories to load parsers from
parsersDir=/opt/arkime/parsers
# Semicolon ';' seperated list of directories to load plugins from
pluginsDir=/opt/arkime/plugins
# Semicolon ';' seperated list of plugins to load and the order to load in
# plugins=tagger.so; netflow.so
# Add suricata.so to your plugins line, or add a new plugins line
plugins=suricata.so
# suricataAlertFile should be the full path to your alert.json or eve.json file
suricataAlertFile=/var/log/suricata/eve.json
suricataExpireMinutes=60
# Plugins to load as root, usually just readers
#rootPlugins=reader-pfring; reader-daq.so
# Semicolon ';' seperated list of viewer plugins to load and the order to load in
# viewerPlugins=wise.js
# NetFlowPlugin
# Input device id, 0 by default
#netflowSNMPInput=1
# Outout device id, 0 by default
#netflowSNMPOutput=2
# Netflow version 1,5,7 supported, 7 by default
#netflowVersion=1
# Semicolon ';' seperated list of netflow destinations
#netflowDestinations=localhost:9993
# Specify the max number of indices we calculate spidata for.
# ES will blow up if we allow the spiData to search too many indices.
spiDataMaxIndices=4
# Uncomment the following to allow direct uploads. This is experimental
#uploadCommand=/opt/arkime/bin/capture --copy -n {NODE} -r {TMPFILE} -c {CONFIG} {TAGS}
# Title Template
# _cluster_ = ES cluster name
# _userId_ = logged in User Id
# _userName_ = logged in User Name
# _page_ = internal page name
# _expression_ = current search expression if set, otherwise blank
# _-expression_ = " - " + current search expression if set, otherwise blank, prior spaces removed
# _view_ = current view if set, otherwise blank
# _-view_ = " - " + current view if set, otherwise blank, prior spaces removed
#titleTemplate=_cluster_ - _page_ _-view_ _-expression_
# Number of threads processing packets
packetThreads=2
# HSTS Header
# If set to true, adds a Strict-Transport-Security response header with a max age of a year
# and includes subdomains (the app must be served over https)
#hstsHeader=true
# Business Hours
# If set, displays a colored background on the sessions timeline graph during business hours
# Values are set in hours from midnight UTC (default is off)
#businessDayStart=9
#businessDayEnd=17
# Business Days
# Comma separated list of days
# If set, displays the business hours on only the days provided here
# Business hours must be set for these to be of use
# Values are the days of the week as numbers, the week starts at Sunday = 0 and ends on Saturday = 6
# (default is Monday - Friday 1,2,3,4,5)
#businessDays=1,2,3,4,5
# ADVANCED - Semicolon ';' seperated list of files to load for config. Files are loaded
# in order and can replace values set in this file or previous files.
#includes=
# ADVANCED - How is pcap written to disk
# simple = use O_DIRECT if available, writes in pcapWriteSize chunks,
# a file per packet thread.
# simple-nodirect = don't use O_DIRECT. Required for zfs and others
pcapWriteMethod=simple
# ADVANCED - Buffer size when writing pcap files. Should be a multiple of the raid 5 or xfs
# stripe size. Defaults to 256k
pcapWriteSize = 262143
# ADVANCED - Number of bytes to bulk index at a time
dbBulkSize = 300000
# ADVANCED - Compress requests to ES, reduces ES bandwidth by ~80% at the cost
# of increased CPU. MUST have "http.compression: true" in elasticsearch.yml file
compressES = false
# ADVANCED - Max number of connections to elastic search
maxESConns = 30
# ADVANCED - Max number of es requests outstanding in q
maxESRequests = 500
# ADVANCED - Number of packets to ask libnids/libpcap to read per poll/spin
# Increasing may hurt stats and ES performance
# Decreasing may cause more dropped packets
packetsPerPoll = 50000
# ADVANCED - Arkime will try to compensate for SYN packet drops by swapping
# the source and destination addresses when a SYN-acK packet was captured first.
# Probably useful to set it false, when running Arkime in wild due to SYN floods.
antiSynDrop = true
# DEBUG - Write to stdout info every X packets.
# Set to -1 to never log status
logEveryXPackets = 100000
# DEBUG - Write to stdout unknown protocols
logUnknownProtocols = false
# DEBUG - Write to stdout elastic search requests
logESRequests = true
# DEBUG - Write to stdout file creation information
logFileCreation = true
### High Performance settings
# https://arkime.com/settings#high-performance-settings
# magicMode=basic
# pcapReadMethod=tpacketv3
# tpacketv3NumThreads=2
# pcapWriteMethod=simple
# pcapWriteSize = 2560000
# packetThreads=5
# maxPacketsInQueue = 200000
### Low Bandwidth settings
# packetThreads=1
# pcapWriteSize = 65536
##############################################################################
# Classes of nodes
# Can override most default values, and create a tag call node:<classname>
[class1]
freeSpaceG = 10%
##############################################################################
# Nodes
# Usually just use the hostname before the first dot as the node name
# Can override most default values
[node1]
nodeClass = class1
# Might use a different elasticsearch node
elasticsearch=elasticsearchhost1
# Uncomment if this node should process the cron queries and packet search jobs, only ONE node should process cron queries and packet search jobs
# cronQueries = true
[node2]
nodeClass = class2
# Might use a different elasticsearch node
elasticsearch=elasticsearchhost2
# Uses a different interface
interface = eth4
##############################################################################
# override-ips is a special section that overrides the MaxMind databases for
# the fields set, but fields not set will still use MaxMind (example if you set
# tags but not country it will use MaxMind for the country)
# Spaces and capitalization is very important.
# IP Can be a single IP or a CIDR
# Up to 10 tags can be added
#
# ip=tag:TAGNAME1;tag:TAGNAME2;country:3LetterUpperCaseCountry;asn:ASN STRING
#[override-ips]
#10.1.0.0/16=tag:ny-office;country:USA;asn:AS0000 This is an ASN
##############################################################################
# It is possible to define in the config file extra http/email headers
# to index. They are accessed using the expression http.<fieldname> and
# email.<fieldname> with optional .cnt expressions
#
# Possible config atributes for all headers
# type:<string> (string|integer|ip) = data type (default string)
# count:<boolean> = index count of items (default false)
# unique:<boolean> = only record unique items (default true)
# headers-http-request is used to configure request headers to index
[headers-http-request]
referer=type:string;count:true;unique:true
authorization=type:string;count:true
content-type=type:string;count:true
origin=type:string
# headers-http-response is used to configure http response headers to index
[headers-http-response]
location=type:string
server=type:string
content-type=type:string;count:true
# headers-email is used to configure email headers to index
[headers-email]
x-priority=type:integer
authorization=type:string
##############################################################################
# If you have multiple clusters and you want the ability to send sessions
# from one cluster to another either manually or with the cron feature fill out
# this section
#[moloch-clusters]
#forensics=url:https://viewer1.host.domain:8005;passwordSecret:password4arkime;name:Forensics Cluster
#shortname2=url:http://viewer2.host.domain:8123;passwordSecret:password4arkime;name:Testing Cluster
# WARNING: This is an ini file with sections, most likely you don't want to put a setting here.
# New settings usually go near the top in the [default] section, or in [nodename] sections.