Arkime3(moloch)安装与配置

00x0 前言

安装服务器环境是CentOS 7，安装当前最新的版本arkime-3.1.1

00x1 安装

1. 下载rpm包

# wget https://s3.amazonaws.com/files.molo.ch/builds/centos-7/arkime-3.1.1-1.x86_64.rpm

2. 安装依赖

# yum -y install perl-libwww-perl perl-JSON libyaml-devel perl-LWP-Protocol-https

3. 安装rpm包

# rpm -ivh arkime-3.1.1-1.x86_64.rpm

4. 安装回显

注意：提示需要按照README的来运行配置文件

00x2 配置

[root@localhost arkime]# cat README.txt 
Instructions for using the prebuilt Arkime packages.
Please report any bugs or feature requests by opening an issue at https://github.com/arkime/arkime/issues

Basic Arkime Installation steps:
 1) Download an Arkime build for 64bit Ubuntu 14.04, Ubuntu 16.04, Centos 6, or Centos 7 from http://arkime.com/index.html#downloads
 2) Install package
 3) Configure basic arkime items by running the Configure script (this needs to be done only once)
     /opt/arkime/bin/Configure
 4) The Configure script can install elasticsearch for you or you can install yourself
      systemctl start elasticsearch.service
 5) Initialize/Upgrade Elasticsearch Arkime configuration
    a) If this is the first install, or want to delete all data
      /opt/arkime/db/db.pl http://ESHOST:9200 init
    b) If this is an update to a moloch/arkime package
      /opt/arkime/db/db.pl http://ESHOST:9200 upgrade
 6) Add an admin user if a new install or after an init
      /opt/arkime/bin/arkime_add_user.sh admin "Admin User" THEPASSWORD --admin
 7) Start everything
      systemctl start arkimecapture.service
      systemctl start arkimeviewer.service
 8) Look at log files for errors
      /opt/arkime/logs/viewer.log
      /opt/arkime/logs/capture.log
 9) Visit http://arkimeHOST:8005 with your favorite browser.
      user: admin
      password: THEPASSWORD from step #6

If you want IP -> Geo/ASN to work, you need to setup a maxmind account and the geoipupdate program.
See https://arkime.com/faq#maxmind

Any configuration changes can be made to /opt/arkime/etc/config.ini
See https://arkime.com/faq#moloch-is-not-working for issues

Additional information can be found at:
  * https://arkime.com/faq
  * https://arkime.com/settings

1. 运行配置向导脚本

1.1 安装jdk环境

# yum install java-11-openjdk -y //yum安装arkime依赖jDK环境

# java -version
openjdk version "11.0.12" 2021-07-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7-LTS, mixed mode, sharing)

1.2 运行配置脚本

主要做几件事：

1. 确定抓包接口
2. 下载并安装默认的es版本，用于快速演示部署(本文采用)，生产需要指向生产es
3. 检查jdk环境，如果没有会提示，安装后再次运行脚本
4. 下载MaxMind的GEO地理位置数据，这个需要去注册一个账号，这样数据显示可以带上地址位置(本文没有选择)

# cd /opt/arkime/bin/

#./Configure 
Found interfaces: eth0;lo
Semicolon ';' seperated list of interfaces to monitor [eth1] eth0 //选择抓包接口
Install Elasticsearch server locally for demo, must have at least 3G of memory, NOT recommended for production use (yes or no) [no] yes //是否要本地安装es，至少要3G内存，这边主要是演示环境，不推荐生产
/usr/bin/java //这边提示有java环境，如果不是则需要安装jdk环境，再运行配置脚步
Password to encrypt S2S and other things [no-default] yourpassword
Arkime - Creating configuration files
Not overwriting /opt/arkime/etc/config.ini, delete and run again if update required (usually not), or edit by hand
Installing systemd start files, use systemctl
Arkime - Downloading and installing demo OSS version of Elasticsearch
Loaded plugins: fastestmirror
elasticsearch-oss-7.10.2-x86_64.rpm                                                                                                         | 220 MB  00:00:33     
Examining /var/tmp/yum-root-Wzs9zH/elasticsearch-oss-7.10.2-x86_64.rpm: elasticsearch-oss-7.10.2-1.x86_64
/var/tmp/yum-root-Wzs9zH/elasticsearch-oss-7.10.2-x86_64.rpm: does not update installed package.
Error: Nothing to do
Download GEO files? You'll need a MaxMind account https://arkime.com/faq#maxmind (yes or no) [yes] no
Arkime - NOT downloading GEO files

Arkime - Configured - Now continue with step 4 in /opt/arkime/README.txt

 4) The Configure script can install elasticsearch for you or you can install yourself
      systemctl start elasticsearch.service
 5) Initialize/Upgrade Elasticsearch Arkime configuration
  a) If this is the first install, or want to delete all data
      /opt/arkime/db/db.pl http://ESHOST:9200 init
  b) If this is an update to a moloch/arkime package
      /opt/arkime/db/db.pl http://ESHOST:9200 upgrade
 6) Add an admin user if a new install or after an init
      /opt/arkime/bin/arkime_add_user.sh admin "Admin User" THEPASSWORD --admin
 7) Start everything
      systemctl start arkimecapture.service
      systemctl start arkimeviewer.service
 8) Look at log files for errors
      /opt/arkime/logs/viewer.log
      /opt/arkime/logs/capture.log
 9) Visit http://arkimeHOST:8005 with your favorite browser.
      user: admin
      password: THEPASSWORD from step #6

If you want IP -> Geo/ASN to work, you need to setup a maxmind account and the geoipupdate program.
See https://arkime.com/faq#maxmind

Any configuration changes can be made to /opt/arkime/etc/config.ini
See https://arkime.com/faq#moloch-is-not-working for issues

Additional information can be found at:
  * https://arkime.com/faq
  * https://arkime.com/settings

2. 开启服务

2.1 开启本地安装的elasticsearch

# systemctl start elasticsearch.service

3. 创建Web访问账号

1qaz2wsx替换为你的密码

# /opt/arkime/bin/arkime_add_user.sh admin "Admin User" 1qaz2wsx --admin
Added

4. 初始化elasticsearch

需要使用arkime提供的db.pl对es进行初始化

# /opt/arkime/db/db.pl http://localhost:9200 init

It is STRONGLY recommended that you stop ALL Arkime captures and viewers before proceeding.  Use 'db.pl http://localhost:9200 backup' to backup db first.

There is 1 elastic search data node, if you expect more please fix first before proceeding.

This is a fresh Arkime install
Erasing
Creating
Finished

5. 启动web服务

5.1 启动arkimeviewer(web服务)

# systemctl start arkimeviewer.service
# systemctl status arkimeviewer.service //查看arkimeviewer服务是否运行正常

5.2 启动arkimecapture(抓包服务)

由于arkimecapture依赖GEO位置数据，运行配置脚本的时候我并没有选择安装GEO，所以也就无法启动了，下面提供了一个办法，通过分析arkime_update_geo.sh脚本内容来达到支持GEO位置信息包更新的目的。

注意：由于国内网络环境有的时候无法正常下载GEO数据，所以我上面就没有选择安装，而是直接手动运行脚本安装，但发现运行脚本也不行，于是我们打开这个shell脚本，看看到底需要下载什么数据

# cd /opt/arkime/bin
# vim arkime_update_geo.sh 

从脚本上主要就是要把2个文件拷贝复制到/opt/arkime/conf目录下，并且用chmod修改权限

#!/bin/bash

# Variables

DEST_DIR="${MOLOCH_DIR:-/opt/arkime}/etc"
TIMEOUT=${WGET_TIMEOUT:-30}

# Check we have a number for timeout

if ! [[ $TIMEOUT =~ ^[0-9]+$ ]] ; then
    echo "WGET_TIMEOUT isn't a number '$TIMEOUT'"
    exit 1;
fi

# Try and download ipv4-address-space.csv, only copy if it works

FILENAME=$(mktemp)
wget -nv --timeout=${TIMEOUT} --no-check-certificate -O "$FILENAME" https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv
if (( $? == 0 )) ; then
  chmod a+r "$FILENAME"
  mv "$FILENAME" "${DEST_DIR}/ipv4-address-space.csv"
fi

# Try and download manuf, only copy if it works

FILENAME=$(mktemp)
wget -nv --timeout=${TIMEOUT} -O "$FILENAME" https://raw.githubusercontent.com/wireshark/wireshark/master/manuf
if (( $? == 0 )) ; then
  chmod a+r "$FILENAME"
  mv "$FILENAME" "${DEST_DIR}/oui.txt"
fi

# Call the maxind geoipupdate program if available. See

# https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

# https://dev.maxmind.com/geoip/geoipupdate/#For_Free_GeoLite2_Databases

if [ -x "/usr/bin/geoipupdate" ]; then
    /usr/bin/geoipupdate
    chmod a+r /usr/share/GeoIP/*.mmdb
fi

文件我下载放到优快云上了，点击下载，文件解压到/opt/arkime/etc，并修改下权限

# chmod a+r oui.txt 
# chmod a+r ipv4-address-space.csv

这个时候就可以启动服务了

# systemctl start arkimecapture.service
# systemctl status arkimecapture.service //观察arkimecapture服务是否启动成功

6. 访问界面

ipaddress替换为你安装的服务器IP

http://ipaddress:8005

7. config.ini

有朋友留言要配置文件的内容，贴一下到这里,不过就是按照上面配置，应该就可以跑起来了

[root@localhost etc]# cat config.ini
# Latest settings documentation: https://arkime.com/settings
#
# Arkime uses a tiered system for configuration variables.  This allows Arkime
# to share one config file for many machines.  The ordering of sections in this
# file doesn't matter.
#
# Order of config variables:
# 1st) [optional] The section titled with the node name is used first.
# 2nd) [optional] If a node has a nodeClass variable, the section titled with
#      the nodeClass name is used next.  Sessions will be tagged with
#      class:<node class name> which may be useful if watching different networks.
# 3rd) The section titled "default" is used last.

[default]

# Comma seperated list of elasticsearch host:port combinations.  If not using a
# Elasticsearch load balancer, a different elasticsearch node in the cluster can be specified
# for each Arkime node to help spread load on high volume clusters.  For user/password
# use http://user:pass@host:port
elasticsearch=http://localhost:9200

# How often to create a new elasticsearch index. hourly,hourly6,daily,weekly,monthly
# Changing the value will cause previous sessions to be unreachable
rotateIndex=daily

# Cert file to use, comment out to use http instead
# certFile=/opt/arkime/etc/arkime.cert

# File with trusted roots/certs. WARNING! this replaces default roots
# Useful with self signed certs and can be set per node.
# caTrustFile=/opt/arkime/etc/roots.cert

# Private key file to use, comment out to use http instead
# keyFile=/opt/arkime/etc/arkime.key

# Password Hash and S2S secret - Must be in default section. Since elasticsearch
# is wide open by default, we encrypt the stored password hashes with this
# so a malicous person can't insert a working new account.  It is also used
# for secure S2S communication. Comment out for no user authentication.
# Changing the value will make all previously stored passwords no longer work.
# Make this RANDOM, you never need to type in
passwordSecret = yourpassword

# Use a different password for S2S communication then passwordSecret.
# Must be in default section.  Make this RANDOM, you never need to type in
#serverSecret=

# HTTP Digest Realm - Must be in default section.  Changing the value
# will make all previously stored passwords no longer work
httpRealm = Moloch

# The base path for Arkime web access.  Must end with a / or bad things will happen
# Default: "/"
# webBasePath = /arkime/

# Semicolon ';' seperated list of interfaces to listen on for traffic
interface=eth0

# The bpf filter of traffic to ignore
bpf=not port 8005

# The yara file name
#yara=

# Host to connect to for wiseService
#wiseHost=127.0.0.1

# Log viewer access requests to a different log file
#accessLogFile = /opt/arkime/logs/access.log

# Control the log format for access requests. This uses URI % encoding.
#accessLogFormat = :date :username %1b[1m:method%1b[0m %1b[33m:url%1b[0m :status :res[content-length] bytes :response-time ms

# The directory to save raw pcap files to
pcapDir = /opt/arkime/raw

# The max raw pcap file size in gigabytes, with a max value of 36G.
# The disk should have room for at least 10*maxFileSizeG
maxFileSizeG = 12

# The max time in minutes between rotating pcap files.  Default is 0, which means
# only rotate based on current file size and the maxFileSizeG variable
#maxFileTimeM = 60

# TCP timeout value.  Arkime writes a session record after this many seconds
# of inactivity.
tcpTimeout = 600

# Arkime writes a session record after this many seconds, no matter if
# active or inactive
tcpSaveTimeout = 720

# UDP timeout value.  Arkime assumes the UDP session is ended after this
# many seconds of inactivity.
udpTimeout = 30

# ICMP timeout value.  Arkime assumes the ICMP session is ended after this
# many seconds of inactivity.
icmpTimeout = 10

# An aproximiate maximum number of active sessions Arkime/libnids will try
# and monitor
maxStreams = 1000000

# Arkime writes a session record after this many packets
maxPackets = 10000

# Delete pcap files when free space is lower then this in gigabytes OR it can be
# expressed as a percentage (ex: 5%).  This does NOT delete the session records in
# the database. It is recommended this value is between 5% and 10% of the disk.
# Database deletes are done by the db.pl expire script
freeSpaceG = 5%

# The port to listen on, by default 8005
viewPort = 8005

# The host/ip to listen on, by default 0.0.0.0 which is ALL
#viewHost = localhost

# By default the viewer process is https://hostname:<viewPort> for each node.
#viewUrl = https://HOSTNAME:8005

# A MaxMind account is now required, Arkime checks several install locations, or
# will work without Geo files installed. See https://arkime.com/faq#maxmind
#geoLite2Country = /var/lib/GeoIP/GeoLite2-Country.mmdb;/usr/share/GeoIP/GeoLite2-Country.mmdb;/opt/arkime/etc/GeoLite2-Country.mmdb
#geoLite2ASN = /var/lib/GeoIP/GeoLite2-ASN.mmdb;/usr/share/GeoIP/GeoLite2-ASN.mmdb;/opt/arkime/etc/GeoLite2-ASN.mmdb

# Path of the rir assignments file
#  https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv
rirFile = /opt/arkime/etc/ipv4-address-space.csv

# Path of the OUI file from whareshark
#  https://raw.githubusercontent.com/wireshark/wireshark/master/manuf
ouiFile = /opt/arkime/etc/oui.txt

# Arkime rules to allow you specify actions to perform when criteria are met with certain fields or state. 
# See https://arkime.com/rulesformat
#rulesFiles=/opt/arkime/etc/arkime.rules

# User to drop privileges to. The pcapDir must be writable by this user or group below
dropUser=root

# Group to drop privileges to. The pcapDir must be writable by this group or user above
dropGroup=daemon

# Semicolon ';' seperated list of tags which once capture sets for a session causes the
# remaining pcap from being saved for the session.  It is likely that the initial packets
# WILL be saved for the session since tags usually aren't set until after several packets
# Each tag can optionally be followed by a :<num> which specifies how many total packets to save
#dontSaveTags=

# Header to use for determining the username to check in the database for instead of
# using http digest.  Use this if apache or something else is doing the auth.
# Set viewHost to localhost or use iptables
# Might need something like this in the httpd.conf
# RewriteRule .* - [E=ENV_RU:%{REMOTE_USER}]
# RequestHeader set ARKIME_USER %{ENV_RU}e
#userNameHeader=arkime_user

#
# Headers to use to determine if user from `userNameHeader` is
# authorized to use the system, and if so create a new user
# in the Arkime user database.  This implementation expects that
# the users LDAP/AD groups (or similar) are populated into an
# HTTP header by the Apache (or similar) referenced above.
# The JSON in userAutoCreateTmpl is used to insert the new
# user into the arkime database (if not already present)
# and additional HTTP headers can be sourced from the request
# to populate various fields.
#
# The example below pulls verifies that an HTTP header called `UserGroup`
# is present, and contains the value "ARKIME_ACCESS".  If this authorization
# check passes, the user database is inspected for the user in `userNameHeader`
# and if it is not present it is created.  The system uses the
# `arkime_user` and `http_auth_mail` headers from the
# request and uses them to populate `userId` and `userName`
# fields for the new user record.
#
# Once the user record is created, this functionaity
# neither updates nor deletes the data, though if the user is no longer
# reported to be in the group, access is denied regardless of the status
# in the arkime database.
#
#requiredAuthHeader="UserGroup"
#requiredAuthHeaderVal="ARKIME_ACCESS"
#userAutoCreateTmpl={"userId": "${this.arkime_user}", "userName": "${this.http_auth_mail}", "enabled": true, "webEnabled": true, "headerAuthEnabled": true, "emailSearch": true, "createEnabled": false, "removeEnabled": false, "packetSearch": true }

# Should we parse extra smtp traffic info
parseSMTP=true

# Should we parse extra smb traffic info
parseSMB=true

# Should we parse HTTP QS Values
parseQSValue=false

# Should we calculate sha256 for bodies
supportSha256=false

# Only index HTTP request bodies less than this number of bytes */
maxReqBody=64

# Only store request bodies that Utf-8?
config.reqBodyOnlyUtf8 = true

# Semicolon ';' seperated list of SMTP Headers that have ips, need to have the terminating colon ':'
smtpIpHeaders=X-Originating-IP:;X-Barracuda-Apparent-Source-IP:

# Semicolon ';' seperated list of directories to load parsers from
parsersDir=/opt/arkime/parsers

# Semicolon ';' seperated list of directories to load plugins from
pluginsDir=/opt/arkime/plugins

# Semicolon ';' seperated list of plugins to load and the order to load in
# plugins=tagger.so; netflow.so
# Add suricata.so to your plugins line, or add a new plugins line
plugins=suricata.so

# suricataAlertFile should be the full path to your alert.json or eve.json file
suricataAlertFile=/var/log/suricata/eve.json
suricataExpireMinutes=60
# Plugins to load as root, usually just readers
#rootPlugins=reader-pfring; reader-daq.so

# Semicolon ';' seperated list of viewer plugins to load and the order to load in
# viewerPlugins=wise.js

# NetFlowPlugin
# Input device id, 0 by default
#netflowSNMPInput=1
# Outout device id, 0 by default
#netflowSNMPOutput=2
# Netflow version 1,5,7 supported, 7 by default
#netflowVersion=1
# Semicolon ';' seperated list of netflow destinations
#netflowDestinations=localhost:9993

# Specify the max number of indices we calculate spidata for.
# ES will blow up if we allow the spiData to search too many indices.
spiDataMaxIndices=4

# Uncomment the following to allow direct uploads.  This is experimental
#uploadCommand=/opt/arkime/bin/capture --copy -n {NODE} -r {TMPFILE} -c {CONFIG} {TAGS}

# Title Template
# _cluster_ = ES cluster name
# _userId_  = logged in User Id
# _userName_ = logged in User Name
# _page_ = internal page name
# _expression_ = current search expression if set, otherwise blank
# _-expression_ = " - " + current search expression if set, otherwise blank, prior spaces removed
# _view_ = current view if set, otherwise blank
# _-view_ = " - " + current view if set, otherwise blank, prior spaces removed
#titleTemplate=_cluster_ - _page_ _-view_ _-expression_

# Number of threads processing packets
packetThreads=2

# HSTS Header
# If set to true, adds a Strict-Transport-Security response header with a max age of a year
# and includes subdomains (the app must be served over https)
#hstsHeader=true

# Business Hours
# If set, displays a colored background on the sessions timeline graph during business hours
# Values are set in hours from midnight UTC (default is off)
#businessDayStart=9
#businessDayEnd=17

# Business Days
# Comma separated list of days
# If set, displays the business hours on only the days provided here
# Business hours must be set for these to be of use
# Values are the days of the week as numbers, the week starts at Sunday = 0 and ends on Saturday = 6
# (default is Monday - Friday 1,2,3,4,5)
#businessDays=1,2,3,4,5

# ADVANCED - Semicolon ';' seperated list of files to load for config.  Files are loaded
# in order and can replace values set in this file or previous files.
#includes=

# ADVANCED - How is pcap written to disk
#  simple          = use O_DIRECT if available, writes in pcapWriteSize chunks,
#                    a file per packet thread.
#  simple-nodirect = don't use O_DIRECT.  Required for zfs and others
pcapWriteMethod=simple

# ADVANCED - Buffer size when writing pcap files.  Should be a multiple of the raid 5 or xfs
# stripe size.  Defaults to 256k
pcapWriteSize = 262143

# ADVANCED - Number of bytes to bulk index at a time
dbBulkSize = 300000

# ADVANCED - Compress requests to ES, reduces ES bandwidth by ~80% at the cost
# of increased CPU. MUST have "http.compression: true" in elasticsearch.yml file
compressES = false

# ADVANCED - Max number of connections to elastic search
maxESConns = 30

# ADVANCED - Max number of es requests outstanding in q
maxESRequests = 500

# ADVANCED - Number of packets to ask libnids/libpcap to read per poll/spin
# Increasing may hurt stats and ES performance
# Decreasing may cause more dropped packets
packetsPerPoll = 50000

# ADVANCED - Arkime will try to compensate for SYN packet drops by swapping
# the source and destination addresses when a SYN-acK packet was captured first.
# Probably useful to set it false, when running Arkime in wild due to SYN floods.
antiSynDrop = true

# DEBUG - Write to stdout info every X packets.
# Set to -1 to never log status
logEveryXPackets = 100000

# DEBUG - Write to stdout unknown protocols
logUnknownProtocols = false

# DEBUG - Write to stdout elastic search requests
logESRequests = true

# DEBUG - Write to stdout file creation information
logFileCreation = true


### High Performance settings
# https://arkime.com/settings#high-performance-settings
# magicMode=basic
# pcapReadMethod=tpacketv3
# tpacketv3NumThreads=2
# pcapWriteMethod=simple
# pcapWriteSize = 2560000
# packetThreads=5
# maxPacketsInQueue = 200000

### Low Bandwidth settings
# packetThreads=1
# pcapWriteSize = 65536


##############################################################################
# Classes of nodes
# Can override most default values, and create a tag call node:<classname>
[class1]
freeSpaceG = 10%

##############################################################################
# Nodes
# Usually just use the hostname before the first dot as the node name
# Can override most default values

[node1]
nodeClass = class1
# Might use a different elasticsearch node
elasticsearch=elasticsearchhost1

# Uncomment if this node should process the cron queries and packet search jobs, only ONE node should process cron queries and packet search jobs
# cronQueries = true

[node2]
nodeClass = class2
# Might use a different elasticsearch node
elasticsearch=elasticsearchhost2
# Uses a different interface
interface = eth4

##############################################################################
# override-ips is a special section that overrides the MaxMind databases for
# the fields set, but fields not set will still use MaxMind (example if you set
# tags but not country it will use MaxMind for the country)
# Spaces and capitalization is very important.
# IP Can be a single IP or a CIDR
# Up to 10 tags can be added
#
# ip=tag:TAGNAME1;tag:TAGNAME2;country:3LetterUpperCaseCountry;asn:ASN STRING
#[override-ips]
#10.1.0.0/16=tag:ny-office;country:USA;asn:AS0000 This is an ASN

##############################################################################
# It is possible to define in the config file extra http/email headers
# to index.  They are accessed using the expression http.<fieldname> and
# email.<fieldname> with optional .cnt expressions
#
# Possible config atributes for all headers
#   type:<string> (string|integer|ip)  = data type                (default string)
#  count:<boolean>                     = index count of items     (default false)
#  unique:<boolean>                    = only record unique items (default true)

# headers-http-request is used to configure request headers to index
[headers-http-request]
referer=type:string;count:true;unique:true
authorization=type:string;count:true
content-type=type:string;count:true
origin=type:string

# headers-http-response is used to configure http response headers to index
[headers-http-response]
location=type:string
server=type:string
content-type=type:string;count:true

# headers-email is used to configure email headers to index
[headers-email]
x-priority=type:integer
authorization=type:string


##############################################################################
# If you have multiple clusters and you want the ability to send sessions
# from one cluster to another either manually or with the cron feature fill out
# this section

#[moloch-clusters]
#forensics=url:https://viewer1.host.domain:8005;passwordSecret:password4arkime;name:Forensics Cluster
#shortname2=url:http://viewer2.host.domain:8123;passwordSecret:password4arkime;name:Testing Cluster



# WARNING: This is an ini file with sections, most likely you don't want to put a setting here.
#          New settings usually go near the top in the [default] section, or in [nodename] sections.