分布式部署es集群

使用hadoop账号操作

1. cd /home/hadoop

安装包下载到/home/hadoop，并解压

elasticsearch.yml配置：

1. cd elasticsearch-7.7.1/config/

10.130.49.84、10.130.49.96、10.130.49.97”需要换成线上实际的IP
node.name参数3台机器分别为node-1、node-2、node-3

1. # ======================== Elasticsearch Configuration =========================
2. #
3. # NOTE: Elasticsearch comes with reasonable defaults for most settings.
4. # Before you set out to tweak and tune the configuration, make sure you
5. # understand what are you trying to accomplish and the consequences.
6. #
7. # The primary way of configuring a node is via this file. This template lists
8. # the most important settings you may want to configure for a production cluster.
9. #
10. # Please consult the documentation for further information on configuration options:
11. # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
12. #
13. # ---------------------------------- Cluster -----------------------------------
14. #
15. # Use a descriptive name for your cluster:
16. #
17. cluster.name: middle-platform-es
18. #
19. # ------------------------------------ Node ------------------------------------
20. #
21. # Use a descriptive name for the node:
22. #
23. node.name: node-1
24. #
25. # Add custom attributes to the node:
26. #
27. #node.attr.rack: r1
28. #
29. # ----------------------------------- Paths ------------------------------------
30. #
31. # Path to directory where to store the data (separate multiple locations by comma):
32. #
33. path.data: /home/aitpm/software/elasticsearch-7.7.1/data
34. #
35. # Path to log files:
36. #
37. path.logs: /home/aitpm/software/elasticsearch-7.7.1/logs
38. #
39. # ----------------------------------- Memory -----------------------------------
40. #
41. # Lock the memory on startup:
42. #
43. #bootstrap.memory_lock: true
44. #
45. # Make sure that the heap size is set to about half the memory available
46. # on the system and that the owner of the process is allowed to use this
47. # limit.
48. #
49. # Elasticsearch performs poorly when the system is swapping the memory.
50. #
51. # ---------------------------------- Network -----------------------------------
52. #
53. # Set the bind address to a specific IP (IPv4 or IPv6):
54. #
55. #network.host: 192.168.0.1
56. network.host: 0.0.0.0
57. #
58. # Set a custom port for HTTP:
59. #
60. http.port: 8011
61. #
62. # For more information, consult the network module documentation.
63. #
64. # --------------------------------- Discovery ----------------------------------
65. #
66. # Pass an initial list of hosts to perform discovery when this node is started:
67. # The default list of hosts is ["127.0.0.1", "[::1]"]
68. #
69. #discovery.seed_hosts: ["host1", "host2"]
70. discovery.seed_hosts: ["10.130.49.84", "10.130.49.96", "10.130.49.97"]
71. #
72. # Bootstrap the cluster using an initial set of master-eligible nodes:
73. #
74. #cluster.initial_master_nodes: ["node-1", "node-2"]
75. cluster.initial_master_nodes: ["10.130.49.84", "10.130.49.96", "10.130.49.97"]
76. #
77. # For more information, consult the discovery and cluster formation module documentation.
78. #
79. # ---------------------------------- Gateway -----------------------------------
80. #
81. # Block initial recovery after a full cluster restart until N nodes are started:
82. #
83. #gateway.recover_after_nodes: 3
84. #
85. # For more information, consult the gateway module documentation.
86. #
87. # ---------------------------------- Various -----------------------------------
88. #
89. # Require explicit names when deleting indices:
90. #
91. #action.destructive_requires_name: true

修改JVM配置：

1. ## JVM configuration
2.  
3. ################################################################
4. ## IMPORTANT: JVM heap size
5. ################################################################
6. ##
7. ## You should always set the min and max JVM heap
8. ## size to the same value. For example, to set
9. ## the heap to 4 GB, set:
10. ##
11. ## -Xms4g
12. ## -Xmx4g
13. ##
14. ## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
15. ## for more information
16. ##
17. ################################################################
18.  
19. # Xms represents the initial size of total heap space
20. # Xmx represents the maximum size of total heap space
21.  
22. -Xms4g
23. -Xmx4g
24.  
25. ################################################################
26. ## Expert settings
27. ################################################################
28. ##
29. ## All settings below this section are considered
30. ## expert settings. Don't tamper with them unless
31. ## you understand what you are doing
32. ##
33. ################################################################
34.  
35. ## GC configuration
36. 8-13:-XX:+UseConcMarkSweepGC
37. 8-13:-XX:CMSInitiatingOccupancyFraction=75
38. 8-13:-XX:+UseCMSInitiatingOccupancyOnly
39.  
40. ## G1GC Configuration
41. # NOTE: G1 GC is only supported on JDK version 10 or later
42. # to use G1GC, uncomment the next two lines and update the version on the
43. # following three lines to your version of the JDK
44. # 10-13:-XX:-UseConcMarkSweepGC
45. # 10-13:-XX:-UseCMSInitiatingOccupancyOnly
46. 14-:-XX:+UseG1GC
47. 14-:-XX:G1ReservePercent=25
48. 14-:-XX:InitiatingHeapOccupancyPercent=30
49.  
50. ## JVM temporary directory
51. -Djava.io.tmpdir=${ES_TMPDIR}
52.  
53. ## heap dumps
54.  
55. # generate a heap dump when an allocation from the Java heap fails
56. # heap dumps are created in the working directory of the JVM
57. -XX:+HeapDumpOnOutOfMemoryError
58.  
59. # specify an alternative path for heap dumps; ensure the directory exists and
60. # has sufficient space
61. -XX:HeapDumpPath=data
62.  
63. # specify an alternative path for JVM fatal error logs
64. -XX:ErrorFile=logs/hs_err_pid%p.log
65.  
66. ## JDK 8 GC logging
67. 8:-XX:+PrintGCDetails
68. 8:-XX:+PrintGCDateStamps
69. 8:-XX:+PrintTenuringDistribution
70. 8:-XX:+PrintGCApplicationStoppedTime
71. 8:-Xloggc:logs/gc.log
72. 8:-XX:+UseGCLogFileRotation
73. 8:-XX:NumberOfGCLogFiles=32
74. 8:-XX:GCLogFileSize=64m
75.  
76. # JDK 9+ GC logging
77. 9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

分别启动3台elasticsearch

1. cd /home/aitpm/software/elasticsearch-7.7.1/bin
2. ./elasticsearch -d

查看启动日志：

1. cd /home/aitpm/software/elasticsearch-7.7.1/logs
2. tail-fn 100 *.log

部署可能出现问题问题及解决方法：

1、max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决：
切换到root用户
执行命令：
sysctl -w vm.max_map_count=262144
查看结果：
sysctl -a|grep vm.max_map_count
显示：
vm.max_map_count = 262144

上述方法修改之后，如果重启虚拟机将失效，所以：
解决办法：
在
/etc/sysctl.conf文件最后添加一行
vm.max_map_count=262144
执行下sysctl -p命令
即可永久修改

2、elasticsearch：max file descriptors [4096] for elasticsearch process is too low
解决方案：
su账号修改 /etc/security/limits.conf
在文件末尾增加以下两行：

• soft nofile 65536
• hard nofile 65536
  修改为退出ssh，并重新进入生效。