Es启动流程
main
启动方法是org.elasticsearch.bootstrap.Elasticsearch.main,代码如下:
public static void main(final String[] args) throws Exception {
// we want the JVM to think there is a security manager installed so that if internal policy decisions that would be based on the
// presence of a security manager or lack thereof act as if there is a security manager present (e.g., DNS cache policy)
System.setSecurityManager(new SecurityManager() {
@Override
public void checkPermission(Permission perm) {
// grant all permissions so that we can later set the security manager to the one that we want
}
});
LogConfigurator.registerErrorListener();
final Elasticsearch elasticsearch = new Elasticsearch();
int status = main(args, elasticsearch, Terminal.DEFAULT);
if (status != ExitCodes.OK) {
exit(status);
}
}
做了三件事:
- 加载安全配置
- 注册错误监听器
- 创建Elasticsearch实例。如果创建失败则exit
接下来细看一下其中的main()。
static int main(final String[] args, final Elasticsearch elasticsearch, final Terminal terminal) throws Exception {
return elasticsearch.main(args, terminal);
}
Elasticsearch的静态main函数调用了一个非静态的main。这个main的实现位于Command中。
Command::main步骤:
- 添加Shutdown Hook。
Shutdown Hook是jvm在关闭服务时用于清理关闭资源的机制。当jvm关闭时,会先调用addShutdownHook中添加的所有Hook,系统执行完毕后,jvm才会关闭。因此,Shutdown Hook可以在关闭jvm时进行内存清理、对象销毁等操作。
- beforeMain.run()执行main之前的操作,如日志配置的加载。
- 调用mainWithoutErrorHandling。该方法位于Command类中。
public final int main(String[] args, Terminal terminal) throws Exception {
if (addShutdownHook()) {
shutdownHookThread = new Thread(() -> {
try {
this.close();
} catch (final IOException e) {
try (
StringWriter sw = new StringWriter();
PrintWriter pw = new PrintWriter(sw)) {
e.printStackTrace(pw);
terminal.println(sw.toString());
} catch (final IOException impossible) {
// StringWriter#close declares a checked IOException from the Closeable interface but the Javadocs for StringWriter
// say that an exception here is impossible
throw new AssertionError(impossible);
}
}
});
Runtime.getRuntime().addShutdownHook(shutdownHookThread);
}
beforeMain.run();
try {
mainWithoutErrorHandling(args, terminal);
} catch (OptionException e) {
printHelp(terminal);
terminal.println(Terminal.Verbosity.SILENT, "ERROR: " + e.getMessage());
return ExitCodes.USAGE;
} catch (UserException e) {
if (e.exitCode == ExitCodes.USAGE) {
printHelp(terminal);
}
terminal.println(Terminal.Verbosity.SILENT, "ERROR: " + e.getMessage());
return e.exitCode;
}
return ExitCodes.OK;
}
Command.mainWithoutErrorHandling中执行启动指令,代码如下:
void mainWithoutErrorHandling(String[] args, Terminal terminal) throws Exception {
final OptionSet options = parser.parse(args);
if (options.has(helpOption)) {
printHelp(terminal);
return;
}
if (options.has(silentOption)) {
terminal.setVerbosity(Terminal.Verbosity.SILENT);
} else if (options.has(verboseOption)) {
terminal.setVerbosity(Terminal.Verbosity.VERBOSE);
} else {
terminal.setVerbosity(Terminal.Verbosity.NORMAL);
}
execute(terminal, options);
}
核心代码是execute()。在Command类当中,该方法是个抽象方法:
protected abstract void execute(Terminal terminal, OptionSet options) throws Exception;
那么具体执行的是哪个实现,就要看一下上文中涉及的两个类的继承关系。
execute
程序入口所在的Elasticsearch继承了EnvironmentAwareCommand, EnvironmentAwareCommand继承Command并实现execute方法。
因此在调用elasticSearch.main()时,实际执行的是EnvironmentAwareCommand的execute实现。实现代码如下:
protected void execute(Terminal terminal, OptionSet options) throws Exception {
final Map<String, String> settings = new HashMap<>();
for (final KeyValuePair kvp : settingOption.values(options)) {
if (kvp.value.isEmpty()) {
throw new UserException(ExitCodes.USAGE, "setting [" + kvp.key + "] must not be empty");
}
if (settings.containsKey(kvp.key)) {
final String message = String.format(
Locale.ROOT,
"setting [%s] already set, saw [%s] and [%s]",
kvp.key,
settings.get(kvp.key),
kvp.value);
throw new UserException(ExitCodes.USAGE, message);
}
settings.put(kvp.key, kvp.value);
}
putSystemPropertyIfSettingIsMissing(settings, "path.data", "es.path.data");
putSystemPropertyIfSettingIsMissing(settings, "path.home", "es.path.home");
putSystemPropertyIfSettingIsMissing(settings, "path.logs", "es.path.logs");
execute(terminal, options, createEnv(terminal, settings));
}
上述方法先读取命令行配置,并检查其中三个关键配置es.path.data、es.path.home、es.path.logs。接下来在进行execute。
这里的execute在EnvironmentAwareCommand中同样也是抽象函数:
protected abstract void execute(Terminal terminal, OptionSet options, Environment env) throws Exception;
这里执行的是Elasticsearch::execute的实现。
protected void execute(Terminal terminal, OptionSet options, Environment env) throws UserException {
...
try {
init(daemonize, pidFile, quiet, env);
} catch (NodeValidationException e) {
throw new UserException(ExitCodes.CONFIG, e.getMessage());
}
}
上述代码省略了参数校验的部分,我们直接关注init方法。
init
void init(final boolean daemonize, final Path pidFile, final boolean quiet, Environment initialEnv)
throws NodeValidationException, UserException {
try {
Bootstrap.init(!daemonize, pidFile, quiet, initialEnv);
} catch (BootstrapException | RuntimeException e) {
// format exceptions to the console in a special way
// to avoid 2MB stacktraces from guice, etc.
throw new StartupException(e);
}
}
Elasticsearch::init调用了Bootstrap.init方法,es启动的执行过程是在方法中。代码很长,但是主要做了以下几个步骤:
- 创建Bootstrap实例
- 加载安全配置
- 加载环境配置
- 配置logger
- 创建pid文件
- 调用Bootstrap::setup
- 调用Bootstrap::start
static void init(
final boolean foreground,
final Path pidFile,
final boolean quiet,
final Environment initialEnv) throws BootstrapException, NodeValidationException, UserException {
// force the class initializer for BootstrapInfo to run before
// the security manager is installed
BootstrapInfo.init();
INSTANCE = new Bootstrap();
final SecureSettings keystore = loadSecureSettings(initialEnv);
final Environment environment = createEnvironment(foreground, pidFile, keystore, initialEnv.settings(), initialEnv.configFile());
try {
LogConfigurator.configure(environment);
} catch (IOException e) {
throw new BootstrapException(e);
}
if (environment.pidFile() != null) {
try {
PidFile.create(environment.pidFile(), true);
} catch (IOException e) {
throw new BootstrapException(e);
}
}
...
INSTANCE.setup(true, environment);
...
INSTANCE.start();
...
} catch (NodeValidationException | RuntimeException e) {
...
}
}
接下来细看一下setup和start方法。
setup方法用于初始化环境配置并创建Node实例。详细步骤如下:
-
environment.settings获取环境配置
-
spawnNativeControllers为每个模块(Plugin)创建native Controller,并通过ProcessBuilder创建子进程(Process)实例
-
initializeNatives初始化本地资源:
- 如果是root用户,抛出错误
- 启用SystemCallFilter(需要JNA)
- 设置mlockAll
mlockAll配置:jvm在gc时会将堆中的页写回磁盘,当再次需要读取该页面时再从磁盘读取。如果Elasticsearch的页被jvm通过gc写入磁盘,再次进行读取时会产生大量磁盘抖动从而影响Es性能。因此,可以通过开启memory lock将Es的堆页面进行锁定,不允许gc将Es的堆页面进行交换,防止上述情况产生。
https://www.elastic.co/guide/en/elasticsearch/reference/6.3/_memory_lock_check.html
- 创建Windows关闭事件的监听器,再监听器当中调用Bootstrap::stop关闭服务
- 初始化Lucene随机数种子
-
initializeProbes初始化探测器。该方法实例化了两种探测器:ProcessProbe和OsProbe。分别对进程情况和系统情况进行监控。
-
添加Shutdown Hook
-
检查Jar Hell
-
格式化输出ifconfig日志
-
初始化SecurityManager
-
创建Node节点。Node的创建还不是这一部分的重点,放到Node相关内容细看。
Jar Hell:jar包冲突
private void setup(boolean addShutdownHook, Environment environment) throws BootstrapException {
// 1. 获取环境配置
Settings settings = environment.settings();
// 2. 创建native controllers并创建进程实例
try {
spawner.spawnNativeControllers(environment);
} catch (IOException e) {
throw new BootstrapException(e);
}
// 3. 初始化本地资源
initializeNatives(
environment.tmpFile(),
BootstrapSettings.MEMORY_LOCK_SETTING.get(settings),
BootstrapSettings.SYSTEM_CALL_FILTER_SETTING.get(settings),
BootstrapSettings.CTRLHANDLER_SETTING.get(settings));
// 4. 初始化进程和系统监控
// initialize probes before the security manager is installed
initializeProbes();
// 5. 添加shutdown hook
if (addShutdownHook) {
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
try {
IOUtils.close(node, spawner);
LoggerContext context = (LoggerContext) LogManager.getContext(false);
Configurator.shutdown(context);
} catch (IOException ex) {
throw new ElasticsearchException("failed to stop node", ex);
}
}
});
}
// 6. 检查jar包冲突
try {
// look for jar hell
final Logger logger = ESLoggerFactory.getLogger(JarHell.class);
JarHell.checkJarHell(logger::debug);
} catch (IOException | URISyntaxException e) {
throw new BootstrapException(e);
}
// 7. 输出ifconfig日志
// Log ifconfig output before SecurityManager is installed
IfConfig.logIfNecessary();
// 8. 初始化SecurityManager
// install SM after natives, shutdown hooks, etc.
try {
Security.configure(environment, BootstrapSettings.SECURITY_FILTER_BAD_DEFAULTS_SETTING.get(settings));
} catch (IOException | NoSuchAlgorithmException e) {
throw new BootstrapException(e);
}
// 9. 创建Node实例
node = new Node(environment) {
@Override
protected void validateNodeBeforeAcceptingRequests(
final BootstrapContext context,
final BoundTransportAddress boundTransportAddress, List<BootstrapCheck> checks) throws NodeValidationException {
BootstrapChecks.check(context, boundTransportAddress, checks);
}
};
}
到这里,节点的相关配置就设置完成了。接下来执行Bootstrap::start。
start方法的作用是启用节点和KeepAliveThread。代码如下:
private void start() throws NodeValidationException {
node.start();
keepAliveThread.start();
}
node.start()的内容不在这里详细说。我们来看看keepAliveThread的定义。
private final CountDownLatch keepAliveLatch = new CountDownLatch(1);
private final Thread keepAliveThread;
/** creates a new instance */
Bootstrap() {
keepAliveThread = new Thread(new Runnable() {
@Override
public void run() {
try {
keepAliveLatch.await();
} catch (InterruptedException e) {
// bail out
}
}
}, "elasticsearch[keepAlive/" + Version.CURRENT + "]");
keepAliveThread.setDaemon(false);
// keep this thread alive (non daemon thread) until we shutdown
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
keepAliveLatch.countDown();
}
});
}
在Bootstrap的构造函数中初始化了keepAliveThread。其中KeepAliveLatch的详细介绍与应用见:
https://blog.youkuaiyun.com/u012637358/article/details/90288585
在构造函数中,keepAliveLatch在Shutdown Hook当中才会调用countDown,因此keepAlive线程会一直处于await直到服务被关闭。
为什么要有一个KeepAlive线城呢?在Es的代码中,main函数执行完毕后会返回ExitCode.OK并退出,查询请求的执行将由NodeClient来执行。而主线程退出后,需要维持一个用户线程来保持进程不退出。