Daemonizing and Upstart

文章出处：https://blog.nekoconeko.nl/blog/2014/11/12/daemonizing-and-upstart.html

When creating command-line applications, the user is usually present at the terminal to provide commands to the application and read its output. This is called interactive mode. But sometimes, you want to start a long-running program in the background, so it runs while you’re not present. This is called daemon mode, and the programs themselves are usually called daemons.

Daemons?

In contrast to their mythical counterpart, daemons under Unix are usually much more benign. Daemons are usually started when the system boots, are will continue running until they are manually stopped or a fatal error occurs. Common examples include Apache, MySQL and OpenSSH; these tools are present on a lot of Unix servers. If these daemons are started during boot, this is handled by a process called init.

There are a few implementations of init under Linux, most notably of which are sysvinit and Upstart. Sysvinit was an implementation used by many Linux distributions until not long ago and mimics the init system used by Unix System V. Ubuntu has since moved to their own system called Upstart which, while still somewhat compatible with the traditional Unix way of starting daemons, adds a lot of new features. Some of these features have implications I was not aware of until recently, and can influence the daemonizing process itself.

The Process Tree

Daemonizing takes advantage of a mechanism defined in POSIX that makes sure that any orphaned process is made a subprocess of init. In Unix land there is one process list, and all processes are children of init either directly or indirectly.

On a (very) clean Linux system with Apache and MySQL installed, this might look like this:

init
  ├─apache2 -k start
  │   ├─apache2 -k start
  │   └─apache2 -k start
  └─mysqld_safe /usr/bin/mysqld_safe
      └─mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql--log-error=/var/log/mysql/

As you can see, all processes are children of init, and they themselves may also have subprocesses. Apache has started two worker processes, and MySQL seems to have forked into a master and worker process. When a user logins it might look like this:

init
  ├─apache2 -k start
  │   ├─apache2 -k start
  │   └─apache2 -k start
  ├─login --      
  │   └─bash
  └─mysqld_safe /usr/bin/mysqld_safe
      └─mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql--log-error=/var/log/mysql/

A login process has been created, and bash was started as a subprocess to provide a shell for the user.

In the situations above the apache and mysql processes are daemons, they are running non-interactively and are direct children of init. The bash process is running interactively, and provides the currently logged in user with a shell. We can see this if we were to run a command.

init
  ├─apache2 -k start
  │   ├─apache2 -k start
  │   └─apache2 -k start
  ├─login --      
  │   └─bash
  │       └─tail -f some.log
  └─mysqld_safe /usr/bin/mysqld_safe
      └─mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql--log-error=/var/log/mysql/

The user has started the tail -f some.log command, for which bash has forked a separate child process.

So far, so good. Up until here there is nothing uncommon about the behaviour of init, and everything is working as I would expect from a normal POSIX environment. Let’s look what happens when we introduce Upstart.

Upstart and init

In the previous examples, a user directly logged in on the terminal and a session was started for him. A similar case is when connecting to a machine using SSH, as you also only have a terminal available. In a desktop environment, an user will most likely be using some sort of graphical user environment, like Unity under Ubuntu. Note that in the examples to come, I will be filtering the output of the pstree command to only show relevant information. A desktop environment will cause a LOT of extra processes to spawn, which are not relevant for this article.

The process tree for a graphical user session in which an instance of terminator (my terminal emulator of choice) has been started, may look like this:

init
  ├─apache2 -k start
  │   ├─apache2 -k start
  │   └─apache2 -k start
  ├─lightdm
  │   ├─Xorg -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
  │   └─lightdm --session-child 12 19
  │       └─init --user
  │           ├─/usr/bin/termin /usr/bin/terminator
  │           │   └─bash
  │           └─gnome-session --session=ubuntu
  │               └─compiz
  └─mysqld_safe /usr/bin/mysqld_safe
      └─mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql--log-error=/var/log/mysql/

LightDM is a display manager, which in turn has started the X.org X server. It has also forked itself into a session-child, which in turn has started init --user, terminator and gnome-session. gnome-session has started compiz which will provide the actual graphical user interface that you see and interact with. terminator has also been started, and provides me with a terminal in which I can execute commands. The observant reader will already have noticed something odd in this process tree, but let’s move on.

Let’s what happens when I start a new openssh daemon instance from terminator.

init
  ├─apache2 -k start
  │   ├─apache2 -k start
  │   └─apache2 -k start
  ├─lightdm
  │   ├─Xorg -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
  │   └─lightdm --session-child 12 19
  │       └─init --user
  │           ├─/usr/bin/termin /usr/bin/terminator
  │           │   └─bash
  │           └─gnome-session --session=ubuntu
  │               └─compiz
  ├─mysqld_safe /usr/bin/mysqld_safe
  │   └─mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql--log-error=/var/log/mysql/
  └─sshd

sshd has been started as a child of init, and is now a daemon waiting for connections. Next, let’s start my own daemonizing software using its initd script from terminator. To clean up the output, I have also stopped apache and mysql.

init
  ├─lightdm
  │   ├─Xorg -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
  │   └─lightdm --session-child 12 19
  │       └─init --user
  │           ├─/usr/bin/python
  │           │   ├─openstack-worker
  │           │   ├─openstack-worker
  │           │   ├─openstack-worker
  │           │   ├─openstack-worker
  │           ├─/usr/bin/termin /usr/bin/terminator
  │           │   └─bash
  │           └─gnome-session --session=ubuntu
  │               └─compiz
  └─sshd

Well then, that’s not quite where I expected my daemon to end up. I expected to find it as a child of init.

So what happened?

Apparently, Upstart has a feature where it can be launched in user mode using the --user flag. In this mode, the PR_SET_CHILD_SUBREAPER flag is set on the process, using the prctl system call. This means that processes that are daemonized as children of init --user will not be attached to the main init instance (process ID 1), but on the init --user instance that is the closest parent. In other words, all orphaned children of init --user will be attached to init --user, and not on init.

Why is this a problem?

This system is called Session Init, and the main purpose is to make some features of Upstart available in an user context. The user is able to create his own startup scripts, and use them like the system startup scripts. The difference is that the user doesn’t need root privileges to start his own processes, as you would with most normal initd scripts.

The direct side-effect that I can see is the daemonizing behaviour above. Daemonizing programs and scripts that are not aware of Upstart will be placed under init --user, whether they like it or not. The documentation claims this to be intentional, so at least we’re not running into a bug.

This behaviour is unwanted for my use-case, because I want to be able to start daemons from an user context, that run in the system context (as a child of the main init).

The solution

There is no way of circumventing the behaviour of init --user and PR_SET_CHILD_SUBREAPER that I know of. It also seems that normal initd scripts, like those of Apache and MySQL, are also affected by this. All initd scripts that have been converted into Upstart jobs are unaffected. Or rather, Upstart provides a mechanism to launch a system daemon even though the command was executed by a process under init --user. Upstart jobs are executed using either the service command or using the start, stop, restart and status commands. Both use the same mechanism, which is uses a named Unix socked to connect directly to the main init instance. The main init instance will execute the command on behalf of the user. This way, any newly started process will never start as a child of init --user, because they were started directly by init.

Based on this behaviour, the solution would then to simply create an Upstart job for your daemon. An alternative would be to disable the usage of init --user, unfortunately this complicates the startup of your graphical session somewhat. You can find an excellent answer detailing all steps that LightDM executes during startup on AskUbuntu. Finally, you could stop using Upstart altogether, but this also complicates matters, because there might be some package or tools that rely on Upstart, because it’s been a part of Ubuntu for a while now.

Well, there is one (very dirty) way to disable the PR_SET_CHILD_SUBREAPER flag on the init --user process. Walter Doekes details in his blog article the exact steps needed to do this. It basically comes down to injecting some code into the running init --user process, which calls prctl and disables the PR_SET_CHILD_SUBREAPER flag. While educational, I can’t see myself using this method on a live environment.

I guess I’ll just write Upstart jobs for new daemons…