Solaris™ 10 System Administration Essentials-2 Boot, Service Management, and Shutdown

Chapter 2 Boot, Service Management, and Shutdown 33

2.1 Boot 33

2.1.1 The Bootloader 33

2.1.2 The Kernel 34

2.1.3 User-Mode Programs 34

2.1.4 GRUB Extensions 35

From the Library of Daniel Johnson

vi Contents

2.1.5 Modifying Boot Behavior 36

2.1.6 Run Levels 37

2.1.7 Troubleshooting 37

2.2 Service Management Facility 39

2.2.1 enabled 40

2.2.2 state, next_state, and state_time 40

2.2.3 logfile 41

2.2.4 dependency 41

2.2.5 How SMF Interacts with Service Implementations 42

2.2.6 The Service Configuration Facility 44

2.2.7 Health and Troubleshooting 44

2.2.8 Service Manifests 45

2.2.9 Backup and Restore of SCF Data 45

2.3 Shutdown 46

2.3.1 Application-Specific Shutdown 46

2.3.2 Application-Independent Shutdown 46

 

Boot, Service

Management, and

Shutdown

This chapter describes how the Solaris 10 operating system boots and explains

options users and administrators have for changing the boot process. The chapter

also describes the two methods of shutting down a Solaris 10 system. In addition, it

describes the Service Management Facility (SMF) utility for managing system services.

Some of the information in this chapter describes Solaris boot processes that

apply to both the x86 and SPARC platform, but the chapter focuses primarily on

booting the x86 platform.

2.1 Boot

Like most contemporary operating systems, Solaris initialization begins with the

bootloader, continues with the kernel, and finishes with user-mode programs.

2.1.1 The Bootloader

On x86 platforms, the Solaris 10 OS is designed to be loaded by GNU Grand Unified

Bootloader (GRUB). By default, the bootloader displays a boot menu with two

entries:

Solaris 10 10/08 s10x_u6wos_07b X86

Solaris failsafe

From the Library of Daniel Johnson

34 Chapter 2 _ Boot, Service Management, and Shutdown

When a Solaris boot entry is chosen, GRUB loads the kernel specified by the entry

into memory and transfers control to it. The entry also directs GRUB to load a boot

archive with copies of the kernel modules and configuration files essential for startup.

See the boot(1M) manual page for more about the boot archive. The failsafe

entry facilitates troubleshooting and recovery.

Note that the GRUB that is supplied with Solaris contains extensions to GNU

GRUB required to load the Solaris OS.

2.1.2 The Kernel

The kernel starts by initializing the hardware, clearing the console, and printing a

banner:

After hardware initialization, the kernel mounts the root file system and

executes user-mode programs.

2.1.3 User-Mode Programs

As with all UNIX operating systems, most Solaris functionality is driven by usermode

programs. The kernel starts them by executing the /sbin/init file in the

first process, which always has process ID (“pid”) 1.

Like other UNIX operating systems, init reads the /etc/inittab configuration

file and executes programs according to it. Unlike most UNIX operating

systems, the default inittab does not instruct init to execute init scripts in the

/etc/rc*.d directories. Instead, the processes that implement most systemdelivered

functionality on Solaris are started by the service management facility or

SMF. Accordingly, the Solaris init contains special-purpose functionality to start

and restart (as necessary) the daemons that implement SMF. In turn, the facility

is responsible for executing the init scripts. SMF is described in more detail in the

next section.

Users accustomed to the Solaris 9 operating system will notice that the Solaris

10 operating system displays much less information on the console during boot.

This is because SMF now starts service daemons with standard output directed to

log files in /var/svc/log, rather than the console.

SunOS Release 5.10 Version Generic_137138-06 64-bit

Copyright 1983-2008 Sun Microsystems, Inc. All rights reserved.

Use is subject to license terms.

From the Library of Daniel Johnson

2.1 BOOT 35

Near the end of startup, SMF will execute the ttymon program on the console

device at the direction of the console-login SMF service:

examplehost login:

If the SUNWdtlog package was installed, SMF will also start an X server on the

console device and the dtlogin greeter on the display as part of the cde-login

SMF service as shown in Figure 2.1.

2.1.4 GRUB Extensions

The GRUB installed by Solaris differs from standard GNU GRUB in a few ways:

_ It can read Solaris UFS file systems (which differ from BSD UFS file systems).

_ It recognizes the kernel$ and module$ commands (since 10/08 release).

Figure 2.1 SMF Login

From the Library of Daniel Johnson

36 Chapter 2 _ Boot, Service Management, and Shutdown

_ It can read Solaris ZFS pools, and recognizes the bootfs command (since

10/08 release).

_ It recognizes the findroot command (since 10/08 release).

As a result, versions of GRUB not delivered with Solaris will generally not be able

to boot a Solaris system image.

2.1.5 Modifying Boot Behavior

The Solaris kernel can accept a string of boot arguments from the bootloader. Recognized

arguments are listed in the kernel(1M) manual page. Commonly used

arguments are shown in Table 2.1.

The boot arguments for a single boot sequence can be set from the GRUB menu.

Select an entry and press the e key. GRUB will display the entry editing screen, as

shown in Figure 2.2.

Figure 2.3 shows the GRUB edit menu. In this menu, you can modify the kernel

behavior for a specified boot entry. This menu is accessed at boot time, by typing e

to interrupt the boot process, then with the boot entry selected, typing e again to

enter the edit menu for the selected entry.

Select the line beginning with kernel and press the e key.

After the path for unix, add the boot arguments. Press enter to commit the

change and b to boot the temporarily modified entry.

Boot arguments for a single boot can also be set on the reboot command line.

See the reboot(1M) manual page.

Boot arguments can be installed permanently by modifying menu.lst file. Use

bootadm list-menu to locate the file in the file system.

Table 2.1 Boot Arguments

Argument Description

-k Start the kernel debugger, kmdb, as soon as possible. See

the kmdb(1M) manual page and later in this chapter.

-s Single-user mode. Start only basic services and present an

sulogin prompt.

-v Be verbose by printing extra information on the console.

-m verbose Instruct the SMF daemons to be verbose.

From the Library of Daniel Johnson

2.1 BOOT 37

2.1.6 Run Levels

The Solaris OS defines eight run levels. Each run level is associated with particular

system behaviors (see Table 2.2).

By default, Solaris boots into run level 3. This is taken from the initdefault

entry of the /etc/inittab configuration file (see inittab(4)). It can be changed to

a single boot sequence by specifying -s in the boot arguments (refer to Table 2.2).

To change the run level while the operating system is running, use the init

command. See its manual page for a detailed description of run levels.

2.1.7 Troubleshooting

If you encounter problems during the boot process, check the tools and solutions

described here for a remedy.

Figure 2.2 Editing a GRUB Entry

From the Library of Daniel Johnson

38 Chapter 2 _ Boot, Service Management, and Shutdown

Figure 2.3 Editing the GRUB Menu at Boot Time

Table 2.2 Run Levels and Corresponding System Behaviors

Run Level Behavior

S Single-user mode. No login services running except for sulogin on the

console.

0 The operating system is shut down and the computer is running its firmware.

1 Like S, except applications which deliver into /etc/rc1.d are also started.

2 Multi-user mode. Local login services running. Some applications—usually

local—may be running.

3 Multi-user server mode. All configured services and applications running,

including remote login and network-visible applications.

4 Alternative multi-user server mode. Third-party applications may behave

differently than under run level 3.

5 Powered off.

6 Reboot.

From the Library of Daniel Johnson

2.2 SERVICE MANAGEMENT FACILITY 39

2.1.7.1 Milestone none

If a problem prevents user programs from starting normally, the Solaris 10 OS

can be instructed to start as few programs as possible during boot by

specifying -m milestone=none in the boot arguments. Once logged in,

svcadm milestone all can be used to instruct SMF to continue initialization

as usual.

2.1.7.2 Using the kmdb Command

If a problem prevents the kernel from starting normally, then it can be started

with the assembly-level kernel debugger, kmdb. When the -k option is specified in

the boot arguments, the kernel loads kmdb as soon as possible. If the kernel panics,

kmdb will stop the kernel and present a debugging prompt on the console. If

the -d option is also specified, kmdb will stop the kernel and present a debugging

prompt as soon as it finishes loading. For more information, see the kmdb(1) manual

page.

2.1.7.3 Failsafe boot

The second GRUB menu entry installed by default is labeled “failsafe”. Selecting it

will start the same kernel, but with the failsafe boot archive. It contains copies of the

kernel modules and configuration files as delivered by the installer, without any user

modifications. By default it also launches an interactive program that facilitates

updating the normal boot archive for instances of the Solaris OS found on the disk.

2.2 Service Management Facility

The service management facility provides means for computer administrators to

observe and control software services. Each service is modeled as an instance of an

SMF service, which allows for a single service implementation to be executed multiple

times simultaneously, as many are capable of doing.

Services and service instances are named by character strings. For example, the

service implemented by cron(1M) is named system/cron, and Solaris includes

an instance of it named default. Tools usually refer to service instances with

fault management resource identifiers, or FMRIs, which combine the service name

and the instance name. The FMRI of the default instance of cron is svc:/

system/cron:default. The service instances known to SMF can be listed with

the svcs -a command. For convenience, most SMF tools accept abbreviations for

service FMRIs – see svcadm(1M)’s manual page.

Service implementations are controlled by the SMF service manager,

svc.startd(1M). The current status and other information for service instances

From the Library of Daniel Johnson

40 Chapter 2 _ Boot, Service Management, and Shutdown

are printed by the svcs command. The -l (ell) option produces long output, like

the following:

The first line, labeled fmri, contains the full FMRI of the service instance. The

name line provides a short description. The remaining output is explained later.

2.2.1 enabled

The service manager considers each service instance to be enabled or disabled.

When enabled, the service manager will attempt to start a service instance’s

implementation and restart it as necessary; when disabled, the facility will try to

stop the implementation if it has been started. Whether a service is enabled can be

changed with svcadm’s enable and disable subcommands.

2.2.2 state, next_state, and state_time

To decide whether a service implementation should be started, the service manager

always considers each service instance to be in one of six states.

examplehost$ svcs -l cron

fmri svc:/system/cron:default

name clock daemon (cron)

enabled true

state online

next_state none

state_time Mon Mar 16 18:25:34 2009

logfile /var/svc/log/system-cron:default.log

restarter svc:/system/svc/restarter:default

contract_id 66

dependency require_all/none svc:/system/filesystem/local (online)

dependency require_all/none svc:/milestone/name-services (online)

disabled The service implementation has not been started, or has

been stopped.

offline The service is not running, but will be started when its

dependencies are met.

online The service has started successfully.

degraded The service is running, but at reduced functionality or

performance.

maintenance An operation failed and administrative intervention is

required.

uninitialized The service’s restarter has not taken control of the service

(restarters are explained later in this chapter).

From the Library of Daniel Johnson

2.2 SERVICE MANAGEMENT FACILITY 41

While a service is in a stable state, the next_state field is none. While an

operation to change the state of a service is incomplete, next_state will contain

the target state. For example, before a service implementation is started the

service manager sets the next_state to online, and if the operation succeeds,

the service manager changes state and next_state to online and none,

respectively.

The state_time line lists the time the state or next_state fields were

updated. This time is not necessarily the last time the service instance changed

states since SMF allows transitions to the same state.

2.2.3 logfile

The service manager logs some information about service events to a separate file

for each service instance. This field gives the name of that file.

2.2.3.1 restarter and contract_id

The service’s restarter interacts with the service’s implementation, and the contract

ID identifies the processes that implement the service. Details of both are

explained in Section 2.2.5, “How SMF Interacts with Service Implementations.”

2.2.4 dependency

These lines list the dependencies of the service instance. SMF dependencies represent

dependencies of the service implementation on other services. The service

manager uses dependencies to determine when to start, and sometimes when to

stop, service instances.

Each dependency has a grouping and a set of FMRIs. The grouping dictates

when a dependency should be considered satisfied. SMF recognizes four dependency

groupings.

require_all All services indicated by the FMRIs must be in the online or

degraded states to satisfy the dependency.

require_any At least one cited service must be online or degraded to

satisfy the dependency.

optional_all The dependency is considered satisfied when all cited services

are online, degraded, disabled, in the maintenance

state, or are offline and will eventually come online

without administrative intervention. Services that don’t exist are

ignored.

exclude_all All cited services must be disabled or in the maintenance

state to satisfy the dependency.

From the Library of Daniel Johnson

42 Chapter 2 _ Boot, Service Management, and Shutdown

When a service is enabled, the service manager will not start it until all of its

dependencies are satisfied. Until then, the service will remain in the offline

state.

The service manager can also stop services according to dependencies. This

behavior is governed by the restart_on value of the dependency, which may take

one of four values.

2.2.5 How SMF Interacts with Service Implementations

SMF manages most services through daemons, though it manages some with what

is called “transient service.” In cases where neither daemons nor transient service

is appropriate, SMF allows for alternative service starters.

2.2.5.1 Services Implemented by Daemons

SMF starts a service implemented by daemons by executing its start method. The

start method is a program specified by the service author; its path and arguments

are stored in the SCF data for the service. (SCF is described in the next section.) If

the method exits with status 0, the service manager infers that the service has

started successfully (the daemons were started in the background and are ready to

provide service) and transitions its state to online. If the method exits with status

1, the service manager concludes that the service failed and re-executes the

method. If the method fails three times consecutively, then the service manager

gives up, transitions the service to the maintenance state, and appends a note to

the service’s SMF log file in /var/svc/log. In all cases, the method is started

with its standard output redirected to the service’s SMF log file. The service daemon

will inherit this unless the author wrote the start method to do otherwise.

After starting a service implemented by a daemon, the service manager will

monitor its processes. If all processes exit, then the service manager will infer that

the service has failed and will attempt to restart it by re-executing the start

method. If this happens more than ten times in ten seconds, then the service manager

will give up and transition the service to the maintenance state. Processes

are monitored through a process contract with the kernel. Contracts are a

new kernel abstraction documented in contract(4); process-type contracts are

none Do not stop the service if the dependency service is stopped.

error Stop the service if the dependency is stopped due to a software or

hardware error.

restart Stop the service if the dependency is stopped for any reason.

refresh Stop the service if the dependency is stopped or its configuration is

changed (refreshed).

From the Library of Daniel Johnson

2.2 SERVICE MANAGEMENT FACILITY 43

documented in process(4). Services treated by the service manager in this way

are referred to as contract services.

To stop a contract service, the service manager executes the stop method specified

by the service author. Stop methods exit with status 0 to signal that the service

has been stopped successfully, in which case the service manager will

transition the service to the disabled state. However, the facility uses process contracts

to ensure that a contract service has been fully stopped. If a service’s stop

method exits with status 0 but processes remain in the contract, then svc.startd

will send SIGKILL signals to the processes once each second until they have exited.

Each time, svc.startd records a note in the service’s /var/svc/log file.

The processes associated with a contract service can be listed with the svcs -p

command. To examine the contract itself, obtain its ID number from the svcs -v

command or the contract_id line of the output of svcs -l and pass it to the

ctstat(1) command.

2.2.5.2 Services Not Implemented by Daemons

Some services are not implemented by daemons. For example, the file system services

(e.g., svc:/system/filesystem/minimal:default) represent behavior

implemented by the kernel. Instead of representing whether the behavior is available

or not, the file system services represent whether parts of the file system

namespace that are allowed to be separate file systems (/var, /var/adm, /tmp)

have been mounted and are available. Ensuring this is the case does require programs

to be executed (e.g., mount(1M)), but the service should still be considered

online once those programs have exited successfully.

For such services, svc.startd provides the transient service model. After the

start method exits with status 0, the service is transitioned to online and any

processes it may have started in the background are not monitored.

2.2.5.3 Alternative Service Models

If a service author requires SMF to interact with his service in still a different way,

then the facility allows him to provide or specify an alternative service restarter.

When an author specifies a service restarter for a service, the facility delegates

interaction with the service to the restarter, which must itself be an SMF service.

Solaris 10 includes a single alternative restarter: inetd(1M). inetd defers execution

of a service’s daemon until a request has been received by a network device.

Before then, inetd reports services delegated to it to be online to signify readiness,

even though no daemons may have been started. Operations specific to

inetd-supervised services can be requested with the inetadm(1M) command.

The restarter for a service is listed by the svcs -l command. Services governed

by the models provided directly by the service manager are listed with the

special FMRI of svc:/system/svc/restarter:default as their restarter.

From the Library of Daniel Johnson

44 Chapter 2 _ Boot, Service Management, and Shutdown

Since restarters usually require distinct SCF configuration for the services they

control, the facility does not provide a way for an administrator to change the

restarter specified for a service.

2.2.6 The Service Configuration Facility

The enabled status, dependencies, method specifications, and other information for

each service instance are stored in a specialized database introduced with SMF

called the service configuration facility. SCF is implemented by the libscf(3LIB)

library and svc.configd(1M) daemon, and svccfg(1M) provides the most direct

access to SCF for command line users.

In addition to SMF-specific configuration, the libscf(3LIB) interfaces are documented

so that services can store service-specific configuration in SCF as well.

2.2.7 Health and Troubleshooting

Standard records of enabled status and states for each service permit an easy check

for malfunctioning services. The svcs -x command, without arguments, identifies

services that are enabled but not in the online state and attempts to diagnose why

they are not running. When all enabled services are online, svcs -x exits without

printing anything.

When a service managed by SMF is enabled but not running, investigation

should start by retrieving the service manager’s state for the service, usually with

the svcs command:

If the state is maintenance, then the service manager’s most recent attempt

to start (or stop) the service failed. The svcs -x command may explain precisely

why the service was placed in that state. The SMF log file for the service in

/var/svc/log should also provide more information. Note that many services

still maintain their own log files in service-specific locations.

When the problem with a service in the maintenance state is resolved, the

svcadm clear command should be executed for the service. The service manager

will re-evaluate the service’s dependencies and start it, if appropriate.

If a service isn’t running because it is in the offline state, SMF considers its

dependencies to be unsatisfied. svcs -l will display the dependencies and their

states, but if one of them is also offline, then following the chain can be

tedious. svcs -x, when invoked for an offline service, will automatically

examplehost$ svcs cron

STATE STIME FMRI

online Mar_16 svc:/system/cron:default

From the Library of Daniel Johnson

2.2 SERVICE MANAGEMENT FACILITY 45

follow dependency links to find the root cause of the problem, even if it is multiple

links away.

2.2.8 Service Manifests

To deliver an SMF service, the author must deliver a service manifest file into a

subdirectory of /var/svc/manifest. These files conform to the XML file format

standard and describe the SCF data SMF requires to start and interact with the

service. On each boot, the service manifests in /var/svc/manifest are loaded

into the SCF database by the special svc:/system/manifest-import:default

service.

Service manifests can also be imported directly into the SCF repository with the

svccfg import command. It allows new SMF services to be created, including

SMF services to control services that were not adapted to SMF by their authors.

2.2.9 Backup and Restore of SCF Data

SMF provides three methods for backing up SCF data.

2.2.9.1 Automatic

During each boot, SMF automatically stores a backup of persistent SCF data in a

file whose path begins with /etc/svc/repository-boot-. Furthermore, whenever

SMF notices that a file in a subdirectory of /var/svc/manifest has

changed, the facility creates another backup of persistent SCF data after it

has been updated according to the new files; the names of these backups begin

with /etc/svc/repository-manifest_import-. In both cases, only the four

most recent backups are retained and older copies are deleted. Two symbolic links,

repository-boot and repository-manifest_import, are updated to refer to

the latest copy of the respective backup type.

The SCF database may be restored by copying one of these files to

/etc/svc/repository.db. However, this must not be done while the svc.configd

daemon is executing.

2.2.9.2 Repository-wide

All persistent SCF data may be extracted with the svccfg archive command. It

can be restored with the svccfg restore command.

2.2.9.3 Service-specific

The SCF data associated with the instances of a particular service may be

extracted with the svccfg extract command. Note that the command only

accepts service FMRIs and not instance FMRIs. To restore the service instances for

From the Library of Daniel Johnson

46 Chapter 2 _ Boot, Service Management, and Shutdown

such a file, delete the service with svccfg delete and import the file with

svccfg import.

2.3 Shutdown

Solaris provides two main mechanisms to shut down the operating system. They

differ in how applications are stopped.

2.3.1 Application-Specific Shutdown

With appropriate arguments, the shutdown(1M) and init(1M) commands begin

operating system shutdown by instructing SMF to stop all services. The facility

complies by shutting down services in reverse-dependency order, so that each service

is stopped before the services it depends on are stopped since Solaris 10 11/06

release. Upon completion, the kernel flushes the file system buffers and powers off

the computer, unless directed otherwise by the arguments.

As in Solaris 9, the kill init scripts (/etc/init.d/K*) for the appropriate runlevel

are run at the beginning of shutdown. This is in parallel with SMF’s shutdown

sequence.

If an SMF service takes longer to stop than the service’s author specified, SMF

will complain and start killing the service’s processes once every second until they

have exited.

2.3.2 Application-Independent Shutdown

The reboot(1M), halt(1M), and poweroff(1M) commands skip both the SMF

shutdown sequence explained previously and the init scripts and instead stop

applications by sending a SIGTERM signal to all processes. After a 5 second wait,

any remaining processes are sent a SIGKILL signal before the kernel flushes the

file system buffers and stops. Since these commands don’t invoke the stop procedures

provided by application authors, this method has a chance of stopping applications

before they have written all of their data to nonvolatile storage.

From the Library of Daniel Johnson