将一个zone clone在flash上

Zones in a Flash - Literally

From: http://blogs.sun.com/bobn/date/20070320

Fantastic improvements have been made in the Solaris installation and upgrade process - even more in OpenSolaris (available in the various community releases). As we examined the cloning feature introduced in Solaris 10 11/06, it became apparent that we have stumbled upon a most intriguing capability. When combining zone cloning with the attach/detach capability we have discovered a model for flashing zones: zoneflash.

In a recent boot camp we took a look at this in more detail. Unfortunately the slides (which will be posted soon) didn't quite follow the level of depth we were exploring. Several people asked for notes on how this works - and here they are. The irony is that it will take longer to read about it than it does to perform the actual process - but it is so cool.

The Promise

We start with a fresh Solaris system. In this case just live upgraded from media, but it could have been jumpstarted from media or a flash archive. The key point here is that the system has had very little done to it, other than naming and some software installation. Since zone attach makes sure that key system components (specifically packages and patches) are compatible, it makes sense to build our flashzones on a system that will look similar to those that will be built in the future.

So how many zones will we build ? That's a good question. If this were system flasharchives the answer would be as few as possible - one per architecture in the most efficient case. But these zoneflashes are different - just applications, some metadata, and perhaps some customizations (naming, security, SMF). It seems reasonable to create one zoneflash for each type of application server you would deploy - think of it as a userspace template. In this example I have chosen four: a blank uncustomized flash (for building a new zoneflash in a flash), database server (MySQL), web server (apache2), and the community edition of webmin (just another application).

Our procedure will be to build a minimal default zoneflash, run it through first boot to populate the SMF repository, and then clone it for the remaining zoneflashes. Each of these will be booted, customized for the particular application, and tested to make sure everything is operating properly.

We will then detach the zones and move the detached zoneroots onto some media that can be transported. Of course, keeping with the theme of zones and flash, the transport could be the flasharchive itself. How cool would it be to jumpstart a server using flasharchives and have all the application zones already present in a known location, such as /zoneflash ? Unfortunately, I'm sitting in seat 18A on an American Airlines flight to Los Angeles and don't quite have the required infrastructure to do that sort of test. But I do have a USB stick and multiple boot environments. That will do nicely.

Once attached, we will clone the zoneflashes as necessary, adding resources (network, local filesystems) and attributes (resource controls) required for the proper operation of the application. When finished we will detach the zoneflashes so they may be used elsewhere.

The Turn

The first step is to build and boot a simple generic sparse root zone. Since this zone isn't really meant for operation, most zonecfg attributes (network configuration, resource limits, et al) will be skipped. We will add them later when we build the real zones - remember, these are just user space application templates.

# zonecfg -z flashdefault
flashdefault: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:flashdefault> create
zonecfg:flashdefault> set zonepath=/local/default
zonecfg:flashdefault> add inherit-pkg-dir; set dir=/opt; end
zonecfg:flashdefault> commit
zonecfg:flashdefault> exit
#
# zoneadm -z default install 

A few minutes later we have an installed zone, ready for first boot. Since I've attended my Solaris Zones Best Practices class, or at least read the materials, I know how to build a sysidcfg file that will satisfy the sysidtool first boot service. This will allow the zone to boot up all the way without any additional console interaction. Let's do that for our new zone.

# echo > /local/default/root/etc/sysidcfg <<EOF
name_service=NONE
nfs4_domain=dynamic
security_policy=NONE
root_password=xxxxxxxxxx        You supply your own encrypted string from /etc/shadow, I'm not going to post mine!
system_locale=C
terminal=ansi
timezone=US/Central
network_interface=NONE {hostname=default}
EOF
# zoneadm -z default boot
# zlogin -C default 

We need to let first boot processing complete. Since we supplied a valid sysidcfg, it is just a matter of waiting for manifest-import and sysidtool to complete their magic. When complete, login in and take a look around to make sure all is well. Once satisfied, shut down the zone (either from inside the zone or from the global zone) - we are through with it for now.

(from the global zone)
# zoneadm -z default halt 

Now we are done with this first zone. Time to clone it for our remaining application zones. Please pardon a bit of inline shell scripting - I hate to type the same thing over and over and over. Sort of makes for a nice script template, doesn't it ? Not quite the sophistication of Brad Digg's zonemanager, but it will do nicely for our example.

# for zone in webmin mysql web
? do
        echo "create -t default; set zonepath=/local/${zone}" | zonecfg -z ${zone}
        zoneadm -z ${zone} clone default
        echo "name_service=NONE" > /local/${zone}/root/etc/sysidcfg
        echo "nfs4_domain=dynamic" >> /local/${zone}/root/etc/sysidcfg
        echo "security_policy=NONE" >> /local/${zone}/root/etc/sysidcfg
        echo "root_password=xxxxxxxxxxx" >> /local/${zone}/root/etc/sysidcfg
        echo "system_locale=C" >> /local/${zone}/root/etc/sysidcfg
        echo "network_interface=NONE {hostname=${zone}}" >> /local/${zone}/root/etc/sysidcfg
        echo "terminal=ansi" >> /local/${zone}/root/etc/sysidcfg
        echo "timezone=US/Central" >> /local/${zone}/root/etc/sysidcfg
        zoneadm -z ${zone} boot
done
#

What in the heck was that all about ? OK, one more time - line by line with annotation.

# for zone in webmin mysql web
do
 

A quick interactive loop for the creation of three application zones. The variable ${zone} will be set to the name of the zone we are trying to construct.

echo "create -t default; set zonepath=/local/${zone}" | zonecfg -z ${zone} 

A one liner that creates a new zone configuration based on the already existing default. At this point the only thing we need to change is the zonepath, and it should be set to /local/${zone}.

        zoneadm -z ${zone} clone default 

We recognize this as a zone cloning operation. The zone root is copied and a /reconfigure is created in the new zone root so that sysidtool performs a complete configuration on first boot. If you happen to be running on a recent release of OpenSolaris, you can put your zoneroot on ZFS and the cloning operating will only take a few seconds and very little additional disk space will be required. Those of us on Solaris 10 11/06 will have to wait for the 160MB or so to be copied. Still better than the 9 minutes to go through a complete zone installation.

        echo "name_service=NONE" > /local/${zone}/root/etc/sysidcfg
        echo "nfs4_domain=dynamic" >> /local/${zone}/root/etc/sysidcfg
        echo "security_policy=NONE" >> /local/${zone}/root/etc/sysidcfg
        echo "root_password=xxxxxxxxxxx" >> /local/${zone}/root/etc/sysidcfg
        echo "system_locale=C" >> /local/${zone}/root/etc/sysidcfg
        echo "network_interface=NONE {hostname=${zone}}" >> /local/${zone}/root/etc/sysidcfg
        echo "terminal=ansi" >> /local/${zone}/root/etc/sysidcfg
        echo "timezone=US/Central" >> /local/${zone}/root/etc/sysidcfg 

This step creates a custom sysidcfg file for each zone. Remember to supply your own root password from /etc/shadow in the global zone. This answers all of the sysidtool questions, including the NFSv4 question.

	zoneadm -z {zone} boot 

Boot the zone. If we have done everything correctly, the next interaction will be with console login.

done 

Close the for loop in the interactive script. This process will take a few minutes on Solaris 10 11/06, or if we are being clever with OpenSolaris and ZFS - a few seconds.

Now for the hard part - customizing the individual application zones. Well, it's not all that difficult. And if you do this regularly, you probably have scripts to do most of the work. It's just individual application installation and customization.

Here is what I did for my example zones.

MySQL

The installation instructions for the Solaris 10 MySQL can be found in /etc/sfw/mysql/README.solaris.mysql. There is a typo in the Solaris 10 version of the README. It will cause a lot of grief if you cut and paste without looking at the results. Fortunately it has been corrected in nevada (aka OpenSolaris Community Edition).

Boot the mysql zone and log in as root.

# /usr/sfw/bin/mysql_install_db
# groupadd mysql
# useradd -g mysql mysql
# chgrp -R mysql /var/mysql
# chmod -R 770 /var/mysql       This line is incorrect in the Solaris 10 README - my chmod works better with two arguments
# installf SUNWmysqlr /var/mysql d 770 root mysql
# cp /usr/sfw/share/mysql/my-medium.cnf /var/mysql/my.cnf 

The installation instructions continue by linking the start script into /etc/rc3.d. Since we are big SMF fans in these parts, let's do that instead. Feel free to use my MySQL manifest as it contains a couple of cool features (value and action authorizations - more on that later).

Since the mysql zone doesn't have any networking configured, perform this next step from the global zone. If you already have a suitable manifest, or have stashed mine away somewhere in the global zone you can use that instead.

# cd /local/mysql/root/var/svc/manifest/application
# wget http://blogs.sun.com/bobn/resource/mysql.xml 

It's probably a good idea to make sure that all of this is working properly. Either reboot the mysql zone, run the manifest-import service manually, or run svccfg import on the new manifest. Your choice. What you should see upon completion is

# svcs mysql
STATE          STIME    FMRI
online         14:41:19 svc:/application/mysql:default

# /usr/sfw/bin/mysqladmin status
Uptime: 459  Threads: 1  Questions: 2  Slow queries: 0  Opens: 6  Flush tables: 1  Open tables: 0  Queries per second avg: 0.004

We're done for now. Unless of course you want to go for some extra credit. In that case

1. Set up a web server with PHP support. Apache 1 plus the SFWmphp package from the Solaris Companion will do just fine.
2. Download and unpack phpMyAdmin in the webserver htdocs directory.
3. Create a user with the mysql.operator authorization
4. Create a user with the mysql.administrator authorization

Shut down the mysql zone.

Web

This is about as easy as it gets. Boot the web zone and perform the following steps.

# cp /etc/apache2/httpd.conf-example /etc/apache2/httpd.conf
# svcadm enable apache2

A quick check to make sure all is well.

# svcs apache2
STATE          STIME    FMRI
online         17:17:41 svc:/network/http:apache2


# telnet localhost 80
Trying ::1...
telnet: connect to address ::1: Network is unreachable
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>501 Method Not Implemented</title>

Connection to localhost closed by foreign host.

We're done for now. Shut down the web zone.

Webmin

This one is a little more complicated. We did this one last time in the zone cloning, but it is worth a second look.

Our task here is to replace the Solaris webmin with the latest download from http://webmin.com The technique we are using will allow us to install a custom version of an application into a sparse root zone. Specifically, webmin.com's package installs into /opt/webmin, but /opt is a read-only inherited-pkg-dir. The easiest solution for this would be the creation of a symbolic link in the global zone /opt to point to a location that can be safely written by each non-global zone. In my example that would be /local-pkgs.

In the global zone, create the link in /opt, create the local package directory in the webmin zoneroot, and download the latest webmin package.

# ln -s ../local-pkgs/webmin /opt/webmin
# mkdir -p /local/webmin/root/local-pkgs/webmin
# cd /local/webmin/root/var/tmp
# wget http://prdownloads.sourceforge.net/webadmin/webmin-1.330.pkg.gz
# gunzip webmin-1.330.pkg.gz

Now boot the webmin zone and log in as root.

# zoneadm -z webmin boot
# zlogin webmin 

Remove the Solaris webmin packages (SUNWwebminu SUNWwebminr). The usr package needs to be removed twice - the first pkgrm will leave it as a partially installed package, the second will completely remove it - at least as far as our zone (and future patching) is concerned. Once removed, install the webmin.com version, which should be conveniently located in /var/tmp.

# pkgrm SUNWwebminu SUNWwebminr SUNWwebminu
# pkgadd -d /var/tmp/webmin-1.330.pkg 

We are done with this zone. Shut it down.

Detach

We have just built four zones: an empty zone suitable for future customizations, one with the Solaris webmin replaced by the community edition, one with a working MySQL database, and one with a webserver. The last task to be performed on these zones in their current state is to be detached, another new feature in Solaris 10 11/06. Zone detach will copy the zone configuration into the zoneroot (to be used with a subsequent zone attach) and sets the current zone state to configured. You can even delete the zone configurations as a final cleanup prior to building a flash archive.

# zoneadm -z default detach
# zoneadm -z webmin detach
# zoneadm -z mysql detach
# zoneadm -z web detach
# zonecfg -z default delete -F
# zonecfg -z webmin delete -F
# zonecfg -z mysql delete -F
# zonecfg -z web delete -F

And flash

Unless the person in 18B wants to be a jumpstart server, we will have to simulate jumpstart/flasharchive process. We can do this by booting into an alternate boot environment and then delivering the detached zoneroots by some sort of shared or removable storage - something like a USB memory stick. When we are done with this exercise, our zoneflashes will still be on the memory device, ready for their next use. Since the zones will never be booted, just cloned, the speed of the memory device really isn't important.

We need to prepare the USB memory stick (currently formatted as FAT16). We will use rmformat -l to locate the device, fdisk to put a proper label on it, finally newfs for installing a proper file system. ZFS would be interesting, but it would just get in our way later.

# rmformat -l
Looking for devices...
     1. Logical Node: /dev/rdsk/c2t0d0p0
        Physical Node: /pci@0,0/pci1179,1@1d,7/storage@4/disk@0,0
        Connected Device:          USB DISK 2.0     PMAP
        Device Type: Removable
        Bus: USB
        Size: 984.0 MB
        Label: 
 
 
  
  
        Access permissions: 
  
  
   
   
     2. Logical Node: /dev/rdsk/c1t0d0p0
   
   
        Physical Node: /pci@0,0/pci-ide@1f,1/ide@1/sd@0,0
   
   
        Connected Device: TEAC     DW-224E-A        7.2A
   
   
        Device Type: CD Reader
   
   
        Bus: IDE
   
   
        Size: 
   
   
    
    
        Label: 
    
    
     
     
        Access permissions: 
     
     
      
      

      
      
# 
      
      fdisk /dev/rdsk/c2t0d0p0
      
      

      
      3 (to delete the existing partition)
      
      

      
      1 (to create a new Solaris partition)
      
      

      
      5 (to exit and write the new label)
      
      

      
      
# 
      
      newfs /dev/rdsk/c2t0d0s2
      
      
newfs: construct a new file system /dev/rdsk/c2t0d0s2: (y/n)? 
      
      y
      
      
/dev/rdsk/c2t0d0s2:     2009088 sectors in 981 cylinders of 64 tracks, 32 sectors
      
      
        981.0MB in 62 cyl groups (16 c/g, 16.00MB/g, 7680 i/g)
      
      
super-block backups (for fsck -F ufs -o b=#) at:
      
      
 32, 32832, 65632, 98432, 131232, 164032, 196832, 229632, 262432, 295232,
      
      
 1705632, 1738432, 1771232, 1804032, 1836832, 1869632, 1902432, 1935232,
      
      
 1968032, 2000832
      
      
 
      
      
# 
      
      mkdir /tmp/flash
      
      
# 
      
      mount /dev/dsk/c2t0d0s2 /tmp/flash
      
      
# 
      
      cd /local
      
      
# 
      
      find default webmin web mysql -print | cpio -pdum /tmp/flash
      
      
# 
      
      umount /tmp/flash 
     
     
    
    
   
   
  
  
 
 

We are now done with the original system. At this point we would create a flasharchive (with the detached zoneroots in a convenient place in the archive).

The Prestige

The final act in our magic trick is the delivery. Specifically the transport, reattachment, and subsequent cloning of the zoneflashes on a new system. 18B is now asleep and I really don't want to disturb him, so I'll do this part myself. I'll boot my laptop into another boot environment - built from the same media using the same Live Upgrade method as the boot environment that created the zones.

We begin by mounting the removable media (USB memory stick) that contains the zoneflash. Do take a look around, it is quite likely that our friend volfs has already done this for us. Remember - if we were using a flasharchive to deliver the zoneflash this step would be unnecessary.

# mkdir /flash
# mount /dev/dsk/c2t0d0s2 /flash        (we used rmformat -l to derive the device name)

Now that our zoneflashes have arrived, time to reattach them. The first step is to create zone configurations. If you recall, these were stored in the zoneroot when they were detached. The zonecfg command create -a is used to retrieve the stored configuration information and adapt it to the new system - specifically the new location of the zoneroot. Once configured we use zoneadm attach to reconnect them.

The sequence to reattach our default zone, now called flashdefault, would look something like this.

# zonecfg -z flashdefault
flashdefault: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:flashdefault> create -a /flash/default
zonecfg:flashdefault> commit
zonecfg:flashdefault> exit
# zoneadm -z flashdefault attach 

We'll be a little more clever attaching the other three zones.

# for zone in webmin web mysql
  do
      echo "create -a /flash${zone}" | zonecfg -z flash${zone}
      zoneadm -z flash${zone} attach
  done 

At this point our zoneroots are still on the USB memory device - but don't worry, these zones will never be booted. Their only purpose is to deliver preconfigured zones. We will use zone cloning to create our real application zones.

Which we will now do. It is very convenient to use the flashzone as a template for our new zone in case there were some special attributes like limitpriv that we might want to preserve. We will also need to add items that were not present in the zoneflashes - specifically networking and local file systems. Once we are satisfied with the zone configurations we will clone the zoneflash. If we are only building one of each type of zone we can detach the zoneflash so that other administrators can use it on their systems.

Let's do this for the mysql zone.

# zonecfg -z mysql
mysql: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:mysql> create -t flashmysql
zonecfg:mysql> set zonepath=/zones/mysql
zonecfg:mysql> add net; set physical=e1000g0; set address=192.168.100.102/24; end
zonecfg:mysql> add fs; set dir=/export; set special=/export; set options=[rw,nosuid,nodevices]; set type=lofs; end
zonecfg:mysql> commit
zonecfg:mysql> exit

# zoneadm -z mysql clone flashmysql
Copying /flash/mysql...

# zoneadm -z flashmysql detach

# echo "name_service=NONE" >    /zones/mysql/root/etc/sysidcfg
# echo "nfs4_domain=dynamic" >> /zones/mysql/root/etc/sysidcfg
# echo "security_policy=NONE" >> /zones/mysql/root/etc/sysidcfg
# echo "root_password=xxxxxxxxxxx" >> /zones/mysql/root/etc/sysidcfg
# echo "system_locale=C" >> /zones/mysql/root/etc/sysidcfg
# echo "network_interface=NONE {hostname=mysql}" >> /zones/mysql/root/etc/sysidcfg
# echo "terminal=ansi" >> /zones/mysql/root/etc/sysidcfg
# echo "timezone=US/Central" >> /zones/mysql/root/etc/sysidcfg

And for the finale - boot the newly flashed mysql zone and you should see an enabled and operating mysql service.

# zoneadm -z mysql boot
# zlogin -C mysql
[Connected to zone 'mysql' console]
Hostname: mysql
Creating new rsa public/private host key pair                           
Creating new dsa public/private host key pair
Mar 20 06:15:44 mysql sendmail[1719]: My unqualified host name (mysql) unknown; sleeping for retry
Mar 20 06:15:44 mysql sendmail[1722]: My unqualified host name (mysql) unknown; sleeping for retry

mysql console login: root
Password: 
Last login: Mon Mar 19 17:10:10 on console
Mar 20 06:15:49 mysql login: ROOT LOGIN /dev/console
Sun Microsystems Inc.   SunOS 5.11      snv_57  October 2007
# 
# svcs mysql
STATE          STIME    FMRI
online          6:31:28 svc:/application/mysql:default
# /usr/sfw/bin/mysqladmin status
Uptime: 8  Threads: 1  Questions: 1  Slow queries: 0  Opens: 6  Flush tables: 1  Open tables: 0  Queries per second avg: 0.125

How cool is that ? Not only did we clone the zone, but since the database is in /var, it was cloned as well. Perhaps not practical for every situation, but still pretty cool.

I will leave the flashing of default, web, and webmin as an exercise to the reader. Follow the sequence we used for the mysql zone and you should have four working zones, built from a flash like mechanism that can be delivered via removable media, flasharchive, or shared storage.

Next time we'll take a closer look at MySQL and explore running it as a less privileged user. We'll also look at the action and value authorizations.