Oracle ASM Advanced Tranning 1 —— Configuration of Devices for ASM

Part 1 : Configure devices for ASM using file system files and loop devices

Check the amount of space available on file system

[root@vasm ~]# df -TH
Filesystem    Type     Size   Used  Avail Use% Mounted on
/dev/sda3     ext3      46G   6.3G    38G  15% /
/dev/sda1     ext3     200M    23M   167M  12% /boot
tmpfs        tmpfs     2.1G      0   2.1G   0% /dev/shm
/dev/sr0   iso9660     3.9G   3.9G      0 100% /media/disk

Create a directory to hold the files to be used as ASM devices

[oracle@vasm ~]$ mkdir -p /u01/app/asm_loop_devices
[oracle@vasm ~]$ ls -l /u01/app
total 8
drwxr-xr-x 2 oracle oinstall 4096 Feb 20 09:06 asm_loop_devices/
drwxr-xr-x 5 oracle oinstall 4096 Feb 19 15:59 oracle/

Create the files to be used as Asm devices

[oracle@vasm ~]$ cd /u01/app/asm_loop_devices
[oracle@vasm asm_loop_devices]$ dd if=/dev/zero of=/u01/app/asm_loop_devices/fs_dsk1 bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 0.79403 seconds, 660 MB/s
[oracle@vasm asm_loop_devices]$ dd if=/dev/zero of=/u01/app/asm_loop_devices/fs_dsk2 bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 0.864626 seconds, 606 MB/s
[oracle@vasm asm_loop_devices]$ dd if=/dev/zero of=/u01/app/asm_loop_devices/fs_dsk3 bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 7.32695 seconds, 71.6 MB/s
[oracle@vasm asm_loop_devices]$ dd if=/dev/zero of=/u01/app/asm_loop_devices/fs_dsk4 bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 3.04805 seconds, 172 MB/s
[oracle@vasm asm_loop_devices]$ ls -ltrh
total 2.0G
-rw-r--r-- 1 oracle oinstall 500M Feb 20 09:07 fs_dsk1
-rw-r--r-- 1 oracle oinstall 500M Feb 20 09:07 fs_dsk2
-rw-r--r-- 1 oracle oinstall 500M Feb 20 09:08 fs_dsk3
-rw-r--r-- 1 oracle oinstall 500M Feb 20 09:08 fs_dsk4
[oracle@vasm asm_loop_devices]$ pwd
/u01/app/asm_loop_devices

Connect to root and check if there are loop devices

[oracle@vasm asm_loop_devices]$ su - root
Password: 
[root@vasm ~]# raw -qa
[root@vasm ~]# 

As root create the loop devices

[root@vasm ~]# losetup /dev/loop1 /u01/app/asm_loop_devices/fs_dsk1 
[root@vasm ~]# losetup /dev/loop2 /u01/app/asm_loop_devices/fs_dsk2 
[root@vasm ~]# losetup /dev/loop3 /u01/app/asm_loop_devices/fs_dsk3
[root@vasm ~]# losetup /dev/loop4 /u01/app/asm_loop_devices/fs_dsk4

Bind the loop devices to raw devices

[root@vasm ~]# raw /dev/raw/raw1 /dev/loop1 
/dev/raw/raw1:	bound to major 7, minor 1
[root@vasm ~]# raw /dev/raw/raw2 /dev/loop2
/dev/raw/raw2:	bound to major 7, minor 2
[root@vasm ~]# raw /dev/raw/raw3 /dev/loop3
/dev/raw/raw3:	bound to major 7, minor 3
[root@vasm ~]# raw /dev/raw/raw4 /dev/loop4
/dev/raw/raw4:	bound to major 7, minor 4

Check the raw devices binding

[root@vasm ~]# raw -qa
/dev/raw/raw1:	bound to major 7, minor 1
/dev/raw/raw2:	bound to major 7, minor 2
/dev/raw/raw3:	bound to major 7, minor 3
/dev/raw/raw4:	bound to major 7, minor 4
[root@vasm ~]# ls -ltr /dev/raw/raw*
crw------- 1 root root 162, 1 Feb 20 09:11 /dev/raw/raw1
crw------- 1 root root 162, 2 Feb 20 09:11 /dev/raw/raw2
crw------- 1 root root 162, 3 Feb 20 09:11 /dev/raw/raw3
crw------- 1 root root 162, 4 Feb 20 09:11 /dev/raw/raw4

Setup raw devices permissions and ownership

[root@vasm ~]# chmod 660 /dev/raw/raw[1-4]
[root@vasm ~]# chown oracle:oinstall /dev/raw/raw[1-4]
[root@vasm ~]# ls -ltr /dev/raw/raw[1-4]
crw-rw---- 1 oracle oinstall 162, 1 Feb 20 09:11 /dev/raw/raw1
crw-rw---- 1 oracle oinstall 162, 2 Feb 20 09:11 /dev/raw/raw2
crw-rw---- 1 oracle oinstall 162, 3 Feb 20 09:11 /dev/raw/raw3
crw-rw---- 1 oracle oinstall 162, 4 Feb 20 09:11 /dev/raw/raw4

Note:That after reboot the ownership and permisions will change back to root and -rw-r--r--, they can be setup on file "/etc/rc.local" to be executed at startup.

To delete a loop device use the following sintax:losetup –d /dev/loopX

[root@vasm ~]# echo "chmod 660 /dev/raw/raw[1-4]" >> /etc/rc.local
[root@vasm ~]# echo "chown oracle:oinstall /dev/raw/raw[1-4]" >> /etc/rc.local
[root@vasm ~]# cat /etc/rc.local 
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
chmod 660 /dev/raw/raw[1-4]
chown oracle:oinstall /dev/raw/raw[1-4]

Part 2 : Configure devices for ASM using LUN’s

Identify available LUN’s and LUN’s sizes

[root@vasm ~]# fdisk -l | grep -v /dev/sda |grep ^Disk
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdd doesn't contain a valid partition table
Disk /dev/sde doesn't contain a valid partition table
Disk /dev/sdf doesn't contain a valid partition table
Disk /dev/sdg doesn't contain a valid partition table
Disk /dev/sdh doesn't contain a valid partition table
Disk /dev/sdi doesn't contain a valid partition table
Disk /dev/sdj doesn't contain a valid partition table
Disk /dev/sdk doesn't contain a valid partition table
Disk /dev/sdl doesn't contain a valid partition table
Disk /dev/sdm doesn't contain a valid partition table
Disk /dev/sdn doesn't contain a valid partition table
Disk /dev/sdo doesn't contain a valid partition table
Disk /dev/sdp doesn't contain a valid partition table
Disk /dev/sdq doesn't contain a valid partition table
Disk /dev/sdb: 5368 MB, 5368709120 bytes
Disk /dev/sdc: 5368 MB, 5368709120 bytes
Disk /dev/sdd: 5368 MB, 5368709120 bytes
Disk /dev/sde: 5368 MB, 5368709120 bytes
Disk /dev/sdf: 5368 MB, 5368709120 bytes
Disk /dev/sdg: 5368 MB, 5368709120 bytes
Disk /dev/sdh: 5368 MB, 5368709120 bytes
Disk /dev/sdi: 5368 MB, 5368709120 bytes
Disk /dev/sdj: 5368 MB, 5368709120 bytes
Disk /dev/sdk: 5368 MB, 5368709120 bytes
Disk /dev/sdl: 5368 MB, 5368709120 bytes
Disk /dev/sdm: 5368 MB, 5368709120 bytes
Disk /dev/sdn: 5368 MB, 5368709120 bytes
Disk /dev/sdo: 5368 MB, 5368709120 bytes
Disk /dev/sdp: 5368 MB, 5368709120 bytes
Disk /dev/sdq: 5368 MB, 5368709120 bytes

Create a single partition per LUN using fdisk

[root@vasm ~]# fdisk /dev/sdb 
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/sdb: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-652, default 1): 
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-652, default 652): 
Using default value 652

Command (m for help): 
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Create single partitions on evary other LUN

[root@vasm ~]# fdisk -l | grep /dev/sd | grep -v sda
Disk /dev/sdb: 5368 MB, 5368709120 bytes
/dev/sdb1               1         652     5237158+  83  Linux
Disk /dev/sdc: 5368 MB, 5368709120 bytes
/dev/sdc1               1        1449     5242848+  83  Linux
Disk /dev/sdd: 5368 MB, 5368709120 bytes
/dev/sdd1               1         652     5237158+  83  Linux
Disk /dev/sde: 5368 MB, 5368709120 bytes
/dev/sde1               1         652     5237158+  83  Linux
Disk /dev/sdf: 5368 MB, 5368709120 bytes
/dev/sdf1               1         652     5237158+  83  Linux
Disk /dev/sdg: 5368 MB, 5368709120 bytes
/dev/sdg1               1         652     5237158+  83  Linux
Disk /dev/sdh: 5368 MB, 5368709120 bytes
/dev/sdh1               1         652     5237158+  83  Linux
Disk /dev/sdi: 5368 MB, 5368709120 bytes
/dev/sdi1               1         652     5237158+  83  Linux
Disk /dev/sdj: 5368 MB, 5368709120 bytes
/dev/sdj1               1         652     5237158+  83  Linux
Disk /dev/sdk: 5368 MB, 5368709120 bytes
/dev/sdk1               1         652     5237158+  83  Linux
Disk /dev/sdl: 5368 MB, 5368709120 bytes
/dev/sdl1               1         652     5237158+  83  Linux
Disk /dev/sdm: 5368 MB, 5368709120 bytes
/dev/sdm1               1         652     5237158+  83  Linux
Disk /dev/sdn: 5368 MB, 5368709120 bytes
/dev/sdn1               1         652     5237158+  83  Linux
Disk /dev/sdo: 5368 MB, 5368709120 bytes
/dev/sdo1               1         652     5237158+  83  Linux
Disk /dev/sdp: 5368 MB, 5368709120 bytes
/dev/sdp1               1         652     5237158+  83  Linux
Disk /dev/sdq: 5368 MB, 5368709120 bytes
/dev/sdq1               1         652     5237158+  83  Linux
[root@vasm ~]# ls /dev/sd?1 | grep -v sda | xargs chown oracle:oinstall
[root@vasm ~]# ls /dev/sd?1 | grep -v sda | xargs chmod 660
[root@vasm ~]# ls -l /dev/sd?1 | grep -v sda
brw-rw---- 1 oracle oinstall  8,  17 Feb 20 09:30 /dev/sdb1
brw-rw---- 1 oracle oinstall  8,  33 Feb 20 09:31 /dev/sdc1
brw-rw---- 1 oracle oinstall  8,  49 Feb 20 09:31 /dev/sdd1
brw-rw---- 1 oracle oinstall  8,  65 Feb 20 09:32 /dev/sde1
brw-rw---- 1 oracle oinstall  8,  81 Feb 20 09:32 /dev/sdf1
brw-rw---- 1 oracle oinstall  8,  97 Feb 20 09:32 /dev/sdg1
brw-rw---- 1 oracle oinstall  8, 113 Feb 20 09:32 /dev/sdh1
brw-rw---- 1 oracle oinstall  8, 129 Feb 20 09:32 /dev/sdi1
brw-rw---- 1 oracle oinstall  8, 145 Feb 20 09:32 /dev/sdj1
brw-rw---- 1 oracle oinstall  8, 161 Feb 20 09:32 /dev/sdk1
brw-rw---- 1 oracle oinstall  8, 177 Feb 20 09:33 /dev/sdl1
brw-rw---- 1 oracle oinstall  8, 193 Feb 20 09:33 /dev/sdm1
brw-rw---- 1 oracle oinstall  8, 209 Feb 20 09:33 /dev/sdn1
brw-rw---- 1 oracle oinstall  8, 225 Feb 20 09:33 /dev/sdo1
brw-rw---- 1 oracle oinstall  8, 241 Feb 20 09:38 /dev/sdp1
brw-rw---- 1 oracle oinstall 65,   1 Feb 20 09:38 /dev/sdq1

Part 3 : Configure File System Files to be Used as ASM Disks

The use of file system files as ASM disks is implemented using the parameter_asm_allow_only_raw_disks = FALSE
Details can be found on Metalink Note: 293095.1

Create a directory to store the files that will be used as ASM DISKS

[oracle@vasm ~]$ mkdir -p /u01/app/asm_filesystem_devices

Create the files to be used as ASM DISKS

[oracle@vasm ~]$ cd /u01/app/asm_filesystem_devices
[oracle@vasm asm_filesystem_devices]$ dd if=/dev/zero of=/u01/app/asm_filesystem_devices/asm_filesystem_dsk1 bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 1.33335 seconds, 393 MB/s
[oracle@vasm asm_filesystem_devices]$ dd if=/dev/zero of=/u01/app/asm_filesystem_devices/asm_filesystem_dsk2 bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 5.16762 seconds, 101 MB/s
[oracle@vasm asm_filesystem_devices]$ dd if=/dev/zero of=/u01/app/asm_filesystem_devices/asm_filesystem_dsk3 bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 4.75629 seconds, 110 MB/s
[oracle@vasm asm_filesystem_devices]$ dd if=/dev/zero of=/u01/app/asm_filesystem_devices/asm_filesystem_dsk4 bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 1.85224 seconds, 283 MB/s

Change Ownership and Permissions of the ASM Files

[oracle@vasm ~]$ chmod 660 /u01/app/asm_filesystem_devices/*
[oracle@vasm ~]$ ls -ltrh /u01/app/asm_filesystem_devices/
total 2.0G
-rw-rw---- 1 oracle oinstall 500M Feb 20 09:50 asm_filesystem_dsk1
-rw-rw---- 1 oracle oinstall 500M Feb 20 09:51 asm_filesystem_dsk2
-rw-rw---- 1 oracle oinstall 500M Feb 20 09:51 asm_filesystem_dsk3
-rw-rw---- 1 oracle oinstall 500M Feb 20 09:51 asm_filesystem_dsk4