RedHat5.4+ASM+oracle11g RAC安装

1、IP地址规划

node1：

public-ip   192.168.2.208     eth0

virtual-ip   192.168.2.210     eth0:1

priv-ip       10.0.1.55              eth1

node2：

public-ip   192.168.2.209     eth0

virtual-ip   192.168.2.211     eth0:1

priv-ip       10.0.1.56              eth1

SCAN-IP:192.168.2.212

2、配置主机IP地址，修改hosts文件，DNS Server配置
-----hosts文件，两个节点一样-----
[root@db1]#vi /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost
::1             localhost6.localdomain6 localhost6
#Public Network
192.168.2.208 db1
192.168.2.209 db2
#Private Interconnect
10.0.1.55 db1-priv
10.0.1.56 db2-priv
#Public Virtual ip
192.168.2.210 db1-vip
192.168.2.211 db2-vip
192.168.2.98 rac-scan.flawless.com   rac-scan


-----网卡，分别配置两个节点-----
[root@db1]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.2.208
NETMASK=255.255.255.0
GATEWAY=192.168.2.254


[root@db1]# vi /etc/sysconfig/network-scripts/ifcfg-eth1
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth1
ONBOOT=yes
IPADDR=10.0.1.55
NETMASK=255.255.255.0


-----DNS服务器配置-----
这里选用192.168.2.98（linux）作为DNS服务器，配置方法另见blog:    http://blog.youkuaiyun.com/flawless_521/article/details/27558529


3、创建数据库用户
[root@db1 etc]# groupadd -g 1000 oinstall
[root@db1 etc]# groupadd -g 1200 asmadmin
[root@db1 etc]# groupadd -g 1201 asmdba
[root@db1 etc]# groupadd -g 1202 asmoper
[root@db1 etc]# useradd -m -u 1100 -g oinstall -G asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash -c "Grid Infrastructure Owner" grid
[root@db1 etc]# id grid
uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper) context=root:system_r:unconfined_t:SystemLow-SystemHigh


[root@db1 etc]# groupadd -g 1300 dba
[root@db1 etc]# groupadd -g 1301 oper
[root@db1 etc]# useradd -m -u 1101 -g oinstall -G dba,oper,asmdba -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle
[root@db1 etc]# id oracle
uid=1101(oracle) gid=1000(oinstall) groups=1000(oinstall),1201(asmdba),1300(dba),1301(oper) context=root:system_r:unconfined_t:SystemLow-SystemHig


[root@db1 etc]passwd oracle
[root@db1 etc]passwd grid


验证nobody用户存在：id nobody，如果输出了用户的信息，则用户存在，如果用户不存在，则使用以下语句创建用户：
[root@db1 etc]#/usr/sbin/useradd nobody


4、在所有节点创建安装目录
[root@db2 ~]# mkdir -p /u01/app/grid
[root@db2 ~]# mkdir -p /u01/app/11.2.0/grid
[root@db2 ~]# chown -R grid:oinstall /u01


[root@db2 ~]# mkdir -p /u01/app/oracle
[root@db2 ~]# chown -R oracle:oinstall /u01/app/oracle
[root@db2 ~]# chmod -R 755 /u01


/u01/app/oracle 是oracle的ORACLE_BASE目录
/u01/app/grid 是grid的ORACLE_BASE目录
/u01/app/11.2.0/grid是grid的ORACLE_HOME目录
说明：grid在安装的时候ORACLE_HOME不能是ORACLE_BASE的子目录，所以要新建立一个目录存放grid的ORACLE_HOME


5、修改oracle和grid的.bash_profile文件
su - grid
vi .bash_profile


export ORACLE_SID=+ASM1
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/11.2.0/grid
JAVA_HOME=/usr/local/java; export JAVA_HOME
ORACLE_PATH=/u01/app/oracle/common/oracle/sql; export ORACLE_PATH
ORACLE_TERM=xterm; export ORACLE_TERM
NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"; export NLS_DATE_FORMAT
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11
PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
THREADS_FLAG=native; export THREADS_FLAG
export TEMP=/tmp
export TMPDIR=/tmp




su - oracle
vi .bash_profile


ORACLE_SID=racdb1; export ORACLE_SID
ORACLE_UNQNAME=racdb; export ORACLE_UNQNAME
JAVA_HOME=/usr/local/java; export JAVA_HOME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export ORACLE_HOME
ORACLE_PATH=/u01/app/common/oracle/sql; export ORACLE_PATH
ORACLE_TERM=xterm; export ORACLE_TERM
NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"; export NLS_DATE_FORMAT
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
ORA_NLS11=$ORACLE_HOME/nls/data; export ORA_NLS11
PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/app/common/oracle/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
THREADS_FLAG=native; export THREADS_FLAG
export TEMP=/tmp
export TMPDIR=/tmp



6、修改系统参数
vi /etc/security/limits.conf
尾部增加以下文件
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536


vi /etc/pam.d/login
尾部增加以下文件
session    required     pam_limits.so


vi /etc/profile
if [ /$USER = "oracle" ] || [ /$USER = "grid" ]; then
    if [ /$SHELL = "/bin/ksh" ]; then
        ulimit -p 16384
        ulimit -n 65536
    else
        ulimit -u 16384 -n 65536
    fi
    umask 022
fi


vi /etc/sysctl.conf


net.core.rmem_default    = 262144
net.core.rmem_max                = 262144
net.core.wmem_default    = 262144
net.core.wmem_max                = 262144
kernel.shmall            = 131072000
kernel.shmmax            = 536870912
kernel.shmmni            = 4096
kernel.sem               = 250 32000 100 128
fs.file-max                   = 65536
net.ipv4.ip_local_port_range = 1024 65500
fs.aio-max-nr=1048576


sysctl -p 使参数生效

7、同步时间
使用集群时间同步服务在集群中提供同步服务，需要卸载网络时间协议 (NTP) 及其配置。
要停用 NTP 服务，必须停止当前的 ntpd 服务，从初始化序列中禁用该服务，并删除 ntp.conf 文件。要在 Oracle Enterprise Linux 上完成这些步骤，以 root 用户身份在两个 Oracle RAC 节点上运行以下命令：
[root@db1 ~]# /sbin/service ntpd stop
[root@db1 ~]# chkconfig ntpd off
[root@db1 ~]# mv /etc/ntp.conf /etc/ntp.conf.original
 
还要删除以下文件：
[root@db1 ~]# rm /var/run/ntpd.pid
此文件保存了 NTP 后台程序的 pid。


8、配置用户等效性
这里注意原来10g安装rac只需要配置oracle用户等效性就可以了，11g的rac还需要配置grid用户的等效性，grid用户配置与oracle用户配置同理，这里只给出oracle用户等效性配置


在db1:
[oracle@db1 ~]$ mkdir ~/.ssh
[oracle@db1 ~]$ chmod 700 ~/.ssh
[oracle@db1 ~]$ ssh-keygen -t rsa
[oracle@db1 ~]$ ssh-keygen -t dsa
 
在db2：
[oracle@db2 ~]$ mkdir ~/.ssh
[oracle@db2 ~]$ chmod 700 ~/.ssh
[oracle@db2 ~]$ ssh-keygen -t rsa
[oracle@db2 ~]$ ssh-keygen -t dsa


切换回db1，接着执行：
[oracle@db1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[oracle@db1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
 
提示：下列命令会提示你输入bjgtj2 的oracle 密码，按照提示输入即可，如果失败可重新尝试执行命令。
db1 节点：
[oracle@db1 ~]$ scp ~/.ssh/authorized_keys db2:~/.ssh/authorized_keys
 
db2节点：
[oracle@db2 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[oracle@db2 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[oracle@db2 ~]$  scp ~/.ssh/authorized_keys db1:~/.ssh/authorized_keys
 
确保2个node都有相互的结点信息。两机相互执行。
[oracle@db1 ~]$ ssh db1 date
[oracle@db1~]$ ssh db2 date
[oracle@db1 ~]$ ssh db1-priv date
[oracle@db1 ~]$ ssh db2-priv date
 
切换至db2 执行
[oracle@db2 ~]$ ssh db1 date
[oracle@db2~]$ ssh db2 date
[oracle@db2 ~]$ ssh db1-priv date
[oracle@db2 ~]$ ssh db2-priv date


9、添加共享磁盘，安装ASM并创建ASM磁盘
这里用vmware新添加了一块50GB的共享磁盘，将磁盘分区成sb1 （1GB存放ocr与votedisk）、sdb2（30GB存放数据文件） 、sdb3（19GB存放FRA数据），以下只给出分区sdb1的过程，大小为1024m，其他的分区同下：
三个磁盘组名字：OCRVOTEDG、DATAFILEDG、FRADG


[root@db1 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 7832.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)


Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)


Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-7832, default 1): 
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-7832, default 7832): +1024M  


Command (m for help): p


Disk /dev/sdb: 64.4 GB, 64424509440 bytes
255 heads, 63 sectors/track, 7832 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes


   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1          62      497983+  83  Linux


Command (m for help): w
The partition table has been altered!


Calling ioctl() to re-read partition table.
Syncing disks.


---安装ASM包（两个节点都执行）
uname -a
Linux db2 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

安装以下三个包，按顺序安装：
rpm -ivh oracleasm-support-2.1.8-1.el5.x86_64.rpm
rpm -ivh oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm
rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpm

安装完后检查安装是否成功：rpm -qa|grep asm


---配置ASM
[root@db1]# oracleasm configure -i（两个节点都执行）
Configuring the Oracle ASM library driver.


This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.


Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done


---加载oracleasm内核模块
[root@db1 ASM_RHL-2.6.8-164.el5]# oracleasm init（两个节点都执行）
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Mounting ASMlib driver filesystem: /dev/oracleasm


---创建asm磁盘
[root@db1]# oracleasm createdisk OCRVOTEDG /dev/sdb1
Writing disk header: done
Instantiating disk: done

[root@db1]# oracleasm createdisk DATAFILEDG /dev/sdb2
Writing disk header: done
Instantiating disk: done

[root@db1]# oracleasm createdisk FRADG /dev/sdb3
Writing disk header: done
Instantiating disk: done




在另一个节点执行扫描
[root@db2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "OCRVOTEDG"
Instantiating disk "DATAFILEDG"
Instantiating disk "FRADG"


[root@db2 ~]# oracleasm listdisks
DATAFILEDG
FRADG
OCRVOTEDG


10、安装CLuster软件
11g中用grid用户管理cluster，故用grid用户安装cluster软件，安装grid软件之前可以先检查安装环境，进入到grid的解压目录里，找到runcluvfy.sh，运行以下命令：
./runcluvfy.sh stage -pre crsinst -n db1,db2 -fixup -verbose    如果检查有failed的则修复，直到没有failed的进行下一步安装，依次选择的顺序如下：


install and  Configure Grid infrastructure for a Cluster
Advanced Installation
语言默认用English
将Configure GNS的勾去掉，输入Cluster Name和SCAN Name，这两处输入的名字可以按照你之前配置的/etc/hosts文件最后添加的内容 192.168.2.212 rac-scan.flawless.com   rac-scan
Cluster Name  rac-scan
SCAN Name   rac-scan.flawless.com
点击add增加节点2的信息，然后点击SSH Connectivity，输入密码后点击Setup，验证成功则进入下一步
网卡eth0和eth1这块直接下一步
存储方式选择ASM
输入Disk Group Name  : OCRVOTEDG，Redundancy选择External，这里先给ocr和votedisk选择ORCL：OCRVOTEDG然后下一步
Use same password for  these account，输入密码
Do not use intellgent Platform Management Interface(IPM)
下面一直下一步就可以了，然后只需要等待，最后会弹出需要执行的脚本，分别用root用户在两个节点执行脚本就可以了，如果脚本执行中有报错，那么需要解决掉，我这里也遇到了一次错误，但是我直接把目录删了，修改了一些东西后重新运行了runInstall，安装完成后，分别在两个节点切换到grid用户运行crs_stat -t，验证安装情况，同时运行命令crsctl check ctss检查时间是否同步，正常情况下应输出以下内容：
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 0


11、安装oracle软件
切换到oracle用户下，安装oracle软件，只需要在第一个节点安装，注意一个地方将两个节点都选上select all，安装完成后用root用户按照提示，分别在两个节点执行脚本


12、创建其他ASM磁盘组
安装grid的时候就创建了ASM实例，但是只创建了一个CRS组来安装ocr和votedisk，现在继续创建DATAFIELDG和FRADG
用grid用户直接运行asmca，直接点击create创建这两个磁盘组，创建完之后直接点击Exit退出窗口即可。


13、创建实例
直接在第一个节点运行dbca，与安装单实例数据库基本相同。