Automatic Stop of Database (dbshut) not working in OL 7 with systemd (文档 ID 2229679.1)SSSq

Automatic Stop of Database (dbshut) not working in OL 7 with systemd (文档 ID 2229679.1)

APPLIES TO:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Information in this document applies to any platform.
Linux x86-64

SYMPTOMS

On : 12.1.0.2 version, RDBMS

When attempting to stop DB during server shutdown the automatic shutdown script getting below error

cat /u01/app/oracle/product/12.1.0.2/dbhome_1/shutdown.log
Processing Database instance "XXXX1": log file /u01/app/oracle/product/12.1.0.2/dbhome_1/shutdown.log
Info: Database instance "XXXX" already down (PMON process not there).

Which indicate that PMON is forcefully being closed before the script is called.

This is the configuration used

cat  dbora.service

[Unit]
Description=The Oracle Database Service
After=network.target
[Service]
Type=forking
# Type=oneshot
RemainAfterExit=yes
KillMode=none
# Set this to something larger if it has an impact
TimeoutStopSec=0
ExecStart=/u01/app/oracle/product/12.1.0.2/dbhome_1/bin/dbora start
ExecStop=/u01/app/oracle/product/12.1.0.2/dbhome_1/bin/dbora stop
[Install]
# Puts wants directive for the other units in the relationship
WantedBy=default.target

systemctl enable dbora.service

systemctl daemon-reload

systemctl start dbora.service

 

CHANGES

 

CAUSE

Issue with the dbora script which is doing su to oracle owner process and systemd is not able to keep track of that process.
 

This is the sample dbora script

 

ORA_HOME=/u01/app/oracle/product/12.1.0.2/dbhome_1
ORA_OWNER=oracle

case "$1" in
'start')
# Start the Oracle databases:
# The following command assumes that the oracle login
# will not prompt the user for any values
# Remove "&" if you don't want startup as a background process.
su - $ORA_OWNER -c "$ORA_HOME/bin/dbstart $ORA_HOME"
touch /var/lock/subsys/dbora
;;

'stop')
# Stop the Oracle databases:
# The following command assumes that the oracle login
# will not prompt the user for any values
su - $ORA_OWNER -c "$ORA_HOME/bin/dbshut $ORA_HOME"
rm -f /var/lock/subsys/dbora
;;
esac

With this arrangement the systemd is losing control over the service and not stopping it in time. 
 

SOLUTION

The better option is to use dbstart and dbstop script directly under the directive with user and owner details

 

[Unit]
Description=The Oracle Database Service
After=network.target

[Service]
Type=forking
RemainAfterExit=yes
KillMode=none
TimeoutStopSec=5min

User=oracle                                      << Replace Oracle software owner details here 
Group=oinstall
ExecStart=$ORACLE_HOME/bin/dbstart $ORACLE_HOME &             ===> Please use absolute path here instead of the $ORACLE_HOME variable
ExecStop=$ORACLE_HOME/bin/dbshut $ORACLE_HOME                 ===> Please use absolute path here instead of the $ORACLE_HOME variable
Restart=no

[Install]
# Puts wants directive for the other units in the relationship
WantedBy=default.target

Then systemd has control about the services

# systemctl status dbora.service
● dbora.service - The Oracle Database Service
Loaded: loaded (/usr/lib/systemd/system/dbora.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2017-02-01 12:19:53 GMT; 22s ago
Process: 3905 ExecStop=/u01/app/oracle/product/12.1.0.2/dbhome_1/bin/dbshut /u01/app/oracle/product/12.1.0.2/dbhome_1 (code=exited, status=0/SUCCESS)
Process: 4043 ExecStart=/u01/app/oracle/product/12.1.0.2/dbhome_1/bin/dbstart /u01/app/oracle/product/12.1.0.2/dbhome_1 & (code=exited, status=0/SUCCESS)
CGroup: /system.slice/dbora.service
├─4051 /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/tnslsnr LISTENER -inherit
├─4143 ora_pmon_XXXXX

<snip>
└─4477 ora_q003_XXXX

 Additional Following directives need to be considered

1) If someone accidentally runs dbshut/dbstart as root , next valid attempt will consistently fail with:

Apr 25 18:04:57 instance-20190425-12corig dbstart: touch: cannot touch ?/u01/app/oracle/product/12.2.0/dbhome_1/startup.log?: Permission denied
Apr 25 18:04:57 instance-20190425-12corig dbstart: chmod: changing permissions of ?/u01/app/oracle/product/12.2.0/dbhome_1/startup.log?: Operation not permitted
Apr 25 18:04:57 instance-20190425-12corig dbstart: Processing Database instance "ORCL": log file /u01/app/oracle/product/12.2.0/dbhome_1/startup.log

to avoid, add these:

ExecStartPre=/bin/rm -rf  /u01/app/oracle/product/12.2.0/dbhome_1/listener.log
ExecStartPre=/bin/rm -rf  /u01/app/oracle/product/12.2.0/dbhome_1/startup.log
ExecStop=/bin/rm -rf  /u01/app/oracle/product/12.2.0/dbhome_1/shutdown.log

2) To avoid such errors during a real reboot:

  Process: 4529 ExecStart=/u01/app/oracle/product/12.2.0/dbhome_1/bin/dbstart /u01/app/oracle/product/12.2.0/dbhome_1 (code=exited, status=203/EXEC)
  Apr 25 17:20:18 instance-20190425-12corig bash[4525]: /u01/app/oracle/product/12.2.0/dbhome_1: /u01/app/oracle/product/12.2.0/dbhome_1/bin/dbstart: No such file or directory

ADD 'local-fs.target' or 'remote-fs.target' (or both)

After=syslog.target network.target local-fs.target remote-fs.target

otherwise the FILESystem where $ORACLE_HOME lives is not ready, and dbstart cannot be found.

3) Add limit* settings as by design systemd does not honor /etc/security/limits.conf settings( Below are the optimal settings as per Database is concern, if you are using different setting at your system level please use those values instead) 

LimitNOFILE=65536
LimitNPROC=16384
LimitSTACK=32M
LimitMEMLOCK=infinity
LimitCORE=infinity


Example with all these together:

[Unit]
Description=Oracle Database Start/Stop Service
After=syslog.target network.target local-fs.target remote-fs.target

[Service]
# systemd, by design does not honor PAM limits
# See: https://bugzilla.redhat.com/show_bug.cgi?id=754285
LimitNOFILE=65536
LimitNPROC=16384
LimitSTACK=32M
LimitMEMLOCK=infinity
LimitCORE=infinity

Type=simple
User=oracle
Group=oinstall
Restart=no
ExecStartPre=/bin/rm -rf  /u01/app/oracle/product/12.2.0/dbhome_1/listener.log
ExecStartPre=/bin/rm -rf  /u01/app/oracle/product/12.2.0/dbhome_1/startup.log
ExecStart=/u01/app/oracle/product/12.2.0/dbhome_1/bin/dbstart /u01/app/oracle/product/12.2.0/dbhome_1
RemainAfterExit=yes
ExecStop=/bin/rm -rf  /u01/app/oracle/product/12.2.0/dbhome_1/shutdown.log
ExecStop=/u01/app/oracle/product/12.2.0/dbhome_1/bin/dbshut /u01/app/oracle/product/12.2.0/dbhome_1
TimeoutStopSec=5min

[Install]
WantedBy=multi-user.target