ASM Instance failed with ORA-27300,ORA-27302 and ORA-27303(_disable_interface_checking)

I was stucked one time and my both the DB Nodes are not working due to some operating system or hardware or network issue

after  rebooted both the DB server Nodes we came to know that

ASM is not comes up on DB Node1

-bash-3.00$ crs_stat -t

Name           Type           Target    State     Host
————————————————————
ora….SM1.asm application    ONLINE    OFFLINE
ora….TS.lsnr application    ONLINE    ONLINE    nr-d…-orcl
ora….orcl.gsd application    ONLINE    ONLINE    nr-d…-orcl
ora….orcl.ons application    ONLINE    ONLINE    nr-d…-orcl
ora….orcl.vip application    ONLINE    ONLINE    nr-d…-orcl
ora….SM2.asm application    ONLINE    ONLINE    nr-d…-orcl
ora….TS.lsnr application    ONLINE    ONLINE    nr-d…-orcl
ora….orcl.gsd application    ONLINE    ONLINE    nr-d…-orcl
ora….orcl.ons application    ONLINE    ONLINE    nr-d…-orcl
ora….orcl.vip application    ONLINE    ONLINE    nr-d…-orcl
ora.orcldel.db application    ONLINE    ONLINE    nr-d…-orcl
ora….l1.inst application    ONLINE    OFFLINE
ora….l2.inst application    ONLINE    ONLINE    nr-d…-orcl
ora….bdel.db application    ONLINE    ONLINE    nr-d…-orcl
ora….l1.inst application    ONLINE    OFFLINE
ora….l2.inst application    ONLINE    ONLINE    nr-d…-orcl

my-live-server$ sqlplus / as sysasm

SQL*Plus: Release 10.2.0.4.0 – Production on Wed Apr 24 13:46:57 2013

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:check if cable failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpcini1
ORA-27303: additional information: requested interface e1000g2 interface not running set _disable_interface_checking = TRUE to disable this check for single instance cluster. Check output from if
SQL> quit
Disconnected

SQL> !oerr ora 27300
27300, 00000, “OS system dependent operation:%s failed with status: %s”
// *Cause:  OS system call error
// *Action: contact Oracle Support

SQL> !oerr ora 27301
27301, 00000, “OS failure message: %s”
// *Cause:  OS system call error
// *Action: contact Oracle Support
//

SQL> !oerr ora 27302
27302, 00000, “failure occurred at: %s”
// *Cause:  OS system call error
// *Action: contact Oracle Support
//

SQL> !oerr ora 27303
27303, 00000, “additional information: %s”
// *Cause:  OS system call error
// *Action: contact Oracle Support

Cause

1) The interface: e1000g2 is not running:

e1000g2 Link encap:Ethernet HWaddr 00:14:22:10:88:15
inet addr:192.168.1.9 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::214:22ff:fe10:8815/64 Scope:Link
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Base address:0xdcc0 Memory:df8e0000-df900000

2) You should see the next message:

UP BROADCAST RUNNING MULTICAST MTU:1500

3) For example the interface: eth0 is up & running (OK):

e1000g2:1 Link encap:Ethernet HWaddr 00:14:22:10:88:14
inet addr:216.52.157.35 Bcast:216.52.157.127 Mask:255.255.255.128
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Base address:0xecc0 Memory:dfae0000-dfb00000

or

check os logs e.g. dmesg, /var/log/message

Solution

After doing R&D in Oracle Meta-Link Troubleshooting ORA-27300 ORA-27301 ORA-27302 errors and Google

We came to know that It may be related to swap space or oracle rac interconnect physical network connectivity

1) Please ask your system administrator to check the interface eth1.

2) The private NIC (e1000g2) should be connected to a network switch.

As workaround, We had rebooted the DB Server Node-1 and Problem got resolved and every things comes up 🙂

Reference

Troubleshooting ORA-27300 ORA-27301 ORA-27302 errors [ID 579365.1]

Cannot Startup ASM Rac Instance due to ORA-27303: requested interface eth1 is not RUNNING [ID 340524.1]

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s