OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [Permission denied] [13]

There was activity @Storage Level by System administrator last night.

today morning 6:00 PM, I got call from support engineer that DB Node1 services not working after storage level activity
DB S/W 10G 10.2.0.4.0 + ASM (2 Node RAC)
Oracle Sun Solaris 10 64bit Platform

DB Node2 is up and running
DB Node1 cluster stack not started automatically after rebooted server

tail -100f /var/adm/messages
Jul 20 07:43:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1284.
Jul 20 07:43:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1269.
Jul 20 07:43:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1268.
Jul 20 07:43:57 node1 pseudo: [ID 129642 kern.info] pseudo-device: bmc0
Jul 20 07:43:57 node1 genunix: [ID 936769 kern.info] bmc0 is /pseudo/bmc@0
Jul 20 07:44:08 node1 scsi: [ID 799468 kern.info] ssd22 at scsi_vhci0: name g600a0b800067e686000003b54c2841d4, bus address g600a0b800067e686000003b54c2841d4
Jul 20 07:44:08 node1 genunix: [ID 936769 kern.info] ssd22 is /scsi_vhci/ssd@g600a0b800067e686000003b54c2841d4
Jul 20 07:44:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1284.
Jul 20 07:44:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1269.
Jul 20 07:44:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1268.
Jul 20 07:45:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1284.
Jul 20 07:45:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1269.
Jul 20 07:45:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1268.
Jul 20 07:46:19 node1 scsi: [ID 193665 kern.info] sd1 at mpt0: target 1 lun 0
Jul 20 07:46:19 node1 genunix: [ID 936769 kern.info] sd1 is /pci@780/pci@0/pci@9/scsi@0/sd@1,0
Jul 20 07:46:19 node1 pseudo: [ID 129642 kern.info] pseudo-device: rsm0
Jul 20 07:46:19 node1 genunix: [ID 936769 kern.info] rsm0 is /pseudo/rsm@0
Jul 20 07:46:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1284.
Jul 20 07:46:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1269.
Jul 20 07:46:51 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1268.
Jul 20 07:47:39 node1 xntpd[242]: [ID 774427 daemon.notice] time reset (step) 1.544583 s
Jul 20 07:47:53 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1269.
Jul 20 07:47:53 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1284.
Jul 20 07:47:53 node1 root: [ID 702911 user.error] Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.1268

$cat /tmp/crsctl.1268

OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [Permission denied] [13]

./crsctl query css votedisk
./ocrcheck

compare the output of vote and ocr disks using above commands

ls -ltr /dev/rdsk/c1
configure permission

chown -h orarac:oinstall c4t600A0B800067E686000003B74C2843F3d0s0
chown -R 664 c4t600A0B800067E686000003B74C2843F3d0s0

select * from v$asm_disk;
SELECT * FROM v$asm_diskgroup;
DB2
=====

GROUP_NUMBER DISK_NUMBER COMPOUND_INDEX INCARNATION MOUNT_S HEADER_STATU MODE_ST STATE    REDUNDA LIBRARY    TOTAL_MB    FREE_MB NAME         FAILGROUP   LABEL   PATH                                              UDID PRODUCT  CREATE_DA MOUNT_DAT REPAIR_TIMER      READS     WRITES  READ_ERRS WRITE_ERRS  READ_TIME WRITE_TIME BYTES_READ BYTES_WRITTEN
------------ ----------- -------------- ----------- ------- ------------ ------- -------- ------- ------------------- ---------- ------------ ----------  ------- ------------------------------------------------------ -------- --------- --------- ------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- -------------
           0           0              0           0 CLOSED  FOREIGN      ONLINE  NORMAL   UNKNOWN System         3048          0                                  /dev/rdsk/c4t600A0B800067E686000003B74C2843F3d0s0                                              0
           1           0       16777216  4042313616 CACHED  MEMBER       ONLINE  NORMAL   UNKNOWN System       204752     187619 DATA_0000    DATA_0000           /dev/rdsk/c4t600A0B800067E686000003C54C297E76d0s0               29-JUN-10 22-JAN-14            0    3109195     649757          0          0   71598.79   74535.59 5.2792E+10    5044413440
           1           1       16777217  4042313617 CACHED  MEMBER       ONLINE  NORMAL   UNKNOWN System       204752     187574 DATA_0001    DATA_0001           /dev/rdsk/c4t600A0B800067E686000003C74C297EBDd0s0               29-JUN-10 22-JAN-14            0    4662501     911380          0          0   85038.87   83353.91 7.5551E+10    9475180032
           1           2       16777218  4042313618 CACHED  MEMBER       ONLINE  NORMAL   UNKNOWN System       204752     187622 DATA_0002    DATA_0002           /dev/rdsk/c4t600A0B800067E686000003C94C297EFFd0s0               29-JUN-10 22-JAN-14            0    4974687     914113          0          0    75236.7   66627.38 7.9971E+10    9682139648
           2           0       33554432  4042313619 CACHED  MEMBER       ONLINE  NORMAL   UNKNOWN System       298948     224182 FLASH_0000   FLASH_0000          /dev/rdsk/c4t600A0B800067E686000003CB4C297F64d0s0               29-JUN-10 22-JAN-14            0      28049      55004          0          0   12685.85    1070.03 9655395840     316669952


GROUP_NUMBER“|”NAME      “|”SECTOR_SIZE“|”BLOCK_SIZE“|”ALLOCATION_UNIT_SIZE“|”STATE     “|”TYPE    “|”TOTAL_MB“|”FREE_MB“|”HOT_USED_MB“|”COLD_USED_MB“|”REQUIRED_MIRROR_FREE_MB“|”USABLE_FILE_MB“|”OFFLINE_DISKS“|”COMPATIBILITY            “|”DATABASE_COMPATIBILITY   “|”VOTIN
------------“|”----------“|”-----------“|”----------“|”--------------------“|”----------“|”--------“|”--------“|”-------“|”-----------“|”------------“|”-----------------------“|”--------------“|”-------------“|”-------------------------“|”-------------------------“|”-----
           7“|”DATA      “|”        512“|”      4096“|”             4194304“|”UNMOUNTED “|”EXTERN  “|”       0“|”      0“|”          0“|”           0“|”                     0“|”         870316“|”            0“|”10.2.0.0.0               “|”10.2.0.0.0               “|”N
           8“|”FLASH     “|”        512“|”      4096“|”             4194304“|”MOUNTED   “|”EXTERN  “|”  870396“|” 870316“|”          0“|”          80“|”                     0“|”         870316“|”            0“|”10.0.0.0.0               “|”10.2.0.0.0               “|”N



DB1
=======

GROUP_NUMBER DISK_NUMBER COMPOUND_INDEX INCARNATION MOUNT_S HEADER_STATU MODE_ST STATE    REDUNDA LIBRARY    TOTAL_MB    FREE_MB NAME         FAILGROUP   LABEL   PATH                                              UDID PRODUCT  CREATE_DA MOUNT_DAT REPAIR_TIMER      READS     WRITES  READ_ERRS WRITE_ERRS  READ_TIME WRITE_TIME BYTES_READ BYTES_WRITTEN
------------ ----------- -------------- ----------- ------- ------------ ------- -------- ------- ------------------- ---------- ------------ ----------  ------- ------------------------------------------------------ -------- --------- --------- ------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- -------------
           0           0              0           0 CLOSED  FOREIGN      ONLINE  NORMAL   UNKNOWN System         3048          0                                  /dev/rdsk/c4t600A0B800067E686000003B74C2843F3d0s0                                              0
           0           1              1           0 CLOSED  FOREIGN      ONLINE  NORMAL   UNKNOWN System         3048          0                                  /dev/rdsk/c4t600A0B800067E686000003B54C2841D4d0s0                                              0
           0           2              2           0 CLOSED  MEMBER       ONLINE  NORMAL   UNKNOWN System       204752          0                                  /dev/rdsk/c4t600A0B800067E686000003C74C297EBDd0s0               29-JUN-10 22-JAN-14            0
           0           3              3           0 CLOSED  MEMBER       ONLINE  NORMAL   UNKNOWN System       204752          0                                  /dev/rdsk/c4t600A0B800067E686000003C94C297EFFd0s0               29-JUN-10 22-JAN-14            0
           2           0       33554432  4042700921 CACHED  MEMBER       ONLINE  NORMAL   UNKNOWN System       298948     224182 FLASH_0000   FLASH_0000          /dev/rdsk/c4t600A0B800067E686000003CB4C297F64d0s0               29-JUN-10 22-JAN-14            0         12          1          0          0        .05          0      53248          4096

GROUP_NUMBER“|”NAME      “|”SECTOR_SIZE“|”BLOCK_SIZE“|”ALLOCATION_UNIT_SIZE“|”STATE     “|”TYPE    “|”TOTAL_MB“|”FREE_MB“|”HOT_USED_MB“|”COLD_USED_MB“|”REQUIRED_MIRROR_FREE_MB“|”USABLE_FILE_MB“|”OFFLINE_DISKS“|”COMPATIBILITY            “|”DATABASE_COMPATIBILITY   “|”VOTIN
------------“|”----------“|”-----------“|”----------“|”--------------------“|”----------“|”--------“|”--------“|”-------“|”-----------“|”------------“|”-----------------------“|”--------------“|”-------------“|”-------------------------“|”-------------------------“|”-----
           7“|”DATA      “|”        512“|”      4096“|”             4194304“|”MOUNTED   “|”EXTERN  “|”  870396“|” 870316“|”          0“|”          80“|”                     0“|”         870316“|”            0“|”10.0.0.0.0               “|”10.2.0.0.0               “|”N
           8“|”FLASH     “|”        512“|”      4096“|”             4194304“|”MOUNTED   “|”EXTERN  “|”  870396“|” 870316“|”          0“|”          80“|”                     0“|”         870316“|”            0“|”10.0.0.0.0               “|”10.2.0.0.0               “|”N


Workaround
==========

1. raw devices ownership and permission reassignment
–OCR/VOTE
chown -h orarac:oinstall c100230492348294
chmod -R 664 c100230492348294
–DB
chown -h orarac:oinstall c24294kwhfksjhfk
chmod -R 777 c24294kwhfksjhfk

2. crsctl stop crs
3. crsctl start crs
4. found that one disk group is not mount in DB1 becuase one disk is not configured permission properly
5. Repeat step 1 to 3.

🙂

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s