FAN Callouts for RAC 12.1.0.2.0

Problem Area:-
If prefer node (Node1) is down then services automatically Failover to available node (Node2)
When the prefer node (Node1) comes up then services stay running on available node (Node2)

About FAN Callouts
FAN callouts are server-side executable files that Oracle RAC runs immediately when high availability events occur. A callout is essentially a shell script or precompiled executable written in any programming language. Some examples of how you can use FAN callouts to automate the actions performed when events occur in a cluster configuration are as follows:
• Starting and stopping server-side applications
• Relocating low-priority services when high-priority services come online
• Sending text or numeric messages to pagers
• Executing shell scripts
Callout script for relocating services back to their preferred instances is available

Script to automatically move RAC 11gR2 services back to preferred instances
Modified and Tested: RHEL 6.4 with 11.2.0.3 Oracle Grid Infrastructure

Create the following callout sample script as Oracle user in
$GRID_HOME/racg/usrco on Node1
You must set the execute permission for the callout
chmod a+x $GRID_HOME/racg/usrco/relocate_services_callout.sh
In addition, you must remember to copy the script to each node in the cluster.
The script will automatically be executed whenever the HA framework generates a RAC event.

Script to automatically move RAC 11gR2 services back to preferred instances

#!/bin/bash
#
# GI callout script to catch INSTANCE up event from clusterware and relocate services to preferred instance
# Copy or symlink this script to $GRID_HOME/racg/usrco
# Tested on Oracle Linux 5.8 with 11.2.0.3 Oracle Grid Infrastructure and 11.2.0.2 & 11.2.0.3 Oracle Database Enterprise Edition
# 2012 Ilmar Kerm ilmar.kerm@gmail.com
#Modified for log location and tested on RHEL 6.4 with 11.2.0.3 Oracle Grid Infrastructure
# 2014 Hitesh G https://hiteshgondalia.wordpress.com/

FAN_LOGFILE=/home/oracle/log/rac_`hostname`.log
echo $* "reported="`date` >> $FAN_LOGFILE &

LOGFILE=/home/oracle/log.log
SCRIPTDIR=`dirname $0`

# Determine grid home
if [[ "${SCRIPTDIR:(-11)}" == "/racg/usrco" ]]; then
  CRS_HOME=""${SCRIPTDIR:0:$(( ${#SCRIPTDIR} - 11 ))}""
  export CRS_HOME
fi

# Only execute script for INSTANCE events
if [ "$1" != "INSTANCE" ]; then
  exit 0
fi

STATUS=""
DATABASE=""
INSTANCE=""

# Parse input arguments
args=("$@")
for arg in ${args[@]}; do
  if [[ "$arg" == *=* ]]; then
    KEY=${arg%=*}
    VALUE=${arg#*=}

    case "$KEY" in
      status)
        STATUS="$VALUE"
        ;;
      database)
        DATABASE="$VALUE"
        ;;
      instance)
        INSTANCE="$VALUE"
        ;;
    esac

  fi
done

# If database, status and instance values are not set, then exit
# status must be up
if [[ -z "$DATABASE" || -z "$INSTANCE" || "$STATUS" != "up" ]]; then
  exit 0
fi

echo "`date`" >> "$LOGFILE"
echo "[$DATABASE][`hostname`] Instance $INSTANCE up" >> "$LOGFILE"

#
# Read database software home directory from clusterware
#
DBCONFIG=`$CRS_HOME/bin/crsctl status res ora.$DATABASE.db -f | grep "ORACLE_HOME="`
if [ -z "$DBCONFIG" ]; then
  exit 0
fi
declare -r "$DBCONFIG"
echo "ORACLE_HOME=$ORACLE_HOME" >> "$LOGFILE"

# Array function
in_array() {
    local hay needle=$1
    shift
    for hay; do
        [[ $hay == $needle ]] && return 0
    done
    return 1
}

#
# Read information about services
#
for service in `$CRS_HOME/bin/crsctl status res | grep -E "ora.$DATABASE.(.+).svc" | sed -rne "s/NAME=ora.$DATABASE.(.+).svc/1/gip"`; do
  SERVICECONFIG=`$ORACLE_HOME/bin/srvctl config service -d $DATABASE -s $service`
  echo "Service $service" >> "$LOGFILE"
  if [[ `echo "$SERVICECONFIG" | grep "Service is enabled" | wc -l` -eq 1 ]]; then
    echo " enabled" >> "$LOGFILE"
    PREFERRED=( `echo "$SERVICECONFIG" | grep "Preferred instances:" | sed -rne "s/.*: ([a-zA-Z0-9]+)/1/p" | tr "," "n"` )
    #
    # Check if current instance is preferred for this service
    #
    if in_array "$INSTANCE" "${PREFERRED[@]}" ; then
      echo " preferred" >> "$LOGFILE"
      #
      # Check if service is already running on current instance
      #
      SRVSTATUS=`$ORACLE_HOME/bin/srvctl status service -d $DATABASE -s $service`
      if [[ "$SRVSTATUS" == *"is not running"* ]]; then
          #
          # if service is not running, then start it
          #
        echo " service stopped, starting" >> "$LOGFILE"
        $ORACLE_HOME/bin/srvctl start service -d "$DATABASE" -s "$service" >> "$LOGFILE"
      else
        #
        # Service is running, but is it running on preferred instance?
        #
        RUNNING=( `echo "$SRVSTATUS" | sed -rne "s/.* ([a-zA-Z0-9]+)/1/p" | tr "," "n"` )
        echo "${RUNNING[@]} = ${PREFERRED[@]}"
        if ! in_array "$INSTANCE" "${RUNNING[@]}" ; then
          echo " not running on preferred $INSTANCE" >> "$LOGFILE"
          #
          # Find the first non-preferred running instance
          #
          CURRENT=""
          for inst in "${RUNNING[@]}"; do
            if ! in_array "$inst" "${PREFERRED[@]}" ; then
              CURRENT="$inst"
              break
            fi
          done
          #
          # Relocate
          #
          if [[ -n "$CURRENT" ]]; then
            echo " relocate $CURRENT -> $INSTANCE" >> "$LOGFILE"
            $ORACLE_HOME/bin/srvctl relocate service -d "$DATABASE" -s "$service" -i "$CURRENT" -t "$INSTANCE" >> "$LOGFILE"
          fi
        else
          #
          # Service is already running on preferred instance, no need to do anything
          #
          echo " running on preferred $INSTANCE" >> "$LOGFILE"
        fi
      fi
    fi
  fi
done

Tested Lab
RHEL 6.464bit OS
3 – Node Oracle RAC 11.2.0.3.0 Working fine
3 – Node Oracle RAC 12.1.0.2.0 Working fine
🙂

Advertisements

2 thoughts on “FAN Callouts for RAC 12.1.0.2.0

  1. Hi Hitesh,

    We have tested this feature perfectly running fine on 11.2.0.3 RAC ASM @ RHEL 6.7

    Thanks for wonderful article.

    Regards,
    Chirag Rathod

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s