High Memory Utilization on RAC Node on Solaris

NOTE:1010818.1 – Solaris[TM] Operating System: DISM double allocation of memory
NOTE:778777.1 – When Will DISM Start On Oracle Database?

Solaris[TM] Operating System: DISM double allocation of memory (Doc ID 1010818.1)

APPLIES TO:

Solaris x64/x86 Operating System - Version 8 6/00 U1 and later
Solaris SPARC Operating System - Version 8.0 and later
OpenSolaris Operating System - Version 2008.05 and later
Oracle Solaris Express - Version 2010.11 and later
All Platforms
GOAL

To understand the differences between memory allocation by Intimate Shared Memory (ISM) and Dynamic Intimate Shared Memory (DISM).

SOLUTION

Summary of differences between ISM and DISM:

ISM (Intimate Shared Memory)

ISM is used by default in Solaris
SGA shared memory segment locking is carried out by the Solaris kernel
ISM segments cannot be dynamically resized
Using ISM, if you change SGA components you must restart the process instance.

NOTE: When using ISM, any memory between real SGA and SGA_MAX_SIZE is not available for other running processes.

DISM (Dynamic Intimate Shared Memory)

DISM allows shared memory segments to be adjusted dynamically.
DISM is enabled by the use of flags during allocation.
With DISM, shared memory locking is carried out by the application using mlock().
DISM can dynamically resize SGA components.
With DISM, it is possible to add or remove memory boards and dynamically resize SGA components without the need to restart the process instance.
Use of DISM means that kernel virtual-to-physical memory address translation structures are shared between processes that attach to the DISM segment, saving kernel memory and CPU time.

NOTE:

Swap should be sized as large as SGA_MAX_SIZE.
In Oracle, the process ora_dism_$ORACLE_SID locks and unlocks memory pages for the attached Oracle process.
Unused segments from the physical memory will be deallocated into the swap area.
In Oracle, if you configure SGA_MAX_SIZE larger than the sum of its components, Oracle will attempt to use DISM.

When a shared memory segment is allocated via shmget(), its size is subtracted from the available virtual swap space. When the shared memory segment is attached to a process, it is declared as ISM or DISM by the flags to the shmat() call:

#define SHM_SHARE_MMU 040000 /* share VM resources such as page table */
#define SHM_PAGEABLE 0100000 /* pageable ISM */

The SHM_SHARE_MMU flag defines the shared memory segment to be an ISM segment, while the SHM_PAGEABLE flag defines it to be a DISM segment.

NOTE: A segment cannot be both ISM and DISM. An attempt to set both flags will fail.

With DISM, shared memory segment locking is performed by the application. A DISM segment can have portions of the segment selectively locked into memory by a (root-privileged) process, using the mlock() call. This allows the program to more selectively manage which portions of the shared memory need to be locked at a given time.

The down side to this is that the DISM segment will subtract the size of the segment from the available memory twice, once for its in-memory space and a second time for its possible swap usage. This effectively doubles the virtual memory requirements of any DISM segments.
The second allocation occurs during the mlock() call. Thus, a system using DISM needs to have twice as much virtual swap available as the total DISM segments in use. If the system exhausts available swap, DISM segments will not be mlock()'able.

Reservation is performed against the physical swap (disk) first. If that is exhausted or not configured then reservation is performed against the physical memory. If both are exhausted then shmat() or mlock() may fail. To avoid such failures due to lack of virtual swap you must configure a large amount of physical swap in the form of disk slices or files.

Virtual swap = Physical swap (disk) + physical memory

If the system is intended to have little or no disk swap, DISM is not considered appropriate and should not be used.

If NUMA (Non-uniform-memory-access) is involved, as found in the T-series servers, it is recommended to set following NUMA tunable in OracleDB init.ora (hidden parameter):

OracleDB 8.1.7->11.1.0 _enable_NUMA_optimization=TRUE
as mentioned in: ISM or DISM Misconfiguration can Slow Down Oracle Database Performance (Doc ID 1472108.1)

Still have questions about shared memory? Consider asking them in the My Oracle Support "Oracle Solaris Performance, Panics, Hangs, and Dtrace" Community.
FOR EACH INSTANCE
=====================
1. Completely disable the DISM feature by setting the following parameter and restarting the instances:

_use_ism=FALSE

OR

2a. Remove the MEMORY_TARGET and MEMORY_MAX_TARGET parameters.

AND

2b. Set the SGA_TARGET=SGA_MAX_SIZE

OR

3. Double the swap to 2 times the amount of physical RAM

Analysis

=======
We had removed Memory_target & Memory_max_target parameters and configured SGA_Target + PGA_Target.

after restart for each instance Ora_dism_+ORCLDB1, Ora_dism_+ORCLDB2 got terminated automatically & memory utilization got down on each servers.

🙂

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s