Oracle memory and /dev/shm

==========================
Oracle memory and /dev/shm
==========================

Consider following real life scenario.

Oracle max_memory and max_memory_target were increased, and Oracle instance restarted.

Following is from alert.log

Starting up ORACLE RDBMS Version: 11.1.0.7.0.
Using parameter settings in server-side pfile /software/oracle/111/dbs/initigt.ora
System parameters with non-default values:
  processes                = 600
  spfile                   = "/software/oracle/admin/igt/pfile/spfileigt.ora"
  nls_length_semantics     = "char"
  filesystemio_options     = "asynch"
  memory_target         = 8G
  control_files            = "/oracle_db/db1/db_igt/ora_control_01.ctl"
  control_files            = "/oracle_db/db1/db_igt/ora_control_02.ctl"
  control_files            = "/oracle_db/db1/db_igt/ora_control_03.ctl"
  db_block_size            = 8192
  compatible               = "11.1.0.0.0"
  log_archive_dest_1       = "location=/oracle_db/db2/db_igt/arch"
  log_archive_format       = "arch%T_%s_%r.arc"
  archive_lag_target       = 1800
  fast_start_mttr_target   = 180
  undo_tablespace          = "UNDOTBS"
  undo_retention           = 3600
  recyclebin               = "OFF"
  remote_login_passwordfile= "EXCLUSIVE"
  db_domain                = ""
  dispatchers              = "(PROTOCOL=TCP) (SERVICE=igtXDB)"
  session_cached_cursors   = 100
  audit_file_dest          = "/software/oracle/admin/igt/adump"
  open_links               = 12
  audit_trail              = "NONE"
  db_name                  = "igt"
  open_cursors             = 300
  os_authent_prefix        = ""
  diagnostic_dest          = "/software/oracle"
Wed Nov 18 12:22:14 2015
PMON started with pid=2, OS id=17174
Wed Nov 18 12:22:14 2015
VKTM started with pid=3, OS id=17178 at elevated priority
VKTM running at (20)ms precision
Wed Nov 18 12:22:14 2015
DIAG started with pid=4, OS id=17184
Wed Nov 18 12:22:14 2015
DBRM started with pid=5, OS id=17188
Wed Nov 18 12:22:14 2015
DBRM started with pid=5, OS id=17188
Wed Nov 18 12:22:14 2015
PSP0 started with pid=6, OS id=17192
Wed Nov 18 12:22:14 2015
DIA0 started with pid=7, OS id=17202
Wed Nov 18 12:22:14 2015
MMAN started with pid=8, OS id=17215
Wed Nov 18 12:22:15 2015
DBW0 started with pid=9, OS id=17219
Wed Nov 18 12:22:15 2015
DBW1 started with pid=10, OS id=17223
Wed Nov 18 12:22:15 2015
LGWR started with pid=11, OS id=17227
Wed Nov 18 12:22:15 2015
CKPT started with pid=12, OS id=17232
Wed Nov 18 12:22:15 2015
SMON started with pid=13, OS id=17242
Wed Nov 18 12:22:15 2015
RECO started with pid=14, OS id=17253
Wed Nov 18 12:22:15 2015
MMON started with pid=15, OS id=17257
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
Wed Nov 18 12:22:15 2015
MMNL started with pid=16, OS id=17261
starting up 1 shared server(s) ...
ORACLE_BASE from environment = /software/oracle
Wed Nov 18 12:22:17 2015
ALTER DATABASE   MOUNT
Setting recovery target incarnation to 1
Successful mount of redo thread 1, with mount id 1134364153
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Wed Nov 18 12:22:21 2015
ALTER DATABASE OPEN
LGWR: STARTING ARCH PROCESSES
Wed Nov 18 12:22:21 2015
ARC0 started with pid=20, OS id=17426
Wed Nov 18 12:22:21 2015
ARC1 started with pid=21, OS id=17430
Wed Nov 18 12:22:21 2015
ARC2 started with pid=22, OS id=17434
Wed Nov 18 12:22:21 2015
ARC3 started with pid=23, OS id=17438
ARC0: Archival started
ARC1: Archival started
ARC2: Archival started
ARC3: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Thread 1 opened at log sequence 441882
  Current log# 3 seq# 441882 mem# 0: /oracle_db/db1/db_igt/ora_redo_03_a.rdo
Successful open of redo thread 1
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is AL32UTF8
Opening with internal Resource Manager plan
Starting background process FBDA
Wed Nov 18 12:22:23 2015
FBDA started with pid=24, OS id=17481
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Wed Nov 18 12:22:24 2015
QMNC started with pid=25, OS id=17505
Completed: ALTER DATABASE OPEN
...
...
...
...
Wed Nov 18 13:38:35 2015
Starting ORACLE instance (normal)
WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to be mounted for at least 8589934592 bytes. /dev/shm is either not mounted or is mounted with available space less than this size. Please fix this so that MEMORY_TARGET can work as expected. Current available is 3500191744 and used is 5089742848 bytes.
memory_target needs larger /dev/shm
Wed Nov 18 13:39:39 2015
Errors in file /software/oracle/diag/rdbms/igt/igt/trace/igt_pmon_17174.trc:
ORA-56710: DBRM process died unexpectedly
PMON (ospid: 17174): terminating the instance due to error 56710
Instance terminated by PMON, pid = 17174

What is /dev/shm?

It is an in-memory mounted file system (tmpfs) and is very fast, but non-persistent when Linux is rebooted.

In Oracle 11g, it is used to hold SGA memory by storing the SGA structures in files with the same granule size. 

When these MEMORY_TARGET and MEMORY_MAX_TARGET parameters are set, oracle will create MEMORY_MAX_TARGET/granule size files. 

In this example, MEMORY_TARGET was set to 8Gb.

8*1024*1024*1024/67108864=128

So there are in total 128 files.

The output of ‘ls -lk /dev/shm’, will show you that not all the 128 files are taking the 4M of space.
ls -lk /dev/shm

-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_0
-rw-r----- 1 oracle dba     0 Nov 18 15:39 ora_igt_4915207_1
-rw-r----- 1 oracle dba     0 Nov 18 15:39 ora_igt_4915207_10
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_100
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_101
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_102
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_103
-rw-r----- 1 oracle dba 65536 Nov 19 07:48 ora_igt_4915207_104
-rw-r----- 1 oracle dba 65536 Nov 19 02:06 ora_igt_4915207_105
-rw-r----- 1 oracle dba 65536 Nov 19 07:48 ora_igt_4915207_106
-rw-r----- 1 oracle dba 65536 Nov 19 07:48 ora_igt_4915207_107
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_108
-rw-r----- 1 oracle dba 65536 Nov 19 02:06 ora_igt_4915207_109
-rw-r----- 1 oracle dba     0 Nov 18 15:39 ora_igt_4915207_11......
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_90
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_91
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_92
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_93
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_94
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_95
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_96
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_97
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_98
-rw-r----- 1 oracle dba 65536 Nov 19 07:49 ora_igt_4915207_99

ls -lk /dev/shm | grep 65536 | wc -l
77

In total there are 77 file, each file has size of 67108864 bytes.

67108864*77/1024/1024=4928 Mb

This is the same value, as checking SGA size in V$SGASTAT

SELECT ROUND(SUM(bytes)/1024/1024) AS MB_MEMORY FROM V$SGASTAT;

4827

When you add up all the files that do take space, it will never take more space than MEMORY_TARGET. 

PGA is not stored in shared memory, because this is private memory. MEMORY_MAX_TARGET (used for SGA and PGA) is ‘allocated’ in /dev/shm, but PGA is not stored in /dev/shm. 
This means, when memory for PGA is allocated (and/or pga_aggregate_target is set), not all files in /dev/shm will get used!

What about the error memory_target needs larger /dev/shm

less /etc/fstab

/dev/Volume00/LogVol00  /                       ext3    defaults,acl        1 1

/dev/cciss/c0d0p1       /boot                   ext3    defaults,acl        1 2

devpts                  /dev/pts                devpts  gid=5,mode=620      0 0

tmpfs                /dev/shm              tmpfs  size=8g          0 0

/dev/Volume00/LogVol05  /home                   ext3    defaults,acl        1 2

/dev/Volume00/LogVol10  /kits                   ext3    defaults,acl        1 2

/dev/Volume00/LogVol04  /opt                    ext3    defaults,acl        1 2

/dev/Volume00/LogVol13  /oracle_db              ext3    defaults,acl        1 2

proc                    /proc                   proc    defaults            0 0

/dev/Volume00/LogVol12  /software               ext3    defaults,acl        1 2

/dev/Volume00/LogVol07  /software/oracle        ext3    defaults,acl        1 2

df -h

Filesystem                     Size  Used Avail Use% Mounted on

/dev/mapper/Volume00-LogVol00  2.0G  697M  1.2G  38% /

/dev/cciss/c0d0p1              927M  142M  738M  17% /boot

tmpfs                      8.0G  4.8G  3.3G  60% /dev/shm

/dev/mapper/Volume00-LogVol05  992M   34M  908M   4% /home

/dev/mapper/Volume00-LogVol10  2.5G  961M  1.4G  41% /kits

/dev/mapper/Volume00-LogVol04  2.0G  499M  1.4G  27% /opt

If you increase the MEMORY_MAX_TARGET above the available /dev/shm space, you will receive:

ORA-00845: MEMORY_TARGET not supported on this system

===============
Solution
===============
Option A.
If you have enough memory on your Linux machine, but /dev/shm is mounted to small by default, one can increase this amount of memory by changing /etc/fstab for permanent changes. The default is half of your physical RAM without swap.

For a permanent change, need to edit /etc/fstab file

vi /etc/fstab

modify /dev/shm entry as follows to set size to 8G
none      /dev/shm        tmpfs   defaults,size=8G        0 0

Save and close the file. 

For the changes to take effect immediately remount /dev/shm:
mount -o remount /dev/shm

Option B.

For temporary change, at least to start the database, execute the following.
In this example, /dev/shm is set to 8000Mb:

> umount tmpfs
> mount -t tmpfs shmfs -o size=8000m /dev/shm

Option C.

The other option would be to decrease the memory_targer, as described here.

References

Oracle 11g AMM: MEMORY_TARGET, MEMORY_MAX_TARGET and /dev/shm

Automatic Memory Management