"Process J000 is dead" "kkjcre1p: unable to spawn jobq slave process"

Errors in alert.log

Multiple errors:

 "Process J000 is dead"

"kkjcre1p: unable to spawn jobq slave process"

Process J000 is dead (pid=10383 req_ver=7762 cur_ver=7762 state=KSOSP_SPAWNED).

2022-08-29T23:59:09.365187+00:00

Process J002 died, see its trace file

2022-08-29T23:59:09.365502+00:00

kkjcre1p: unable to spawn jobq slave process

2022-08-29T23:59:09.365830+00:00

Errors in file /software/oracle/diag/rdbms/igt/igt/trace/igt_cjq0_8620.trc:

2022-08-29T23:59:10.566884+00:00

Process J002 died, see its trace file

2022-08-29T23:59:10.567017+00:00

kkjcre1p: unable to spawn jobq slave process

Check 1

Was checked - but not related in this case 

Per Oracle Technote: "ORA-00610 And/Or "unable to spawn jobq secondary process " And/Or "Process(<>) creation failed" In The Alert Log And/Or TNS-12518/ TNS-12500 In Listener Log (Doc ID 416244.1)"

Check your SGA and PGA settings, make sure they are not set too low or too high. 

Preferably PGA_AGGREGATE_TARGET is set at a rate of 2 MBs per process defined by the PROCESSES

Current settings

pga_aggregate_limit  = 60129542144

pga_aggregate_target = 15728640000 

processes            = 1000

So pga_aggregate_target should be higher than 1000*2=2Gb

Now it is set to 16Gb.

check 1 - OK

Check 2

Check that kernel.shmmax and kernel.shmall are sized appropriately

Per Oracle Database Installation Guide for Linux

shmmax - should be "Half the size of physical memory in bytes"

shmall - should be "Greater than or equal to the value of shmmax, in pages."

shmmni - should be set to 4096

/sbin/sysctl -a | grep shm

This command displays the details of the shared memory segment sizes.

Per Oracle Technote 

What is the maximum value of SHMMAX for a 64-bit (x86-64) Linux system?

Oracle Global Customer Support officially recommends a " maximum" for SHMMAX of "1/2 of physical RAM".

Host memory is 193280Mb (202668769280 bytes)

free -m

            total     used   free   shared  buff/cache   available

Mem:       193280    82467  60637    33699       50175       76511

Swap:       10239       0   10239

/sbin/sysctl -a | grep shm

kernel.shmmax=549755813888

kernel.shmall=268435456

kernel.shmmaxis higher than Host memory!!

Edit /etc/sysctl.conf to set kernel.shmall and kernel.shmall

193280*1024*1024/2=101334384640

101334384640/(4*1024)=24739840

vi /etc/sysctl.conf 

kernel.shmmax = 101334384640

kernel.shmall = 24739840

root@server>% /sbin/sysctl -a | grep shm

kernel.shmall = 268435456

kernel.shmmax = 549755813888

kernel.shmmni = 4096

change settings

root@server>%/sbin/sysctl --system

Now

root@server>% /sbin/sysctl -a | grep shm

kernel.shmall = 24739840

kernel.shmmax = 101334384640