Oracle by Example - ORA-09968: unable to lock file

====================
General
====================
Oracle has stopped unexpectedly, without any apparent error in alert.log
When trying to startup after this shutdown, following error appears:
ORA-09968: unable to lock file

====================
alert.log
====================
Starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
Thu Nov 24 14:54:48 2016
MMNL started with pid=16, OS id=578
starting up 1 shared server(s) ...
Oracle Data Guard is not available in this edition of Oracle.
ORACLE_BASE from environment = /software/oracle
Thu Nov 24 14:54:48 2016
ALTER DATABASE   MOUNT
sculkget: failed to lock /software/oracle/111/dbs/lkIGT exclusive
sculkget: lock held by PID: 2278
ORA-09968: unable to lock file
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 2278
ORA-1102 signalled during: ALTER DATABASE   MOUNT...
Thu Nov 24 14:54:54 2016
Shutting down instance (abort)
License high water mark = 1
USER (ospid: 2127): terminating the instance
Instance terminated by USER, pid = 2127

====================
Solution
====================

Per Oracle Metalink #160395

fact: Oracle Server - Enterprise Edition

fact: Unix platforms only

symptom: Database startup fails

symptom: ORA-01102: cannot mount database in EXCLUSIVE mode

symptom: ORA-09968: scumnt: unable to lock file

symptom: The lk<SID> file is created successfully

cause: There are some client shadow processes hanging. Although the lk<SID> file is deleted the hanging processes still have a lock on the open file handle. This prevents the database to startup although a new lk<SID> file can be created successfully.

An oracle process (background or shadow process) that exists while the instance 

is not started (crashed or not cleanly stopped) can have a lock on a file while 

this file is actually removed from the system. This is because on UNIX there is 

still a lock on the open file handle.

The fix:

Once Oracle is stopped, verify if there are existing oracle processes for the database :

  $ ps -ef | grep $ORACLE_SID

Kill the hanging processes to release the lock on the file handle :

  $ kill -signal <PID>