ORA-27086: unable to lock file - already in use Linux-x86_64 Error: 37: No locks available

##########################
## Error
##########################

ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 37: No locks available

### Full Error

Tue May 20 08:58:08 2014
Control file backup creation failed.
Backup target file size found to be zero.
Errors in file /u01/app/oracle/diag/rdbms/oralin/oralin1/trace/oralin1_ckpt_14497.trc:
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 37: No locks available
Additional information: 10

##########################
#  Error Occurred
##########################

Error occured in RMAN FULL backup while taking backup of control file in 11.2.0.3.0 version

##########################
## Command Executed
##########################

Backup of control file

**************************************** Step By Step Analysis ******************************************************

#########################################
# Alert Log
#########################################

Tue May 20 08:58:08 2014
Control file backup creation failed.
Backup target file size found to be zero.
Errors in file /u01/app/oracle/diag/rdbms/oralin/oralin1/trace/oralin1_ckpt_14497.trc:
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 37: No locks available
Additional information: 10

#########################################
# Trace File ( /u01/app/oracle/diag/rdbms/oralin/oralin1/trace/oralin1_ckpt_14497.trc )
#########################################

ORA-27086: unable to lock file - already in use

*** 2014-05-20 08:58:08.285
Linux-x86_64 Error: 37: No locks available
Additional information: 10
Control file enqueue hold time tracking dump at time: 36768

#########################################
# 1) Check the Snapshot Controlfile
#########################################

RMAN> show snapshot controlfile name;
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name oralin are:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/backup/db/oralin/CONTROLFILE_SNAPSHOT_oralin/snapcf_oralin1.f';

RMAN>

Here, /backup is the shared NFS Mount Point.

=====================================================================================================================

#########################################
# 2) Check NFS lock status
#########################################

service nfslock status

[root@host01 ~]# service nfslock status
rpc.statd is stopped
[root@host01 ~]# 

NFS lock is in stopped state.

[root@host01 ~]# chkconfig --list | grep nfs
nfs             0:off   1:off   2:off   3:off   4:off   5:off   6:off
nfslock         0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@host01 ~]#

=====================================================================================================================

#########################################
# 3) Reason for Failure
#########################################

Error is occuring when autobackup of controlfile is being done. When RMAN is trying to create a snapshot control file in the shared NFS MP, its says unable to lock file.

From Step 2, we can see that nfs lock service is in stopped state, so RMAN cannot hold a lock.

This server was rebooted recently and nfs lock service didnt come up and also we can see from step 2 that auto start is in OFF

=====================================================================================================================

#########################################
# 4) Reproduce the Error
#########################################

A simple backup of controlfile in NFS Mount Point will help us to reproduce the error.

backup current controlfile format '/backup/db/oralin/CONTROLFILE_SNAPSHOT_oralin/oralin.ctl';

RMAN> backup current controlfile format '/backup/db/oralin/CONTROLFILE_SNAPSHOT_oralin/oralin.ctl';
Starting backup at 20-MAY-14
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=4 instance=oralin1 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/20/2014 09:27:03
ORA-01580: error creating control backup file /backup/db/oralin/CONTROLFILE_SNAPSHOT_oralin/snapcf_oralin1.f
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 37: No locks available
Additional information: 10

=====================================================================================================================

##########################
## Solution
##########################

#### Start NFS Lock services in the server

[root@host01 ~]# service nfslock start
Starting NFS statd:                                        [  OK  ]
[root@host01 ~]# service nfslock status
rpc.statd (pid 29693) is running...
[root@host01 ~]#

=====================================================================================================================
Test the backup, now it completes successfully...
=====================================================================================================================

RMAN> backup current controlfile format '/backup/db/oralin/CONTROLFILE_SNAPSHOT_oralin/oralin.ctl';

Starting backup at 20-MAY-14
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 20-MAY-14
channel ORA_DISK_1: finished piece 1 at 20-MAY-14
piece handle=/backup/db/oralin/CONTROLFILE_SNAPSHOT_oralin/oralin.ctl tag=TAG20140520T092820 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 20-MAY-14
Starting Control File and SPFILE Autobackup at 20-MAY-14
piece handle=/backup/db/oralin/autobackup/c-419595185-20140520-02 comment=NONE
Finished Control File and SPFILE Autobackup at 20-MAY-14
RMAN>

=====================================================================================================================
 Comments Are Always welcome
=====================================================================================================================