ORA-19505: failed to identify file "/opt/backup/arch/orahost3_oralin_oralin_20140707_1096_1_1_arch"

##########################
## Error
##########################

ORA-19505: failed to identify file "/opt/backup/arch/orahost3_oralin_oralin_20140707_1096_1_1_arch"

### Full Error

RMAN> restore archivelog logseq 5481 thread 2;
Starting restore at 07-JUL-14
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=2 sequence=5481
channel ORA_DISK_1: reading from backup piece /opt/backup/arch/orahost3_oralin_oralin_20140707_1096_1_1_arch
channel ORA_DISK_1: ORA-19870: error while restoring backup piece /opt/backup/arch/orahost3_oralin_oralin_20140707_1096_1_1_arch
ORA-19505: failed to identify file "/opt/backup/arch/orahost3_oralin_oralin_20140707_1096_1_1_arch"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

##########################
#  Error Occurred
##########################

Error occured while trying to restore a archivelog from a backup in 11.2.0.3.0 version of RAC Database.

##########################
## Command Executed
##########################

restore archivelog logseq 5481 thread 2;

**************************************** Step By Step Analysis ******************************************************

#########################################
# 1) Check Archive log Information
#########################################

set lines 200
col name for a75
col thread# for 9999
col creator for a8
col completion_time for a20

Select name,creator,thread#,sequence#,applied,status,to_char(first_time,'DD-MON-YYYY HH24:MI:SS') "first_time",to_char(completion_time,'DD-MON-YYYY HH24:MI:SS') "completion_time",resetlogs_id,archived,backup_count
from v$archived_log where sequence#>=5481 and thread#=2 order by sequence# desc;

NAME                                                                             CREATOR  THREAD#  SEQUENCE# APPLIED   S completion_time      RESETLOGS_ID ARC BACKUP_COUNT
-------------------------------------------------------------------------------- -------- ------- ---------- --------- - -------------------- ------------ --- ------------
+RECO/oralin/archivelog/2014_07_07/thread_2_seq_5484.411.852301159              ARCH           2       5484 NO        A 07-JUL-2014 14:19:32    825774564 YES            0
+RECO/oralin/archivelog/2014_07_07/thread_2_seq_5483.683.852301111              ARCH           2       5483 NO        A 07-JUL-2014 14:18:46    825774564 YES            0
                                                                                 ARCH           2       5482 NO        D 07-JUL-2014 12:01:56    825774564 YES            0
                                                                                 ARCH           2       5481 NO        D 07-JUL-2014 12:01:54    825774564 YES            1

Backup_count is 1, so archive log sequence 5481 is backed up once.

[oracle@orahost3 arch]$ ls -ltr
total 10648680
-rw-r----- 1 oracle oinstall 2199391232 Jul  7 12:04 orahost3_oralin_oralin_20140707_1096_1_1_arch
[oracle@orahost3 arch]$ 

=====================================================================================================================

#########################################
# 2) Reason for Failure
#########################################

RMAN tries to fetch the backup piece to restore the archive log but it can't find the backup piece.

This is a RAC cluster database. Backups are scheduled to run from orahost3 server. Restore command was run from orahost1 server.

=====================================================================================================================

#########################################
# 3) Reproduce the Error
#########################################

[oracle@orahost1 srini]$ rman target /
Recovery Manager: Release 11.2.0.3.0 - Production on Mon Jul 7 15:20:31 2014
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
connected to target database: oralin (DBID=422297444)
RMAN> restore archivelog logseq 5481 thread 2;
Starting restore at 07-JUL-14
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=2 sequence=5481
channel ORA_DISK_1: reading from backup piece /opt/backup/arch/orahost3_oralin_oralin_20140707_1096_1_1_arch
channel ORA_DISK_1: ORA-19870: error while restoring backup piece /opt/backup/arch/orahost3_oralin_oralin_20140707_1096_1_1_arch
ORA-19505: failed to identify file "/opt/backup/arch/orahost3_oralin_oralin_20140707_1096_1_1_arch"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
failover to previous backup
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 07/07/2014 15:20:54
RMAN-06026: some targets not found - aborting restore
RMAN-06025: no backup of archived log for thread 2 with sequence 5481 and starting SCN of 9619365298 found to restore
RMAN>

=====================================================================================================================

##########################
## Solution
##########################

Backups are scheduled to run in orahost3, and the backup mount point /opt seems to be not a shared one.

Backup mount point mostly will be a shared one in RAC databases, but looks like in this server has some exemption for it.

Ran a restore by logging into orahost3 and the restore went successful.

=====================================================================================================================
In case if you encounter the same issues, please check the backup pieces in all the nodes of a cluster before coming to the conclusion that there is no backups available.
=====================================================================================================================

[oracle@orahost3 ~]$ rman target /
Recovery Manager: Release 11.2.0.3.0 - Production on Mon Jul 7 15:23:06 2014
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
connected to target database: oralin (DBID=422297444)
RMAN> restore archivelog logseq 5481 thread 2;
Starting restore at 07-JUL-14
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=566 instance=oralin3 device type=DISK
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=2 sequence=5481
channel ORA_DISK_1: reading from backup piece /opt/backup/arch/orahost3_oralin_oralin_20140707_1096_1_1_arch
channel ORA_DISK_1: piece handle=/opt/backup/arch/orahost3_oralin_oralin_20140707_1096_1_1_arch tag=%TAG
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:55
Finished restore at 07-JUL-14
RMAN>

=====================================================================================================================
 Comments Are Always welcome
=====================================================================================================================

Restore Archive logs from Disk Based Backups gives RMAN-06026: some targets not found - aborting restore RMAN-06102: no channel to restore a backup or copy of log thread 1 seq

One of our DG has gone Out of SYNC. Primary database is in MAXIMUM AVAILABILITY mode and we have specified standby location as an alternate destination.

Now our Primary archive diskgroup got 100% full and one of our team member has ran backup of archive logs to disk (Our Backup Tapes are not Working at that Moment) with the delete input option which has backed up all the Archives and deleted from Primary. 

Now Standby database is looking for archive log which is not in Diskgroup but in RMAN disk backup. 

##########################

#   Task 
##########################

Restore Archive logs from Disk Based Backups.

So our team member has started to make it sync by trying to restore the archives from the backup and got the below errors.

##########################
#   Command Used  
##########################

Command to Restore Archive logs to different location.


run
{
allocate channel c2 device type disk;
set archivelog destination to '/opt/oracle/backup/oralin/arch_bkp/';
restore archivelog from logseq 105408 until logseq 105413;
}

When running the restore got the below errors, 

##########################
#   Errors  
##########################

RMAN> run
{
allocate channel c2 device type disk;
set archivelog destination to '/opt/oracle/backup/oralin/arch_bkp/';
restore archivelog from logseq 105408 until logseq 105413;
}
2> 3> 4> 5> 6> 7>
released channel: ORA_SBT_TAPE_1
released channel: ORA_SBT_TAPE_2
released channel: ORA_SBT_TAPE_3
released channel: ORA_SBT_TAPE_4
released channel: ORA_DISK_1
allocated channel: c1
channel c1: sid=862 devtype=DISK
allocated channel: c2
channel c2: sid=1309 devtype=DISK
executing command: SET ARCHIVELOG DESTINATION
Starting restore at 27-FEB-13
released channel: c1
released channel: c2
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 02/27/2013 22:28:14
RMAN-06026: some targets not found - aborting restore
RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 105413 lowscn 13117372976201
RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 105412 lowscn 13117372752915
RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 105411 lowscn 13117372598071
RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 105410 lowscn 13117372461415
RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 105408 lowscn 13117372091300
RMAN>

We spent more time on investigating this error and to resolve the above error. But later on further investigation i found the problem made by the team member and the solution is very simple.

##########################
#   Pre checks 
##########################

1.  Check whether backup of Archive Logs available. 
2.  Is the Backup Stored in TAPE or in DISK. 

##########################

#  Solution
##########################

 Check whether backup of Archive Logs available. 

RMAN> list backup of archivelog from logseq 105408 until logseq 105413 thread 1;
List of Backup Sets
===================
BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
56800656 577.25M    SBT_TAPE    00:07:21     27-FEB-13
        BP Key: 56800662   Status: AVAILABLE  Compressed: NO  Tag: oralin_FULL_0
        Handle: bk_11040_1_808444905   Media: O00100
  List of Archived Logs in backup set 56800656
  Thrd Seq     Low SCN    Low Time  Next SCN   Next Time
  ---- ------- ---------- --------- ---------- ---------
  1    105408  13117372091300 26-FEB-13 13117372283649 26-FEB-13
  1    105409  13117372283649 26-FEB-13 13117372461415 26-FEB-13
  1    105410  13117372461415 26-FEB-13 13117372598071 26-FEB-13
  1    105411  13117372598071 26-FEB-13 13117372752915 26-FEB-13
  1    105412  13117372752915 26-FEB-13 13117372976201 26-FEB-13
  1    105413  13117372976201 26-FEB-13 13117373012970 27-FEB-13

From the above output if you can see the Backup is stored in SBT_TAPE, but our team member has allocated disk Channel because he took backup of archives to disk.

But the archives which needed to restore are already backed up in TAPE and its his understanding that the archives are stored in DISK Backup.

So After removing the Disk channel allocation command, the restore was successful.

RMAN>  run

{
set archivelog destination to '/opt/oracle/backup/oralin/arch_bkp/';
restore archivelog logseq 105409 thread 1;
}
2> 3> 4> 5>

executing command: SET ARCHIVELOG DESTINATION

Starting restore at 27-FEB-13
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: sid=862 devtype=SBT_TAPE
channel ORA_SBT_TAPE_1: Veritas NetBackup for Oracle - Release 7.1 (2011020313)
allocated channel: ORA_SBT_TAPE_2
channel ORA_SBT_TAPE_2: sid=1309 devtype=SBT_TAPE
channel ORA_SBT_TAPE_2: Veritas NetBackup for Oracle - Release 7.1 (2011020313)
allocated channel: ORA_SBT_TAPE_3
channel ORA_SBT_TAPE_3: sid=617 devtype=SBT_TAPE
channel ORA_SBT_TAPE_3: Veritas NetBackup for Oracle - Release 7.1 (2011020313)
allocated channel: ORA_SBT_TAPE_4
channel ORA_SBT_TAPE_4: sid=576 devtype=SBT_TAPE
channel ORA_SBT_TAPE_4: Veritas NetBackup for Oracle - Release 7.1 (2011020313)
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=359 devtype=DISK
channel ORA_SBT_TAPE_1: starting archive log restore to user-specified destination
archive log destination=/opt/oracle/backup/oralin/arch_bkp/
channel ORA_SBT_TAPE_1: restoring archive log
archive log thread=1 sequence=105409
channel ORA_SBT_TAPE_1: reading from backup piece bk_11063_1_808494442
channel ORA_SBT_TAPE_1: restored backup piece 1
piece handle=bk_11063_1_808494442 tag=TAG20130227T134717
channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:02:37
Finished restore at 27-FEB-13

So the moral of this story is listen what your team member says but always start with your own investigation path. :)