Whenever we see the error ORA-19815, we always assume that FRA is Nearing 100% and that is the reason we received the error.
But today i came across this and found that its not true all the times.
We used to get frequent ORA- errors from one of our Primary Dataguard database and the error recorded in Primary is ORA-16055 : FAL request rejected.
At the same time when we check the standby Alert log, error recorded in Standby is ORA-19815 which indicates that FRA is getting filled and reached 100%
##########################
# Error
##########################
ORA-00270: error creating archive log
ORA-16055: FAL request rejected
ORA-16041: Remote File Server fatal error
ORA-19815: WARNING: db_recovery_file_dest_size of 424128020480 bytes is 96.93% used, and has 13021216768 remaining bytes available.
##########################
# Errors Recorded in Primary
##########################
Mon Jan 7 16:35:49 2013
Errors in file /opt/oracle/admin/primdb/bdump/primdb_arc2_5171.trc:
ORA-00270: error creating archive log
Mon Jan 7 16:35:49 2013
FAL[server, ARC2]: FAL archive failed, see trace file.
Mon Jan 7 16:35:49 2013
Errors in file /opt/oracle/admin/primdb/bdump/primdb_arc2_5171.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
Mon Jan 7 16:35:49 2013
ORACLE Instance primdb - Archival Error. Archiver continuing.
Mon Jan 7 16:36:03 2013
Errors in file /opt/oracle/admin/primdb/bdump/primdb_arc1_5169.trc:
ORA-16041: Remote File Server fatal error
##########################
# Errors Recorded in Standby
##########################
Mon Jan 7 16:35:49 2013
Errors in file /opt/oracle/admin/primdb/udump/primdb_rfs_17693.trc:
ORA-19815: WARNING: db_recovery_file_dest_size of 424128020480 bytes is 96.93% used, and has 13021216768 remaining bytes available.
Mon Jan 7 16:35:49 2013
Errors in file /opt/oracle/admin/primdb/udump/primdb_rfs_17693.trc:
ORA-00270: error creating archive log +primdb_RECO
ORA-19809: limit exceeded for recovery files
ORA-19804: cannot reclaim 1062207488 bytes disk space from 424128020480 limit
##########################
# Error Occurred
##########################
Error occurs randomly when RFS process tries to receive redo from Primary.
##########################
## Cause
##########################
Problem is actually caused by the parallel archive options set for Standby database destination in Primary Database.
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_2 string service=stdbydb OPTIONAL ARCH SYNC NOAFFIRM REOPEN=30
MAX_CONNECTIONS=2
NET_TIMEOUT=15 VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=stdbydb
Eventhough we have log_archive_max_processes was 4 which can very well handle the parallel archiving option, error occurs.
I have also tried setting upto 10 archiver process which didnt help.
##########################
## Solution
##########################
As the problem is mainly because of the Parallel archiving option, i have modified to use only one connection for archiving by setting
MAX_CONNECTIONS = 1
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_2 string service=stdbydb OPTIONAL ARCH SYNC NOAFFIRM REOPEN=30
MAX_CONNECTIONS = 1
NET_TIMEOUT=15 VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=stdbydb
- Modify MAX_CONNECTIONS=1 in log_archive_dest_2
- Defer and Enable log_archive_dest_state_2
This helped me and the false errors stopped recording in my primary and standby alert logs.