One of our Oracle 12c Cloud Control Management Agent has started producing Errors like below.
##########################
## Error
##########################
##########################
# Error Occurred
##########################
Oracle Cloud Control Managment Agent of 12.1.0.1.0 version has stopped monitoring the targets and started producing the alerts.
**************************************** Step By Step Analysis ******************************************************
#########################################
# 1) Check Agent Status
#########################################
cd /u01/app/oracle/product/agent12c/core/12.1.0.1.0/bin
./emctl status agent
[oracle@host01 bin]$ ./emctl status agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.1.0
OMS Version : 12.1.0.1.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oracle/product/agent12c/agent_inst
Agent Binaries : /u01/app/oracle/product/agent12c/core/12.1.0.1.0
Agent Process ID : 34527
Parent Process ID : 34343
Agent URL : https://host01:3872/emd/main/
Repository URL : https://oem:4901/empbs/upload
Started at : 2014-01-10 21:42:00
Started by user : oracle
Last Reload : (none)
Last successful upload : 2014-01-23 11:55:26
Last attempted upload : 2014-01-23 11:55:26
Total Megabytes of XML files uploaded so far : 359.89
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 98.90%
Collection Status : Collections enabled
Last attempted heartbeat to OMS : 2014-01-23 11:55:24
Last successful heartbeat to OMS : 2014-01-23 11:55:24
---------------------------------------------------------------
Agent is Running and Ready
./emctl upload agent
[oracle@host01 bin]$ ./emctl upload agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully
[oracle@host01 bin]$
=====================================================================================================================
Agent is Running good and upload of Agent is also working good but it has stopped Monitoring the Targets.
=====================================================================================================================
#########################################
# 2) Reason for Failure
#########################################
This error seems to be a Bug 14083191 for which a Patch 6895422 is Available and needs to be applied to Management Agent ORACLE_HOME
=====================================================================================================================
#########################################
# 3) Check Patch Details
#########################################
./opatch lsinventory
[oracle@host01 OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.1.0.9.2
Copyright (c) 2011, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/agent12c/core/12.1.0.1.0
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/agent12c/core/12.1.0.1.0/oraInst.loc
OPatch version : 11.1.0.9.2
OUI version : 11.1.0.9.0
Log file location : /u01/app/oracle/product/agent12c/core/12.1.0.1.0/cfgtoollogs/opatch/opatch2014-01-23_11-19-48AM_1.log
Lsinventory Output file location : /u01/app/oracle/product/agent12c/core/12.1.0.1.0/cfgtoollogs/opatch/lsinv/lsinventory2014-01-23_11-19-48AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
EM Platform (Agent) 12.1.0.1.0
There are 1 products installed in this Oracle Home.
Interim patches (1) :
Patch 6895422 : applied on Sat Jan 11 03:15:36 UTC 2014
Unique Patch ID: 14696320
Created on 6 Apr 2012, 00:13:27 hrs PST8PDT
Bugs fixed:
6895422
--------------------------------------------------------------------------------
OPatch succeeded.
[oracle@host01 OPatch]$
=====================================================================================================================
From Above output we can see like Patch 6895422 is already applied but still we are getting Errors and Agent stopped Monitoring.
=====================================================================================================================
#########################################
# 4) Root Cause
#########################################
From 3) Output we can see that Patch 6895422 applied on 11-Jan-2014.
Patch 6895422 : applied on Sat Jan 11 03:15:36 UTC 2014
From 2) Output we can see that Agent has Last Started on 10-Jan-2014
Started at : 2014-01-10 21:42:00
=====================================================================================================================
##########################
## Solution
##########################
To Fix the Error, the proposed solution is to Apply Patch 6895422.
In our case its already applied, but we found out that Agent is not Re-started after patch was applied to its HOME.
Agent needs to be bounced after Patch has been applied to it. A Bounce of Agent fixed the Issue.
./emctl stop agent
./emctl start agent
=====================================================================================================================
Comments Are Always welcome
=====================================================================================================================
