Sunday, January 19, 2014

12c Database Status pending ( Unknown )



We have recently moved from 11g Grid control to 12c cloud services. The way we resolve the targets which were in pending status from 11g to 12c is different. Please take a look into the simple steps.

**************************************** Step By Step Procedure ******************************************************

#########################################
# 1) Check 12c Agent Status
#########################################

Set 12c Agent environment or go to the $AGENT_HOME and check the status of Agent.

[oracle@host01 bin]$ pwd
/u01/app/oracle/product/agent12c/core/12.1.0.1.0/bin

./emctl status agent


[oracle@host01 bin]$ ./emctl status agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.1.0
OMS Version       : 12.1.0.1.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/app/oracle/product/agent12c/agent_inst
Agent Binaries    : /u01/app/oracle/product/agent12c/core/12.1.0.1.0
Agent Process ID  : 6790
Parent Process ID : 6688
Agent URL         : https://host01.dt.stepintooracledba.com:3873/emd/main/
Repository URL    : https://host02.stepintooracledba.com:4900/empbs/upload
Started at        : 2013-12-16 12:06:35
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2014-01-15 21:51:45
Last attempted upload                        : 2014-01-15 21:51:45
Total Megabytes of XML files uploaded so far : 158.69
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 35.93%
Collection Status                            : Collections enabled
Last attempted heartbeat to OMS              : 2014-01-15 21:51:49
Last successful heartbeat to OMS             : 2014-01-15 21:51:49
---------------------------------------------------------------
Agent is Running and Ready

=====================================================================================================================
 From above output check the below lines,

Last successful upload                       : This should be the latest timeframe
Number of XML files pending upload           : Check Any Files needs to be Uploaded
Collection Status                            : Make Sure Collection is enabled
Last successful heartbeat to OMS             : This should be the latest timeframe

=====================================================================================================================
 If Agent is not Running then start the agent
=====================================================================================================================

#########################################
# 2) Upload Agent
#########################################

Upload Agent and check if its uploading without any issues.

./emctl upload agent

[oracle@host01 bin]$ ./emctl upload agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

./emctl status agent

[oracle@host01 bin]$ ./emctl status agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.1.0
OMS Version       : 12.1.0.1.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/app/oracle/product/agent12c/agent_inst
Agent Binaries    : /u01/app/oracle/product/agent12c/core/12.1.0.1.0
Agent Process ID  : 6790
Parent Process ID : 6688
Agent URL         : https://host01.dt.stepintooracledba.com:3873/emd/main/
Repository URL    : https://host02.stepintooracledba.com:4900/empbs/upload
Started at        : 2013-12-16 12:06:35
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2014-01-15 21:53:32
Last attempted upload                        : 2014-01-15 21:53:32
Total Megabytes of XML files uploaded so far : 158.7
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 35.93%
Collection Status                            : Collections enabled
Last attempted heartbeat to OMS              : 2014-01-15 21:52:49
Last successful heartbeat to OMS             : 2014-01-15 21:52:49
---------------------------------------------------------------
Agent is Running and Ready

=====================================================================================================================

#########################################
# 3) Check Agent can Evaluate Response Metric
#########################################

cd /u01/orabase/12cagent/core/12.1.0.1.0/bin

./emctl getmetric agent (database_name_as_shown_in_console),oracle_database,Response

./emctl getmetric agent db01.stepintooracledba.com,oracle_database,Response

[oracle@host01 bin]$ ./emctl getmetric agent db01.stepintooracledba.com,oracle_database,Response
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
Status,State,oraerr,Archiver,DatabaseStatus,ActiveState
1,OPEN,,STOPPED,ACTIVE,NORMAL
 1 = database is up

=====================================================================================================================

#########################################
# 4) Check Targets in Suspension State
#########################################

First we will check get the Agent Name using below command.

./emctl config agent listtargets | grep oracle_emd

[oracle@host01 bin]$ ./emctl config agent listtargets | grep oracle_emd
[host01.dt.stepintooracledba.com:3873, oracle_emd]

Now we will see the targets which are in Suspension State in this Host.

./emctl getmetric agent host01.dt.stepintooracledba.com:3873,oracle_emd,z#Tasks_in_Suspension

[oracle@host01 bin]$ ./emctl getmetric agent host01.dt.stepintooracledba.com:3873,oracle_emd,z#Tasks_in_Suspension
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
TaskName,SuspendedUntil,Points
oracle_listener:LISTENER_host01.dt.stepintooracledba.com:Response,292278994-08-17 02:12:55,2147483647
oracle_emd:host01.dt.stepintooracledba.com:3873:z#AgentStat_PingStats,292278994-08-17 02:12:55,2147483647
oracle_emd:host01.dt.stepintooracledba.com:3873:z#CurrentRequests,292278994-08-17 02:12:55,2147483647
oracle_emd:host01.dt.stepintooracledba.com:3873:z#AgentStat_OutStandingSeverities,292278994-08-17 02:12:55,2147483647
oracle_database:db01.stepintooracledba.com:activity_pending,292278994-08-17 02:12:55,2147483647
oracle_database:db01.stepintooracledba.com:dataguard_11gR2,292278994-08-17 02:12:55,2147483647
oracle_listener:LISdb01_host01.dt.stepintooracledba.com:Response,292278994-08-17 02:12:55,2147483647
oracle_emd:host01.dt.stepintooracledba.com:3873:z#LastPing,292278994-08-17 02:12:55,2147483647
oracle_database:db01.stepintooracledba.com:observer_11g,292278994-08-17 02:12:55,2147483647
oracle_emd:host01.dt.stepintooracledba.com:3873:z#AgentStat_UploadSendingQueueStatistics,292278994-08-17 02:12:55,2147483647
oracle_database:db01.stepintooracledba.com:Response,292278994-08-17 02:12:55,2147483647
oracle_emd:host01.dt.stepintooracledba.com:3873:z#UploadSystemStats,292278994-08-17 02:12:55,2147483647
[oracle@host01 bin]$


=====================================================================================================================

#########################################
# 5) Check Current Status of the Database Target
#########################################

./emctl status agent target db01.stepintooracledba.com,oracle_database | grep Response

[oracle@host01 bin]$ ./emctl status agent target db01.stepintooracledba.com,oracle_database | grep Response
Response      State                   n/a             UNDEFINED       Mon Dec 16 12:06:40 EST 2013
Response      Status                  n/a             UNDEFINED       Mon Dec 16 12:06:40 EST 2013
[oracle@host01 bin]$

=====================================================================================================================
 Response Status Shows as UNDEFINED
=====================================================================================================================

#########################################
# 6) Check Response Metric
#########################################

We will check whether response metric is scheduled for this database or not.

./emctl status agent scheduler | grep Response

[oracle@host01 bin]$ ./emctl status agent scheduler | grep Response
2014-01-15 22:07:30.370 : osm_instance:+ASM_host01.dt.stepintooracledba.com:Response
[oracle@host01 bin]$

=====================================================================================================================

#########################################
# 7) Collect the Response Metric Manually
#########################################

From Step 6, we can see that Response Metric is not scheduled for the database.

Running the response metric manually, causes the agent to perfom an immediate collection of the metric:-


./emctl control agent runCollection db01.stepintooracledba.com:oracle_database Response

[oracle@host01 bin]$ ./emctl control agent runCollection db01.stepintooracledba.com:oracle_database Response
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully

[oracle@host01 bin]$

If no errors are returned by running the metric manually, refresh the console (by clicking on the 'refresh data' arrow at the right hand side of the top of

the console).

Now the Database Status is showing as Up & Running....

=====================================================================================================================
 Comments Are Always welcome
=====================================================================================================================

2 comments:

Anonymous said...

Clearly written post.
Thanks!
Shivaswamy

Sri said...

Thanks much Shivaswamy..