Configure SSH Manually for RAC 4 node Cluster

Configure SSH Manually for RAC 4 node Cluster

Before 11gR2, Grid Infrastructure Installation requires SSH to be configured. But from 11gR2 onwards, using Oracle Universal Installer the ssh setup can be done by using the  'SSH Connectivity' button.

And also to run cluvfy, SSH configuration is needed. Below are the steps for manual SSH configuration.

**************************************** Step By Step Procedure ******************************************************

#########################################
# 1) Create Necessary Folders & Permissions
#########################################

Login to ORACLE user HOME and create a directory called .ssh and permissions should be given like below.

$ cd $HOME

$ mkdir .ssh

$ chmod 700 .ssh

cd /u01/home/oracle/.ssh

=====================================================================================================================
Do Step 1 in all the 4 nodes.
=====================================================================================================================

#########################################
# 2) Create Keygen
#########################################

On Node 1 ( hostU13 )
---------------------------

no_sid @ hostu13:/u01/home/oracle/.ssh
> ls -ltr
total 4
-rw-r--r-- 1 oracle oinstall 407 Dec 18 05:40 known_hosts

/usr/bin/ssh-keygen -t rsa

no_sid @ hostu13:/u01/home/oracle/.ssh
> /usr/bin/ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/u01/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /u01/home/oracle/.ssh/id_rsa.
Your public key has been saved in /u01/home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
af:75:72:de:a8:1c:1e:0a:11:64:9e:5b:cd:0e:30:b2 oracle@hostu13

Note :
---------

Accept the default location for the key file, Enter and confirm a passphrase. (you can also press enter twice)

no_sid @ hostu13:/u01/home/oracle/.ssh
> ls
id_rsa  id_rsa.pub  known_hosts

no_sid @ hostu13:/u01/home/oracle/.ssh
> ls -altr
total 20
-rw-r--r-- 1 oracle oinstall  407 Dec 18 05:40 known_hosts
drwxr-xr-x 4 oracle dba      4096 Dec 18 21:58 ..
-rw-r--r-- 1 oracle oinstall  400 Dec 18 22:02 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Dec 18 22:02 id_rsa
drwx------ 2 oracle oinstall 4096 Dec 18 22:02 .
no_sid @ hostu13:/u01/home/oracle/.ssh

=====================================================================================================================
Now we have got 2 files generated "id_rsa.pub" & "id_rsa"
Do Step 2 in all the 4 nodes.
=====================================================================================================================

#########################################
# 3) Create Authorization Keys
#########################################

Now we will create an authorization key file from "id_rsa.pub" file in all the nodes.

cat id_rsa.pub >> authorized_keys.$HOSTNAME

no_sid @ hostu13:/u01/home/oracle/.ssh
> cat id_rsa.pub >> authorized_keys.$HOSTNAME
no_sid @ hostu13:/u01/home/oracle/.ssh
> ls -ltr
total 16
-rw-r--r-- 1 oracle oinstall  408 Dec 18 06:32 known_hosts
-rw-r--r-- 1 oracle oinstall  400 Dec 18 23:10 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Dec 18 23:10 id_rsa
-rw-r--r-- 1 oracle oinstall  400 Dec 19 07:44 authorized_keys.hostu13
no_sid @ hostu13:/u01/home/oracle/.ssh

Copy All the authorized_keys.$HOSTNAME from all the 4 nodes to Node 1, Once copied it will be like below,

no_sid @ hostu13:/u01/home/oracle/.ssh
> ls -ltr
total 28
-rw-r--r-- 1 oracle oinstall  400 Dec 18 22:02 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Dec 18 22:02 id_rsa
-rw-r--r-- 1 oracle oinstall 1631 Dec 19 07:49 known_hosts
-rwxrwxrwx 1 oracle oinstall  400 Dec 18 22:06 authorized_keys.hostu13
-rw-r--r-- 1 oracle oinstall  400 Dec 19 07:48 authorized_keys.hostu14
-rw-r--r-- 1 oracle oinstall  400 Dec 19 07:49 authorized_keys.hostu15
-rw-r--r-- 1 oracle oinstall  400 Dec 19 07:49 authorized_keys.hostu16
no_sid @ hostu13:/u01/home/oracle/.ssh
>

Now Create  a file called "authorized_keys" which will hold all the 4 nodes key files.

cat *.host* >> authorized_keys

no_sid @ hostu13:/u01/home/oracle/.ssh
> cat *.host* >> authorized_keys
no_sid @ hostu13:/u01/home/oracle/.ssh
> ls -ltr
total 32
-rw-r--r-- 1 oracle oinstall  400 Dec 18 22:02 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Dec 18 22:02 id_rsa
-rw-r--r-- 1 oracle oinstall 1631 Dec 19 07:49 known_hosts
-rwxrwxrwx 1 oracle oinstall  400 Dec 18 22:06 authorized_keys.hostu13
-rw-r--r-- 1 oracle oinstall  400 Dec 19 07:48 authorized_keys.hostu14
-rw-r--r-- 1 oracle oinstall  400 Dec 19 07:49 authorized_keys.hostu15
-rw-r--r-- 1 oracle oinstall  400 Dec 19 07:49 authorized_keys.hostu16
-rw-r--r-- 1 oracle oinstall 1600 Dec 19 07:51 authorized_keys
no_sid @ hostu13:/u01/home/oracle/.ssh

Now FTP the authorized_keys file to all the nodes to /u01/home/oracle/.ssh location

=====================================================================================================================

#########################################
# 4) Test SSH
#########################################

Now run the below commands, from all the nodes. Initially it may prompt like below, Give "YES". "known_hosts" file will be added with the RSA key values.

Below commands should return date.

ssh hostu13 date
ssh hostu14 date
ssh hostu15 date
ssh hostu16 date

no_sid @ hostu13:/u01/home/oracle/.ssh
> ssh hostu13 date
The authenticity of host 'hostu13 (192.168.29.205)' can't be established.
RSA key fingerprint is e6:e2:24:ad:1f:bf:88:0e:f2:6d:09:2c:5d:7b:f2:0e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hostu13,192.168.29.205' (RSA) to the list of known hosts.
Thu Dec 19 07:57:18 CST 2013
no_sid @ hostu13:/u01/home/oracle/.ssh
>
no_sid @ hostu13:/u01/home/oracle/.ssh
> ssh hostu14 date
Thu Dec 19 07:57:30 CST 2013
no_sid @ hostu13:/u01/home/oracle/.ssh
>
no_sid @ hostu13:/u01/home/oracle/.ssh
> ssh hostu15 date
Thu Dec 19 07:57:37 CST 2013
no_sid @ hostu13:/u01/home/oracle/.ssh
>
no_sid @ hostu13:/u01/home/oracle/.ssh
> ssh hostu16 date
Thu Dec 19 07:57:44 CST 2013
no_sid @ hostu13:/u01/home/oracle/.ssh
>
no_sid @ hostu13:/u01/home/oracle/.ssh
>

=====================================================================================================================

#########################################
# 5) Sample "known_hosts"
#########################################

After ssh to hostu13, and YES was provided, it added an entry.

no_sid @ hostu15:/u01/home/oracle/.ssh
> more known_hosts
hostu13,192.168.29.205 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvLbffewHf7FSEB7FuRZUpq8Swk8Yy9uUgGdbUiTsF2firlYi7lfkDywlAHvtpixxhmkyTHV92LFk0Dwr0HhYTYSBBxlNTBWPr
4sfjvcIweWSlF/HMoTerG4QE7dwbRxYej+mDqVxs4Zujh6verF1GMKkrsG8tMVfnVQUxeRj1q4YifY2nZR6vG3JpVv+4aV8KG6ViCFORzdR6AKD20YtQ7sqe3O1Dg7gqUo5UA5/XCdHUgcKIw4lza6WzRVdj6QZQ
E+aFl/f3TKgbI9jqhIwiBqtPPw69VeYGdcF0RpwhAHxvvVR5U34BUeOs+TuE/KpKe4MEGMZJHeP1Wu6rc3inw==
no_sid @ hostu15:/u01/home/oracle/.ssh
>

Now, After ssh to hostu14, it has entry of hostU13 & hostU14

no_sid @ hostu15:/u01/home/oracle/.ssh
> more known_hosts
hostu13,192.168.29.205 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvLbffewHf7FSEB7FuRZUpq8Swk8Yy9uUgGdbUiTsF2firlYi7lfkDywlAHvtpixxhmkyTHV92LFk0Dwr0HhYTYSBBxlNTBWPr
4sfjvcIweWSlF/HMoTerG4QE7dwbRxYej+mDqVxs4Zujh6verF1GMKkrsG8tMVfnVQUxeRj1q4YifY2nZR6vG3JpVv+4aV8KG6ViCFORzdR6AKD20YtQ7sqe3O1Dg7gqUo5UA5/XCdHUgcKIw4lza6WzRVdj6QZQ
E+aFl/f3TKgbI9jqhIwiBqtPPw69VeYGdcF0RpwhAHxvvVR5U34BUeOs+TuE/KpKe4MEGMZJHeP1Wu6rc3inw==
hostu14,192.168.29.206 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwyx+PEkivwKs1RGoLiBNbgKU0KloyKdkL2HVouWZzRr36veV+SDYNJkR7ek8kcNdakVlV20aKzfqlaOhfOq1GfI7Hd0Z3epUh
Bbs/YkdaKdPwtRpeB7iD/kBSTkpiuXbcS06QjtQ5NLlsGOLrviProDucMjQIHg1wrU5PE12fQ4NHFBGSArSk/d4kQUxyYotPcjbACjKmdnY/TpTuRCy5eyIXDiDHN/opXX+0j+rH9A4j5eIBe1G9cZcQda/57BBL
Z/ESIGkZ444YqbwT5dxTlrpaKPtyK12UCeLw9xoSAOcnQUm/4ICj96M95kMeMezUBSjlZIN3S79/9rVpOTUtw==
no_sid @ hostu15:/u01/home/oracle/.ssh
>

=====================================================================================================================

11.2.0.1 Grid Infrastructure De-Install ( ClusterWare ) using De-Install Utility

##########################
## Objective
##########################

11.2.0.1 Grid Infrastructure De-Install ( ClusterWare ) using De-Install Utility

Grid Infrastructure Version : 11.2.0.1.0
Grid Infrastructure Home : /u01/app/11.2/grid
Nodes in the Cluster : hostu13, hostu14, hostu15, hostu16

********************************************** Step By Step Procedure ******************************************

#########################################
# 1) Go To $ORACLE_HOME (GI) and Start the De-Install Utility
#########################################

+ASM1 @ hostu13:/tmp
> echo $ORACLE_HOME
/u01/app/11.2/grid
+ASM1 @ hostu13:/tmp


cd $ORACLE_HOME/deinstall

+ASM1 @ hostu13:/u01/app/11.2/grid/deinstall
> ls -ltr
total 68
-rwxr-xr-x 1 oracle oinstall 31544 Mar 31  2009 sshUserSetup.sh
-rw-r--r-- 1 oracle oinstall  3154 Jul 13  2009 deinstall.xml
drwxr-xr-x 2 oracle oinstall  4096 Dec 19 10:56 response
drwxr-xr-x 2 oracle oinstall  4096 Dec 19 10:56 jlib
-rwxr-xr-x 1 oracle oinstall 20663 Dec 19 10:57 deinstall
+ASM1 @ hostu13:/u01/app/11.2/grid/deinstall


+ASM1 @ hostu13:/u01/app/11.2/grid/deinstall
> script /tmp/grid_deinstall.log
Script started, file is /tmp/grid_deinstall.log
+ASM1 @ hostu13:/u01/app/11.2/grid/deinstall

+ASM1 @ hostu13:/u01/app/11.2/grid/deinstall
> ./deinstall

=====================================================================================================================

#########################################
# 2) Options To De-Install
#########################################


Click  "De-Install Log" to find the Full De-Install Log

Press " Enter " when it asks like below ( Default is No )

 If you want to retain the existing diskgroups or if any of the information detected is incorrect, you can modify by entering 'y'. Do you  want to modify above information (y|n) [n]:

Enter " y " when it asks like below ( Default is No )

ASM instance will be de-configured from this Oracle home. Do you want to continue (y - yes, n - no)? [n]: y

Run the commands as root user in a sequential basis, ( hostu16, hostu15, hostu14, hostu13 )

Run the following command as the root user or the administrator on node "hostu16".

Rootcrs.pl Script Execution Log

Note :

I have tried running the commands with sudo options which didn't work. So run the above commands Root user.

Once i de-installed, Grid Infrastructure, ORACLE_HOME binaries are not removed.

To Remove ORACLE_HOME and its binary files, please check the below link.

=====================================================================================================================

#########################################
# 3) De-Install Log
#########################################

Back To Step 2

+ASM1 @ hostu13:/u01/app/11.2/grid/deinstall
> script /tmp/grid_deinstall.log
Script started, file is /tmp/grid_deinstall.log
+ASM1 @ hostu13:/u01/app/11.2/grid/deinstall
> ./deinstall
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /tmp/deinstall2013-12-20_09-10-20-AM/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############


######################## CHECK OPERATION START ########################
Install check configuration START


Checking for existence of the Oracle home location /u01/app/11.2/grid
Oracle Home type selected for de-install is: CRS
Oracle Base selected for de-install is: /u01/app/oracle
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2/grid
The following nodes are part of this cluster: hostu13,hostu14,hostu15,hostu16

Install check configuration END

Traces log file: /tmp/deinstall2013-12-20_09-10-20-AM/logs//crsdc.log

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2013-12-20_09-10-20-AM/logs/netdc_check306264010968614796.log

Specify all RAC listeners that are to be de-configured [LISTENER]:

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /tmp/deinstall2013-12-20_09-10-20-AM/logs/asmcadc_check8042716245868975026.log

Automatic Storage Management (ASM) instance is detected in this Oracle home /u01/app/11.2/grid.
ASM Diagnostic Destination : /u01/app/oracle
ASM Diskgroups : +OCR_VOTE
Diskgroups will be dropped
De-configuring ASM will drop all the diskgroups and it's contents at cleanup time. This will affect all of the databases and ACFS that use this ASM instance(s).
 If you want to retain the existing diskgroups or if any of the information detected is incorrect, you can modify by entering 'y'. Do you  want to modify above

information (y|n) [n]:

######################### CHECK OPERATION END #########################


####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/11.2/grid
The cluster node(s) on which the Oracle home exists are: (Please input nodes seperated by ",", eg: node1,node2,...)hostu13,hostu14,hostu15,hostu16
Oracle Home selected for de-install is: /u01/app/11.2/grid
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
Following RAC listener(s) will be de-configured: LISTENER
ASM instance will be de-configured from this Oracle home
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/tmp/deinstall2013-12-20_09-10-20-AM/logs/deinstall_deconfig2013-12-20_09-10-36-AM.out'
Any error messages from this session will be written to: '/tmp/deinstall2013-12-20_09-10-20-AM/logs/deinstall_deconfig2013-12-20_09-10-36-AM.err'

######################## CLEAN OPERATION START ########################
ASM de-configuration trace file location: /tmp/deinstall2013-12-20_09-10-20-AM/logs/asmcadc_clean8089613939205285348.log
ASM Clean Configuration START
ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2013-12-20_09-10-20-AM/logs/netdc_clean5072791991986593696.log

De-configuring RAC listener(s): LISTENER

De-configuring listener: LISTENER
    Stopping listener: LISTENER
    Listener stopped successfully.
    Unregistering listener: LISTENER
    Listener unregistered successfully.
Listener de-configured successfully.

De-configuring Naming Methods configuration file on all nodes...
Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file on all nodes...
Local Net Service Names configuration file de-configured successfully.

De-configuring Directory Usage configuration file on all nodes...
Directory Usage configuration file de-configured successfully.

De-configuring backup files on all nodes...
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END


---------------------------------------->

Run the following command as the root user or the administrator on node "hostu16".

/tmp/deinstall2013-12-20_09-10-20-AM/perl/bin/perl -I/tmp/deinstall2013-12-20_09-10-20-AM/perl/lib -I/tmp/deinstall2013-12-20_09-10-20-AM/crs/install

/tmp/deinstall2013-12-20_09-10-20-AM/crs/install/rootcrs.pl -force  -delete -paramfile /tmp/deinstall2013-12-20_09-10-20-

AM/response/deinstall_Ora11g_gridinfrahome1.rsp

Run the following command as the root user or the administrator on node "hostu15".

/tmp/deinstall2013-12-20_09-10-20-AM/perl/bin/perl -I/tmp/deinstall2013-12-20_09-10-20-AM/perl/lib -I/tmp/deinstall2013-12-20_09-10-20-AM/crs/install

/tmp/deinstall2013-12-20_09-10-20-AM/crs/install/rootcrs.pl -force  -delete -paramfile /tmp/deinstall2013-12-20_09-10-20-

AM/response/deinstall_Ora11g_gridinfrahome1.rsp

Run the following command as the root user or the administrator on node "hostu14".

/tmp/deinstall2013-12-20_09-10-20-AM/perl/bin/perl -I/tmp/deinstall2013-12-20_09-10-20-AM/perl/lib -I/tmp/deinstall2013-12-20_09-10-20-AM/crs/install

/tmp/deinstall2013-12-20_09-10-20-AM/crs/install/rootcrs.pl -force  -delete -paramfile /tmp/deinstall2013-12-20_09-10-20-

AM/response/deinstall_Ora11g_gridinfrahome1.rsp

Run the following command as the root user or the administrator on node "hostu13".

/tmp/deinstall2013-12-20_09-10-20-AM/perl/bin/perl -I/tmp/deinstall2013-12-20_09-10-20-AM/perl/lib -I/tmp/deinstall2013-12-20_09-10-20-AM/crs/install

/tmp/deinstall2013-12-20_09-10-20-AM/crs/install/rootcrs.pl -force  -delete -paramfile /tmp/deinstall2013-12-20_09-10-20-

AM/response/deinstall_Ora11g_gridinfrahome1.rsp -lastnode

Press Enter after you finish running the above commands

<---------------------------------------- p="">
Oracle Universal Installer clean START

Oracle Universal Installer cleanup completed with errors.

Oracle Universal Installer clean END


Oracle install clean START

Clean install operation removing temporary directory '/tmp/install' on node 'hostu13'
Clean install operation removing temporary directory '/tmp/install' on node 'hostu14'
Clean install operation removing temporary directory '/tmp/install' on node 'hostu15'
Clean install operation removing temporary directory '/tmp/install' on node 'hostu16'

Oracle install clean END


######################### CLEAN OPERATION END #########################


####################### CLEAN OPERATION SUMMARY #######################
ASM instance was de-configured successfully from the Oracle home
Following RAC listener(s) were de-configured successfully: LISTENER
Oracle Clusterware is stopped and successfully de-configured on node "hostu16"
Oracle Clusterware is stopped and successfully de-configured on node "hostu15"
Oracle Clusterware is stopped and successfully de-configured on node "hostu14"
Oracle Clusterware is stopped and successfully de-configured on node "hostu13"
Oracle Clusterware is stopped and de-configured successfully.
Oracle Universal Installer cleanup completed with errors.

Oracle install successfully cleaned up the temporary directories.
#######################################################################


############# ORACLE DEINSTALL & DECONFIG TOOL END #############

+ASM1 @ hostu13:/u01/app/11.2/grid/deinstall
> exit
Script done, file is /tmp/grid_deinstall.log

=====================================================================================================================

#########################################
# 4) Rootcrs.pl Script Execution Log
#########################################

Back To Step 2 =====================================================================================================================
hostu16
=====================================================================================================================
2013-12-20 10:02:30: Parsing the host name
2013-12-20 10:02:30: Checking for super user privileges
2013-12-20 10:02:30: User has super user privileges
Using configuration parameter file: /tmp/deinstall2013-12-20_09-10-20-AM/response/deinstall_Ora11g_gridinfrahome1.rsp
VIP exists.:hostu13
VIP exists.: /hostu13-vip/192.20.102.198/255.255.254.0/bond0
VIP exists.:hostu14
VIP exists.: /hostu14-vip/192.20.102.199/255.255.254.0/bond0
VIP exists.:hostu15
VIP exists.: /hostu15-vip/192.20.102.200/255.255.254.0/bond0
VIP exists.:hostu16
VIP exists.: /hostu16-vip/192.20.102.201/255.255.254.0/bond0
GSD exists.
ONS daemon exists. Local port 6100, remote port 6200
eONS daemon exists. Multicast port 21337, multicast IP address 234.99.125.214, listening port 2016
ACFS-9200: Supported
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'hostu16'
CRS-2677: Stop of 'ora.registry.acfs' on 'hostu16' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hostu16'
CRS-2673: Attempting to stop 'ora.crsd' on 'hostu16'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'hostu16'
CRS-2673: Attempting to stop 'ora.asm' on 'hostu16'
CRS-2677: Stop of 'ora.asm' on 'hostu16' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'hostu16' has completed
CRS-2677: Stop of 'ora.crsd' on 'hostu16' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'hostu16'
CRS-2673: Attempting to stop 'ora.ctssd' on 'hostu16'
CRS-2673: Attempting to stop 'ora.evmd' on 'hostu16'
CRS-2673: Attempting to stop 'ora.asm' on 'hostu16'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'hostu16'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'hostu16'
CRS-2677: Stop of 'ora.cssdmonitor' on 'hostu16' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'hostu16' succeeded
CRS-2677: Stop of 'ora.evmd' on 'hostu16' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'hostu16' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'hostu16' succeeded
CRS-2677: Stop of 'ora.asm' on 'hostu16' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'hostu16'
CRS-2677: Stop of 'ora.cssd' on 'hostu16' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'hostu16'
CRS-2673: Attempting to stop 'ora.diskmon' on 'hostu16'
CRS-2677: Stop of 'ora.gpnpd' on 'hostu16' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'hostu16'
CRS-2677: Stop of 'ora.diskmon' on 'hostu16' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'hostu16' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'hostu16' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node

=====================================================================================================================
hostu15
=====================================================================================================================

2013-12-20 10:04:22: Parsing the host name
2013-12-20 10:04:22: Checking for super user privileges
2013-12-20 10:04:22: User has super user privileges
Using configuration parameter file: /tmp/deinstall2013-12-20_09-10-20-AM/response/deinstall_Ora11g_gridinfrahome1.rsp
VIP exists.:hostu13
VIP exists.: /hostu13-vip/192.20.102.198/255.255.254.0/bond0
VIP exists.:hostu14
VIP exists.: /hostu14-vip/192.20.102.199/255.255.254.0/bond0
VIP exists.:hostu15
VIP exists.: /hostu15-vip/192.20.102.200/255.255.254.0/bond0
GSD exists.
ONS daemon exists. Local port 6100, remote port 6200
eONS daemon exists. Multicast port 21337, multicast IP address 234.99.125.214, listening port 2016
ACFS-9200: Supported
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'hostu15'
CRS-2677: Stop of 'ora.registry.acfs' on 'hostu15' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hostu15'
CRS-2673: Attempting to stop 'ora.crsd' on 'hostu15'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'hostu15'
CRS-2673: Attempting to stop 'ora.asm' on 'hostu15'
CRS-2677: Stop of 'ora.asm' on 'hostu15' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'hostu15' has completed
CRS-2677: Stop of 'ora.crsd' on 'hostu15' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'hostu15'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'hostu15'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'hostu15'
CRS-2673: Attempting to stop 'ora.ctssd' on 'hostu15'
CRS-2673: Attempting to stop 'ora.evmd' on 'hostu15'
CRS-2673: Attempting to stop 'ora.asm' on 'hostu15'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'hostu15'
CRS-2677: Stop of 'ora.cssdmonitor' on 'hostu15' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'hostu15' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'hostu15' succeeded
CRS-2677: Stop of 'ora.evmd' on 'hostu15' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'hostu15' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'hostu15' succeeded
CRS-2677: Stop of 'ora.asm' on 'hostu15' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'hostu15'
CRS-2677: Stop of 'ora.cssd' on 'hostu15' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'hostu15'
CRS-2673: Attempting to stop 'ora.gipcd' on 'hostu15'
CRS-2677: Stop of 'ora.gipcd' on 'hostu15' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'hostu15' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'hostu15' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node

=====================================================================================================================
hostu14
=====================================================================================================================

2013-12-20 10:06:02: Parsing the host name
2013-12-20 10:06:02: Checking for super user privileges
2013-12-20 10:06:02: User has super user privileges
Using configuration parameter file: /tmp/deinstall2013-12-20_09-10-20-AM/response/deinstall_Ora11g_gridinfrahome1.rsp
VIP exists.:hostu13
VIP exists.: /hostu13-vip/192.20.102.198/255.255.254.0/bond0
VIP exists.:hostu14
VIP exists.: /hostu14-vip/192.20.102.199/255.255.254.0/bond0
GSD exists.
ONS daemon exists. Local port 6100, remote port 6200
eONS daemon exists. Multicast port 21337, multicast IP address 234.99.125.214, listening port 2016
ACFS-9200: Supported
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'hostu14'
CRS-2677: Stop of 'ora.registry.acfs' on 'hostu14' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hostu14'
CRS-2673: Attempting to stop 'ora.crsd' on 'hostu14'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'hostu14'
CRS-2673: Attempting to stop 'ora.asm' on 'hostu14'
CRS-2677: Stop of 'ora.asm' on 'hostu14' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'hostu14' has completed
CRS-2677: Stop of 'ora.crsd' on 'hostu14' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'hostu14'
CRS-2673: Attempting to stop 'ora.ctssd' on 'hostu14'
CRS-2673: Attempting to stop 'ora.evmd' on 'hostu14'
CRS-2673: Attempting to stop 'ora.asm' on 'hostu14'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'hostu14'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'hostu14'
CRS-2677: Stop of 'ora.cssdmonitor' on 'hostu14' succeeded
CRS-2677: Stop of 'ora.evmd' on 'hostu14' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'hostu14' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'hostu14' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'hostu14' succeeded
CRS-2677: Stop of 'ora.asm' on 'hostu14' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'hostu14'
CRS-2677: Stop of 'ora.cssd' on 'hostu14' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'hostu14'
CRS-2673: Attempting to stop 'ora.diskmon' on 'hostu14'
CRS-2677: Stop of 'ora.gpnpd' on 'hostu14' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'hostu14'
CRS-2677: Stop of 'ora.gipcd' on 'hostu14' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'hostu14' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'hostu14' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node

=====================================================================================================================
hostu13
=====================================================================================================================

VIP exists.:hostu13
VIP exists.: /hostu13-vip/192.20.102.198/255.255.254.0/bond0
GSD exists.
ONS daemon exists. Local port 6100, remote port 6200
eONS daemon exists. Multicast port 21337, multicast IP address 234.99.125.214, listening port 2016
ACFS-9200: Supported
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'hostu13'
CRS-2677: Stop of 'ora.registry.acfs' on 'hostu13' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'hostu13'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'hostu13'
CRS-2673: Attempting to stop 'ora.asm' on 'hostu13'
CRS-2677: Stop of 'ora.asm' on 'hostu13' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'hostu13' has completed
CRS-2677: Stop of 'ora.crsd' on 'hostu13' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'hostu13'
CRS-2673: Attempting to stop 'ora.ctssd' on 'hostu13'
CRS-2673: Attempting to stop 'ora.evmd' on 'hostu13'
CRS-2673: Attempting to stop 'ora.asm' on 'hostu13'
CRS-2677: Stop of 'ora.cssdmonitor' on 'hostu13' succeeded
CRS-2677: Stop of 'ora.evmd' on 'hostu13' succeeded
CRS-2677: Stop of 'ora.asm' on 'hostu13' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'hostu13' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'hostu13'
CRS-2677: Stop of 'ora.cssd' on 'hostu13' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'hostu13'
CRS-2677: Stop of 'ora.diskmon' on 'hostu13' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hostu13'
CRS-2676: Start of 'ora.cssdmonitor' on 'hostu13' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hostu13'
CRS-2672: Attempting to start 'ora.diskmon' on 'hostu13'
CRS-2676: Start of 'ora.diskmon' on 'hostu13' succeeded
CRS-2676: Start of 'ora.cssd' on 'hostu13' succeeded
CRS-4611: Successful deletion of voting disk +OCR_VOTE.
CRS-2672: Attempting to start 'ora.ctssd' on 'hostu13'
CRS-2676: Start of 'ora.ctssd' on 'hostu13' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'hostu13'
CRS-2676: Start of 'ora.asm' on 'hostu13' succeeded
ASM de-configuration trace file location: /tmp/deinstall2013-12-20_09-10-20-AM/logs/asmcadc_clean4777203075223230276.log
ASM Clean Configuration START
ASM Clean Configuration END

ASM with SID +ASM1 deleted successfully. Check /tmp/deinstall2013-12-20_09-10-20-AM/logs/asmcadc_clean4777203075223230276.log for details.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hostu13'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'hostu13'
CRS-2673: Attempting to stop 'ora.ctssd' on 'hostu13'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'hostu13'
CRS-2677: Stop of 'ora.cssdmonitor' on 'hostu13' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'hostu13' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'hostu13' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'hostu13'
CRS-2677: Stop of 'ora.cssd' on 'hostu13' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'hostu13'
CRS-2673: Attempting to stop 'ora.diskmon' on 'hostu13'
CRS-2677: Stop of 'ora.gpnpd' on 'hostu13' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'hostu13'
CRS-2677: Stop of 'ora.gipcd' on 'hostu13' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'hostu13' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'hostu13' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node

=====================================================================================================================

ORA-01035 ORACLE only available to users with RESTRICTED SESSION privilege

##########################
## Error
##########################

ORA-01035: ORACLE only available to users with RESTRICTED SESSION privilege

### Full Error

oralin>conn rm_usr/*****@oralin
ERROR:
ORA-01035: ORACLE only available to users with RESTRICTED SESSION privilege

##########################
#  Error Occurred
##########################

Error occured while connecting to a database in sqlplus in 11.2.0.3.0 version

##########################
## Command Executed
##########################

In Sqlplus,

conn rm_usr/*****@oralin

**************************************************************** Step By Step Analysis ***********************************************************************

#########################################
# 1) Check DB Status
#########################################

select instance_number,instance_name,startup_time,status,logins,shutdown_pending,database_status,instance_role from v$instance;

INSTANCE_NUMBER INSTANCE_NAME             STARTUP_T STATUS               LOGINS                    SHU DATABASE_STATUS      INSTANCE_ROLE
--------------- ------------------------- --------- -------------------- ------------------------- --- -------------------- --------------------
              1 oralin                    26-NOV-13 OPEN                 RESTRICTED                NO  ACTIVE               PRIMARY_INSTANCE

=====================================================================================================================

#########################################
# 2) Reason for Login Failure
#########################################

Users are not allowed to login because the database is being placed in RESTRICTED mode. So only users with restricted session privilege can login to the database.

=====================================================================================================================
Check the reason for placing the db in RESTRICTED mode or whether we can disable it.
=====================================================================================================================

#########################################
# 3) To disable ( Solution )
#########################################

alter system disable restricted session;

select instance_number,instance_name,startup_time,status,logins,shutdown_pending,database_status,instance_role from v$instance;

INSTANCE_NUMBER INSTANCE_NAME             STARTUP_T STATUS               LOGINS                    SHU DATABASE_STATUS      INSTANCE_ROLE
--------------- ------------------------- --------- -------------------- ------------------------- --- -------------------- --------------------
              1 oralin                    26-NOV-13 OPEN                 ALLOWED    NO  ACTIVE               PRIMARY_INSTANCE


=====================================================================================================================

ORA-01580 : error creating control backup file

ORA-01580 : error creating control backup file

##########################
## Error
##########################

ORA-01580: error creating control backup file /u01/app/oracle/product/11.2/db_4/dbs/snapcf_oralin01p_2.f
ORA-27040: file create error, unable to create file

### Full Error

Starting Control File and SPFILE Autobackup at 2013-12-17:00:17:53
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of Control File and SPFILE Autobackup command on ORA_SBT_TAPE_1 channel at 12/17/2013 00:18:09
ORA-01580: error creating control backup file /u01/app/oracle/product/11.2/db_4/dbs/snapcf_oralin01p_2.f
ORA-27040: file create error, unable to create file
Linux-x86_64 Error: 2: No such file or directory

##########################
#  Error Occurred
##########################

Error occured while a scheduled FULL backup is run in 11.2.0.3.0 version

##########################
## Command Executed
##########################

Full Database backup command

**************************************************************** Step By Step Analysis ***********************************************************************

#########################################
# 1) Check the Snapshot Controlfile
#########################################

RMAN> show snapshot controlfile name;

RMAN configuration parameters for database with db_unique_name oralin01P_STL are:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/11.2/db_4/dbs/snapcf_oralin01p_2.f';

RMAN>

=====================================================================================================================

#########################################
# 2) Reason for Failure
#########################################

Error is occuring when autobackup of controlfile is being done. Whenever oracle tries to update the recovery catalog, it checks with recovery catalog to find how much updated recovery catalog is.

So when trying to backup the controlfile, RMAN tries to write the information to snapshot controlfile. If a snapshot controlfile is not available then RMAN creates a new file.

In our Case, both updating the existing file and new file creation failed with the below error.

=====================================================================================================================
ORA-27040: file create error, unable to create file
Linux-x86_64 Error: 2: No such file or directory
=====================================================================================================================

#########################################
# 3) Reproduce the Error
#########################################

A simple Resync will help us to reproduce the error.

oralin01p_2 @ hostp03:/u01/app/oracle/product/11.2
> rman target / catalog rman_user/****@rcat

Recovery Manager: Release 11.2.0.2.0 - Production on Tue Dec 17 03:49:13 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: oralin01P (DBID=4249580499)
connected to recovery catalog database

RMAN> resync catalog;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of resync command on default channel at 12/17/2013 03:53:34
ORA-01580: error creating control backup file /u01/app/oracle/product/11.2/db_4/
ORA-27040: file create error, unable to create file
Linux-x86_64 Error: 2: No such file or directory

#########################################
# Alert Log
#########################################

Tue Dec 17 03:53:33 2013
Errors in file /u01/app/oracle/diag/rdbms/oralin01p_stl/oralin01p_2/trace/oralin01p_2_ora_27685.trc:
ORA-01580: error creating control backup file /u01/app/oracle/product/11.2/db_4/dbs/snapcf_oralin01p_2.f
ORA-27040: file create error, unable to create file
Linux-x86_64 Error: 2: No such file or directory

=====================================================================================================================

#########################################
# 4) Check PATH for snapshot controlfile
#########################################

oralin01p_2 @ hostp03:/u01/app/oracle/product/11.2
> ls
agent11g  db_1  db_10  db_11  db_2  emcli  lost+found
oralin01p_2 @ hostp03:/u01/app/oracle/product/11.2

=====================================================================================================================
If we can see from above, there is no DB_HOME db_4, so because of this Snapshot controlfile creation/update fails.
=====================================================================================================================

#########################################
# 5) Configure Snapshot Controlfile
#########################################

Snapshot Controlfile should be in Shared Location, else we will face ORA-00245 Error

http://stepintooracledba.blogspot.com/2013/04/ora-00245-control-file-backup-failed.html

=====================================================================================================================
Before Configuring snapshot controlfile, lets see if any file exists in that location
=====================================================================================================================

ASMCMD [+oralin01P_RECO] > ls -lt
Type         Redund  Striped  Time             Sys  Name
                                               Y    oralin01P_STL/
ASMCMD [+oralin01P_RECO] >

No file Exists

##########################
## Solution
##########################

configure snapshot controlfile name to '+oralin01P_RECO/snapcf_oralin01p.f';

RMAN> configure snapshot controlfile name to '+oralin01P_RECO/snapcf_oralin01p.f';

old RMAN configuration parameters:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/11.2/db_4/dbs/snapcf_oralin01p_2.f';
new RMAN configuration parameters:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+oralin01P_RECO/snapcf_oralin01p.f';
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete

=====================================================================================================================
New file named snapcf_oralin01p.f is created.
=====================================================================================================================

ASMCMD [+oralin01P_RECO] > ls -lt
Type         Redund  Striped  Time             Sys  Name
                                               Y    oralin01P_STL/
                                               N    snapcf_oralin01p.f => +oralin01P_RECO/oralin01P_STL/CONTROLFILE/Backup.1698.834378939
ASMCMD [+oralin01P_RECO] >


Now Resync of Catalog is working good.

RMAN>  resync catalog;

starting full resync of recovery catalog
full resync complete


=====================================================================================================================

Different Methods to find or Check whether the installed Oracle Client Software is 32 bit or 64 bit in Windows Environments

Some times we may need to check whether the oracle client software which is installed in Windows Environments is 32 bit or 64 bit, and this sometimes comes mostly because of the application team who bugs us and say that we need 64 bit client sofware or 32 bit.

Eventhough they have same version of client software, still they will say i want it. This is app team's default behaviour. ( Just Joking ).

If you wanna check in Unix / Linux Environments, please use below link.

http://stepintooracledba.blogspot.com/2013/08/different-methods-to-find-or-check.html

Below are the two methods which i use mostly to find out the bit size.

**************************************************************** Methods ******************************************************************************

#########################################
# 1) Using Task Manager
#########################################

In Command Prompt, Change directory (cd) to Oracle Client Home bin directory ( $ORACLE_HOME/bin ) and invoke Sqlplus

After invoking, go to Task Manager & check the process running.

From the below screenshot, if you can see, we have two Sqlplus Prompts, And from Image its clear that

32 bit version shows as "Oracle SQL*PLUS(32 bit)" and

64 bit version shows as "Oracle SQL*PLUS"

Note : Below Task Manager output is from Windows Server 2012

=====================================================================================================================

#########################################
# 2) Using Registry ( regedit )
#########################################

Go to run and type regedit.

In registry Editor navigate as below,

HKEY_LOCAL_MACHINE --> SOFTWARE --> ORACLE --> KEY_OraHome1 ( If oracle Client Software folder exists in this place then its of 64 bit version )

HKEY_LOCAL_MACHINE --> SOFTWARE --> Wow6432Node --> ORACLE --> KEY_OraHome1 ( If oracle Client Software folder exists in this place then its of 32 bit version )

=====================================================================================================================

Modify Hidden Parameter in Oracle Database

Lets see the steps to Modify Hidden Parameter in Oracle Database

*********************************************** Step By Step Procedure **************************************

Click Here to know how to check Hidden Parameter @ database

#########################################
# 1) check Present Value of the parameter
#########################################

Select name,VALUE,ISDEFAULT,ISSES_MODIFIABLE,ISSYS_MODIFIABLE from v$parameter where name like '%real%';

NAME                                VALUE                     ISDEFAULT ISSES ISSYS_MOD
----------------------------------- ------------------------- --------- ----- ---------
_realfree_heap_pagesize_hint        65536 FALSE     FALSE FALSE

=====================================================================================================================

#########################################
# 2) Modify the parameter in Spfile
#########################################

alter system set "_realfree_heap_pagesize_hint"=262144 scope=spfile sid='*';

Note :

a) As this is RAC environment, we are using sid='*'
b) For Hidden Parameter we are using double quotes (" "), this is not needed for normal parameters

=====================================================================================================================

#########################################
# 3) Shutdown / Startup Database
#########################################

srvctl stop database -d oralin

srvctl start database -d oralin -n hostu01

srvctl status database -d oralin

=====================================================================================================================

#########################################
# 4) Check the modified Value
#########################################

Select name,VALUE,ISDEFAULT,ISSES_MODIFIABLE,ISSYS_MODIFIABLE from v$parameter where name like '%real%';

NAME                                VALUE                     ISDEFAULT ISSES ISSYS_MOD
----------------------------------- ------------------------- --------- ----- ---------
_realfree_heap_pagesize_hint        262144                    FALSE     FALSE FALSE

=====================================================================================================================

Enterprise Manager Agent Installation fails with emnoseed.rsp file doesnot exists. Error

##########################
## Error
##########################

emnoseed.rsp file doesnot exists.

### Full Error

ERROR:  /u01/app/oracle/product/11.2.0.3/linux_x64/agent//install/response//emnoseed.rsp file doesnot exists.

##########################
#  Error Occurred
##########################

Error occured while trying to install Enterprise Manager Grid Agent of 11.1.0.1.0 version using Oracle Universal Installer ( OUI ) in Linux Platform

##########################
## Command Executed
##########################

In terminal,

./runInstaller

##########################
## Issue Description.
##########################

### 11.1.0.1.0

#########################################
# AGent Installation Logs
#########################################

Error occured while trying to install Enterprise Manager Grid Agent of 11.1.0.1.0 version using Oracle Universal Installer ( OUI ) in Linux Platform.

no_sid @ hostp20:/u01/app/oracle/product/11.2.0.3/linux_x64/agent
> ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 150 MB.   Actual 1834 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 32765 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 32768    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-12-03_12-07-46AM. Please wait ...no_sid @ hostp20:/u01/app/oracle/product/11.2.0.3/linux_x64/agent
>
ERROR:  /u01/app/oracle/product/11.2.0.3/linux_x64/agent//install/response//emnoseed.rsp file doesnot exists.

#### Note :

In 11.1.0.1 versions, Agent installation is not possible in GUI mode.

when we try the same in Other platforms of 11.1.0.1 version, OUI will warn that the GUI-based OUI is not a valid option for the 11.1.0.1 Agent installation.

In Linux Platforms, there is a internal BUG due to which this message is not being displayed to user and emnoseed.rsp file doesnot exists error occurs.

##########################
## Solution
##########################

Try a silent mode of installation. Below are the methods available for Agent Installation. Oracle Documentation is available for these methods.

I will try to post the detailed steps for different methods of agent installation.

1) Installing Oracle Management Agent on a Cluster Using Agent Deployment Wizard

2) Installing Oracle Management Agent Using Shared Oracle Home

3) Cloning Oracle Management Agent Using Agent Cloning Wizard

4) Installing Oracle Management Agent Using Response File

5) Installing Oracle Management Agent Using agentDownload Script

6) Installing Oracle Management Agent on a Cluster Using Response File or agentDownload Script

7) Installing Oracle Management Agent Using Shared Oracle Home Using nfsagentinstall Script

8) Cloning Oracle Management Agent Using ZIP File

9) Cloning Oracle Management Agent on Clusters Using ZIP File

Relocate a voting disk from ASM Diskgroup to a NFS shared file system and Vice Versa in 11g

##########################
## Objective
##########################

Relocate a voting disk from ASM Diskgroup to a NFS shared file system and Vice Versa in 11g

======================================================================================================================

Disk Group Where Voting Disk is now : +OCR_VOTE
New Voting disk location (NFS Mount Point ) : /u07/oracle

### Points to note

Voting Disk must be placed on shared devices; either on cluster filesystems, NFS or shared raw devices.

======================================================================================================================

Step  1 : Check the  Present Location of Voting disk
Step  2 : Relocate voting disk from ASM to NFS shared Mount point
Step  3 : Now Check the New Location of Voting disk
Step  4 : Backup of Voting Disk
Step  5 : Relocate the Vote disk back to ASM Disk group
Step  6 : Now Check the New Location of Voting disk

======================================================================================================================

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Relocate voting Disk @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

======================================================================================================================
Step 1 :
--------
Check the  Present Location of Voting disk
======================================================================================================================

crsctl query css votedisk

+ASM1 @ hostd01:/u01/home/oracle
> crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   19ac2decfd6c4f6ebf87cd9d247d7e7a (/dev/mapper/ocr01p1) [OCR_VOTE]
 2. ONLINE   086dd7a7970e4f5dbf62d8760b8ab480 (/dev/mapper/ocr02p1) [OCR_VOTE]
 3. ONLINE   4a96ec03cee14f54bfa8835a98c5d31b (/dev/mapper/ocr03p1) [OCR_VOTE]
Located 3 voting disk(s).
+ASM1 @ hostd01:/u01/home/oracle

======================================================================================================================
Step 2 :
--------
Relocate voting disk from ASM to NFS shared Mount point
======================================================================================================================

crsctl replace votedisk /u07/oracle/votedisk.dt


+ASM1 @ hostd01:/u07/oracle
> crsctl replace votedisk /u07/oracle/votedisk.dt
Now formatting voting disk: /u07/oracle/votedisk.dt.
CRS-4256: Updating the profile
Successful addition of voting disk 78e3c124a6f34f93bf2141cc31a97759.
Successful deletion of voting disk 19ac2decfd6c4f6ebf87cd9d247d7e7a.
Successful deletion of voting disk 086dd7a7970e4f5dbf62d8760b8ab480.
Successful deletion of voting disk 4a96ec03cee14f54bfa8835a98c5d31b.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
+ASM1 @ hostd01:/u07/oracle

Note : If you want to check the File system type, then in Linux use below command to find it.

df -T .

+ASM1 @ hostd01:/u07/oracle
> df -T .
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
hostp02:/vol/u07/u07
               nfs   637009920 400898560 236111360  63% /u07

======================================================================================================================
Step 3 :
--------
Now Check the New Location of Voting disk
======================================================================================================================

+ASM1 @ hostd01:/u07/oracle
> crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   78e3c124a6f34f93bf2141cc31a97759 (/u07/oracle/votedisk.dt) []
Located 1 voting disk(s).
+ASM1 @ hostd01:/u07/oracle

======================================================================================================================
Step 4 :
--------
Backup of Voting Disk ( We will take a backup of voting disk for safety purpose )
======================================================================================================================

dd if=/u07/oracle/votedisk.dt of=/u07/oracle/vd.bkp

+ASM1 @ hostd01:/u07/oracle
> dd if=/u07/oracle/votedisk.dt of=/u07/oracle/vd.bkp
41025+0 records in
41025+0 records out
21004800 bytes (21 MB) copied, 3.80843 seconds, 5.5 MB/s
+ASM1 @ hostd01:/u07/oracle


+ASM1 @ hostd01:/u07/oracle

-rw-r--r--  1 oracle oinstall 21004800 Nov 29 03:15 vd.bkp
-rw-r-----  1 oracle oinstall 21004800 Nov 29 03:15 votedisk.dt


======================================================================================================================
Step 5 :
--------
Relocate the Vote disk back to ASM Disk group
======================================================================================================================

crsctl replace votedisk +OCR_VOTE

+ASM1 @ hostd01:/u07/oracle
> crsctl replace votedisk +OCR_VOTE
CRS-4256: Updating the profile
Successful addition of voting disk 02b990acc2414f21bf4d483e72ccf1ec.
Successful addition of voting disk 889aa822e04a4f1ebf648fccef8fbe69.
Successful addition of voting disk 0c28a49e64344f64bfe2ec1792305602.
Successful deletion of voting disk 78e3c124a6f34f93bf2141cc31a97759.
Successfully replaced voting disk group with +OCR_VOTE.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced

Backup a Voting Disk in Oracle 11g & 10g Versions

##########################
## Objective
##########################

Backup a Voting Disk

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Voting Disk Backup @@@@@@@@@@@@@@
s
##########################
## From 11g R2 Onwards
##########################

Backing up voting disks manually is no longer required because voting disk data is backed up automatically in the OCR as part of any configuration change and voting disk data is automatically restored to any added voting disks.

##########################
## In 10g
##########################


======================================================================================================================

Step  1 : Backup Voting Disk
Step  2 : Find Voting disks in the Database

======================================================================================================================

======================================================================================================================
Step 1 :
--------
Find Voting disks in the Database
======================================================================================================================

Set crs environment,

crsctl query css votedisk

crs10g @ hostu03:/opt/oracle/admin
> crsctl query css votedisk
 0.     0    /dev/raw/vot1
 1.     0    /dev/raw/vot2
 2.     0    /dev/raw/vot3

located 3 votedisk(s).
crs10g @ hostu03:/opt/oracle/admin

======================================================================================================================
Step 2 :
--------
Backup Voting Disk
======================================================================================================================

dd if=/dev/raw/vot1 of=/opt/oracle/admin/voting.bkp

crs10g @ hostu03:/opt/oracle/admin
> dd if=/dev/raw/vot1 of=/opt/oracle/admin/voting.bkp

280709+0 records in
280709+0 records out

crs10g @ hostu03:/opt/oracle/admin
> du -sh voting.bkp
 137M   voting.bkp
crs10g @ hostu03:/opt/oracle/admin

======================================================================================================================

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ END @@@@@

Enterprise Manager Agent Installation Pre Checks Utility for a sucessful Installation

We have built a new RAC Server for which agent installation needs to be done. So to confirm whether the installation of Agent will be successful.

I have performed the below steps which tells that installation of Agent will be successful.

########################
     Command to Run Pre Checks
########################

cd /u01/app/oracle/exp/Software/agent/linux_x64/agent/install/

./runInstaller -prereqchecker PREREQ_CONFIG_LOCATION=/u01/app/oracle/exp/Software/agent/linux_x64/agent/stage/prereq -entrypoint "oracle.sysman.top.agent_Complete" -prereqLogLoc /tmp/agent_prereq_check -silent -waitForCompletion

Where,

Software Location - /u01/app/oracle/exp/Software/agent/

A Folder will be created in "/tmp/agent_prereq_check" which stores the logs and pre-requisties logs.

+ASM1 @ hostp16:/tmp/agent_prereq_check
> ls -ltr
total 16
drwxr-xr-x 2 oracle oinstall 4096 Nov 26 01:13 logs
-rw------- 1 oracle oinstall    0 Nov 26 01:13 prereq2013-11-26_01-13-39AM.err
drwxr-xr-x 2 oracle oinstall 4096 Nov 26 01:13 results
-rw------- 1 oracle oinstall 2516 Nov 26 01:13 prereq2013-11-26_01-13-39AM.out
-rw------- 1 oracle oinstall 3528 Nov 26 01:13 prereq2013-11-26_01-13-39AM.log
+ASM1 @ hostp16:/tmp/agent_prereq_check

########################
     Pre Checks Log
########################

+ASM1 @ hostp16:/u01/home/oracle
> tage/prereq -entrypoint "oracle.sysman.top.agent_Complete" -prereqLogLoc /tmp/agent_prereq_chck.txt -silent -waitForCompletion                             <
Starting Oracle Prerequisite Checker...

Checking Temp space: must be greater than 150 MB.   Actual 1404 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 32301 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-11-26_01-13-39AM. Please wait ...
Oracle Prerequisite Checker version Version 11.1.0.8.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.


Starting execution of Prerequisites...
Total No of checks: 6

Performing check for CertifiedVersions_agent
Checking if Oracle software certified on the current O/S...
Expected result: One of redhat-5.4,redhat-5.3,redhat-5.2,redhat-5.1,redhat-5,redhat-4,SuSE-11,SuSE-10,SuSE-9,SuSE-8,enterprise-5.4,enterprise-5.3,enterprise-5.2,enterprise-5.1,enterprise-5,enterprise-4,UnitedLinux-1.0,asianux-3,asianux-2,asianux-1
Actual Result: redhat-5
Check complete. The overall result of this check is: Passed


Check complete: Passed
========================================================
Performing check for Packages_agent
Checking for required packages installed on the system ....
Checking for make-3.81; found make-1:3.81-3.el5-x86_64. Passed
Checking for binutils-2.17.50.0.6; found binutils-2.17.50.0.6-20.el5_8.3-x86_64.        Passed
Checking for gcc-4.1.1; found gcc-4.1.2-54.el5-x86_64.  Passed
Checking for libstdc++-4.1.1; found libstdc++-4.1.2-54.el5-x86_64.      Passed
Check complete. The overall result of this check is: Passed


Check complete: Passed
========================================================
Performing check for CompatibilityCheck
Checking whether the software compatibile for current o/s...
Check complete. The overall result of this check is: Passed


Check complete: Passed
========================================================
Performing check for AvailableTempSpace
Checking for sufficient diskspace in TEMP location...
Expected result: 400.0MB
Actual Result: 1224704MB
Check complete. The overall result of this check is: Passed


Check complete: Passed
========================================================
Performing check for AvailableInventorySpace
Checking for sufficient diskspace in Inventory location...
Expected result: 100.0MB
Actual Result: 38866944MB
Check complete. The overall result of this check is: Passed


Check complete: Passed
========================================================
Performing check for CheckHostName
Checking for the Hostname...
Check complete. The overall result of this check is: Passed


Check complete: Passed
========================================================
PrereqChecks complete

Emctl Upload Agent Fails with OMS version not checked yet. Error

##########################
## Error
##########################

OMS version not checked yet.

### Full Error

no_sid @ hostp16:/u01/app/oracle/product/11.2/agent11g/bin
> emctl upload agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors.

##########################
#  Error Occurred
##########################

Error occured while trying to upload the Agent in 11.1.0.1.0 version of agent

##########################
## Command Executed
##########################

emctl upload agent

##########################
## Issue Description.
##########################

### 11.1.0.1.0

#########################################
# Happened When....
#########################################

Enterprise Manager Grid Agent was installed already on this server, and due to some situation, Agent software was de-installed and it was installed back.

After this Agent Upload to OMS is not working.

#########################################
# Upload of Agent
#########################################

no_sid @ hostp16:/u01/app/oracle/product/11.2/agent11g/bin
> emctl upload agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors.

#########################################
# Status of Agent
#########################################

Agent is Up and Running but uploading is not happening.

no_sid @ hostp16:/u01/app/oracle/product/11.2/agent11g/bin
> ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /u01/app/oracle/product/11.2/agent11g
Agent binaries    : /u01/app/oracle/product/11.2/agent11g
Agent Process ID  : 25235
Parent Process ID : 25200
Agent URL         : https://hostp16:3872/emd/main/
Repository URL    : https://oemp02:1159/em/upload
Started at        : 2013-11-20 04:44:02
Started by user   : oracle
Last Reload       : 2013-11-20 04:44:02
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far :     0.00
Number of XML files pending upload           :       27
Size of XML files pending upload(MB)         :     5.23
Available disk space on upload filesystem    :    66.16%
Last attempted heartbeat to OMS              : 2013-11-20 04:49:26
Last successful heartbeat to OMS             : unknown
---------------------------------------------------------------
Agent is Running and Ready
no_sid @ hostp16:/u01/app/oracle/product/11.2/agent11g/bin

#########################################
# EM Agent Trace Info..
#########################################

no_sid @ hostp16:/u01/app/oracle/product/11.2/agent11g/sysman/log
> vi emagent.trc
no_sid @ hostp16:/u01/app/oracle/product/11.2/agent11g/sysman/log

2013-11-20 04:58:11,039 Thread-896039232 ERROR pingManager: Did not receive valid response to ping "ERROR-Agent is blocked.
Blocked reason is: Agent is out-of-sync with repository. This most likely means that the agent was reinstalled or recovered.
Please contact an EM administrator to unblock the agent by performing an agent resync from the console. Please contact EM adminstrator to unblock the agent"

##########################
## Solution
##########################

Login to EM Grid console,

Setup >> Agents >> hostp16 >>

Now Agent STatus will show as " Agent Unreachable, Agent is Blocked "

Click on "Upload Metric Data" from Upload TAB and it has given failure Message.

Now we have to do Resync of Agent,

Agent Resynchronization (Setup >> Agents >> hostp16 >> Click on Agent resynchronization),

click continue, Once it completes, again click "Upload Metric Data".

Now we can see Agent is uploading DAta's successfully.




no_sid @ hostp16:/u01/app/oracle/product/11.2/agent11g/bin
> emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /u01/app/oracle/product/11.2/agent11g
Agent binaries    : /u01/app/oracle/product/11.2/agent11g
Agent Process ID  : 30627
Parent Process ID : 30594
Agent URL         : https://hostp16:3872/emd/main/
Repository URL    : https://oemp02:1159/em/upload
Started at        : 2013-11-20 04:58:03
Started by user   : oracle
Last Reload       : 2013-11-20 05:06:11
Last successful upload                       : 2013-11-20 05:06:43
Total Megabytes of XML files uploaded so far :     0.93
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    66.18%
Last successful heartbeat to OMS             : 2013-11-20 05:06:51
---------------------------------------------------------------
Agent is Running and Ready
no_sid @ hostp16:/u01/app/oracle/product/11.2/agent11g/bin


no_sid @ hostp16:/u01/app/oracle/product/11.2/agent11g/bin
> emctl upload agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully
no_sid @ hostp16:/u01/app/oracle/product/11.2/agent11g/bin

How to find the Cluster Name in RAC Environment?

How to find the Cluster Name in RAC Environment?

Method 01 :
-------------

$ORA_CRS_HOME/bin/cemutlo -n


no_sid @ hostp18:/u01/app/oracle/product/11.2
> /u01/app/11.2/grid/bin/cemutlo -n
clusprod06
no_sid @ hostp18:/u01/app/oracle/product/11.2


Method 02 :
-------------

ls of $GRID_HOME/cdata will list out the Cluster Name. ( We will have directories called localhost, servername, and then cluster name )


no_sid @ hostorap16:/u01/app/11.2/grid/cdata
> ls -ltr
total 2924
drwxr-xr-x 2 oracle oinstall      4096 Nov  6 11:27 localhost
drwxr-xr-x 2 oracle oinstall      4096 Nov  6 11:47 hostorap16
drwxrwxr-x 2 oracle oinstall      4096 Nov 20 10:49 clusprod06

To Set SQL Plus Prompt to the database Name of which we were logged in using login.sql

########################
 Task
########################

To Set SQL Plus Prompt to the database Name of which we were logged in.

Usually When we login to SQLPLUS, the prompt will be like below,

SQL>

And in many situations we need to check which database we are logged in. A Simple Solution is.

==================================================================================================
Step 1 : Use of login.sql
==================================================================================================

Locate login.sql file. The location for this file is $ORACLE_HOME/dbs

If the file is not available then create a file with the name login.sql and add the below content to it and save the file.

==================================================================================================
set termout off
spool set_prompt.sql
     select 'set sqlprompt '||instance||'>'
     from   sys.v_$thread;
     spool off
     start set_prompt.sql
set termout on
==================================================================================================

==================================================================================================
Step 2 : Now login and Check
==================================================================================================

Now login to the database and check.

oralin @ exampled01:/u01/app/oracle/product/11.2/db_2/dbs
> sps

SQL*Plus: Release 11.2.0.3.0 Production on Fri Nov 8 04:42:17 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> conn test/test@oradup as sysdba
Connected.
oradup>
oradup>conn test/test@orawin as sysdba
Connected.
orawin>
orawin>

==================================================================================================

ORA-12154: TNS: could not resolve the connect identifier specified

##########################
## Error
##########################

ORA-12154: TNS: could not resolve the connect  identifier specified

### Full Error

Mon Nov 04 21:01:00 2013
Error 12154 received logging on to the standby

##########################
#  Error Occurred
##########################

Error occured while Primary database is trying to connect to a standby database for transfering archive logs in 11.2.0.3.0 version

##########################
## Command Executed
##########################

Primary trying to transfer Archive logs to Standby

##########################
## Issue Description / Troubleshooting
##########################

### 11.2.0.3.0

#########################################
# 1) Check Whether archive destination is showing any errors
#########################################

set lines 200
col dest_name for a20
col destination for a50
col error for a15
col db_unique_name for a15
col name for a100

select dest_id,dest_name,target,name_space,destination,status,error,db_unique_name from v$archive_dest where destination is not null;

   DEST_ID DEST_NAME            TARGET  NAME_SP DESTINATION                                        STATUS    ERROR           DB_UNIQUE_NAME
---------- -------------------- ------- ------- -------------------------------------------------- --------- --------------- ---------------
         1 LOG_ARCHIVE_DEST_1   PRIMARY SYSTEM  USE_DB_RECOVERY_FILE_DEST                          VALID                     NONE
         2 LOG_ARCHIVE_DEST_2   STANDBY SYSTEM  oralin_stdby                                        ERROR     ORA-12154:      oralin_stdby
                                                                                                             TNS:could not
                                                                                                             resolve the
                                                                                                             connect
                                                                                                             identifier
                                                                                                             specified

#########################################
# 2) TNS Names Used to Connect
#########################################


oralin_stdby.example.com =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = kstdbylorap08)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = oralin_stdby.example.com)
    )
  )

#########################################
# 3) Check Service & Domain parameters
#########################################

In standby :

SQL> sho parameter servi

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
service_names                        string      oralin_stdby
db_domain                            string

##########################
## Solution
##########################

From Troubleshooting 2) output, we can see from TNSNAMES that service name is oralin_stdby and Domain is example.com

From Troubleshooting 2) output, Service name is correct but there is no domain set at the database level.

So the problem is with the domain name. I have removed Domain (example.com) from TNSNAMES and the connection to database was successful.

Modified TNSNAMES :


oralin_stdby.example.com =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = kstdbylorap08)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = oralin_stdby)
    )
  )