RAC Instance not recognized by SRVCTL Utility

Issue:
-------

Version: 2 NODE 11.1.0.7 RDBMS RAC on 11.1.0.7 CRS + 11.1.0.7 ASM.

One of the RAC Instance is not being recongnized by SRVCRL; it was not getting started / stopped by SRVCTL utility. Below error were coming in adding / removing the instance when trying to fix it. In this example database sid is used as orcl1 and RAC nodes are NODEa and NODEb.

NODEa:/opt/ororcl1> srvctl status database -d orcl1
Instance orcl12 is running on NODE NODEb

NODEa:/opt/ororcl1> srvctl add database -d orcl1 -o /opt/ororcl1/product/11.1.0/rac -p +orcl1_DB/spfile.ora;
NODEa:/opt/ororcl1> srvctl add instance -d orcl1 -i orcl11 -n NODEa;
PRKO-2010 : Error in adding instance to NODE: NODEa
PRKR-1008 : adding of instance orcl11 on NODE NODEa to cluster database orcl1 failed.
CRS-0211: Resource 'ora.orcl1.orcl11.inst' has already been registered.
NODEa:/opt/ororcl1>
NODEa:/opt/ororcl1> srvctl modify instance -d orcl1 -i orcl11 -s +ASM1;
PRKO-2007 : Invalid instance name: orcl11


Cause:
--------

Database orcl1 doesn't recognize the instance keys anymore; Instance and database needs to be reconfigured with SRVCTL.


Resolution:
-------------

In this case please remove the instance with '-f' option

srvctl remove instance -d orcl1 -i orcl11 -f
srvctl remove instance -d orcl1 -i orcl12 -f
srvctl remove database -d orcl1 -f
srvctl config database


Try to add the database/instance again

srvctl add database -d orcl1 -o /opt/ororcl1/product/11.1.0/rac -p +orcl1_DB/spfile.ora;
srvctl add instance -d orcl1 -i orcl11 -n NODEa;
srvctl add instance -d orcl1 -i orcl12 -n NODEb;
srvctl modify instance -d orcl1 -i orcl11 -s +ASM1;
srvctl modify instance -d orcl1 -i orcl12 -s +ASM2;
srvctl setenv instance -d orcl1 -i orcl11 -T "TNS_ADMIN=/opt/ororcl1/product/11.1.0/rac/network/admin/orcl11_NODEa";
srvctl setenv instance -d orcl1 -i orcl12 -T "TNS_ADMIN=/opt/ororcl1/product/11.1.0/rac/network/admin/orcl12_NODEb";
srvctl start database -d orcl1;
srvctl status database -d orcl1;


Now both the Instances are getting recognized by SRVCTL.


NODEa:/opt/ororcl1> srvctl start database -d orcl1;
NODEa:/opt/ororcl1> srvctl status database -d orcl1;
Instance orcl11 is running on NODE NODEa
Instance orcl12 is running on NODE NODEb
NODEa:/opt/ororcl1> srvctl stop database -d orcl1;
NODEa:/opt/ororcl1> srvctl status database -d orcl1;
Instance orcl11 is not running on NODE NODEa
Instance orcl12 is not running on NODE NODEb
NODEa:/opt/ororcl1> srvctl start database -d orcl1;
NODEa:/opt/ororcl1> srvctl status database -d orcl1;
Instance orcl11 is running on NODE NODEa
Instance orcl12 is running on NODE NODEb