ASM startup fail with ORA-29701

How to Reconfigure Oracle Restart :
Cause: Today I faced one issue while cloning my VM for making a dataguard server,
I done the server name change while crs was up and running and end up in ASM startup failure.

Solution:

The solution is to reconfigure Oracle Restart.

[root@srv2 grid]# /u01/app/12.1.0/grid/crs/install/roothas.pl -deconfig
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2016/07/21 19:20:30 CLSRSC-39: Oracle Restart stack is not active on this node

2016/07/21 19:20:30 CLSRSC-312: Failed to verify HA resources

Died at /u01/app/12.1.0/grid/crs/install/crsdeconfig.pm line 1358.
***********************************************************************************
######OK LOOKS LIKE WE NEED TO ADD FORCE OPTION ###################
***********************************************************************************

[root@srv2 grid]# /u01/app/12.1.0/grid/crs/install/roothas.pl -deconfig -force
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Stop failed, or completed with errors.
2016/07/21 19:22:03 CLSRSC-337: Successfully deconfigured Oracle Restart stack

[root@srv2 grid]# ./root.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0/grid
Copying dbhome to /usr/local/bin …
Copying oraenv to /usr/local/bin …
Copying coraenv to /usr/local/bin …

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user ‘grid’, privgrp ‘oinstall’..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
CRS-4664: Node srv2 successfully pinned.
2016/07/21 19:23:17 CLSRSC-330: Adding Clusterware entries to file ‘oracle-ohasd.conf’

srv2 2016/07/21 19:23:37 /u01/app/12.1.0/grid/cdata/srv2/backup_20160721_192337.olr 0
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘srv2’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘srv2’
CRS-2677: Stop of ‘ora.evmd’ on ‘srv2’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘srv2′ has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2016/07/21 19:25:54 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

2016/07/21 19:35:56 CLSRSC-352: CRS is already configured on this node for the CRS home location /u01/app/12.1.0/grid

[root@srv2 grid]# sudo su – grid
[grid@srv2 ~]$ srvctl add asm
[grid@srv2 ~]$ srvctl start asm
[grid@srv2 ~]$ ps -ef|grep pmon
grid 4902 1 0 19:37 ? 00:00:00 asm_pmon_+ASM
grid 4996 4778 0 19:37 pts/0 00:00:00 grep pmon
[grid@srv2 ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.1.0.2.0 Production on Thu Jul 21 19:38:11 2016

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 – 64bit Production
With the Automatic Storage Management option

SQL> show parameter spfile

NAME TYPE VALUE
———————————— ———– ——————————
spfile string
SQL> create spfile=’+CRS’ from pfile=’/u01/test.ora’;
create spfile=’+CRS’ from pfile=’/u01/test.ora’
*
ERROR at line 1:
ORA-17635: failure in obtaining physical sector size for ‘+CRS’
ORA-15001: diskgroup “CRS” does not exist or is not mounted

SQL> alter diskgroup CRS mount;
alter diskgroup CRS mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup “CRS” cannot be mounted
ORA-15040: diskgroup is incomplete

SQL> alter diskgroup DATA mount;
alter diskgroup DATA mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup “DATA” cannot be mounted
ORA-15040: diskgroup is incomplete

SQL> show parameter spfile

SQL>
Broadcast message from root@srv2.localdomain
(unknown) at 19:42 …

The system is going down for halt NOW!

Broadcast message from root@srv2.localdomain
(unknown) at 19:42 …

The system is going down for halt NOW!
login as: oracle
oracle@192.168.1.144’s password:
Last login: Thu Jul 21 19:01:31 2016 from 192.168.1.5
-bash: Page:: command not found
[oracle@srv2 ~]$ ps -ef|grep pmon
oracle    5157 5129 0 19:48 pts/0    00:00:00 grep pmon
[oracle@srv2 ~]$ sudo su – grid
[sudo] password for oracle:
[grid@srv2 ~]$ srvctl add asm
PRCA-1095 : Unable to create ASM resource because it already exists.
[grid@srv2 ~]$ srvctl start asm
PRCC-1014 : asm was already running
PRCR-1004 : Resource ora.asm is already running
PRCR-1079 : Failed to start resource ora.asm
CRS-5702: Resource ‘ora.asm’ is already running on ‘srv2′
[grid@srv2 ~]$ ps -ef|grep pmon
grid      5159     1 0 19:48 ?        00:00:00 asm_pmon_+ASM
grid      5437 5299 0 19:49 pts/0    00:00:00 grep pmon
[grid@srv2 ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.1.0.2.0 Production on Thu Jul 21 19:49:38 2016

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 – 64bit Production
With the Automatic Storage Management option

SQL> show parameter spfile

NAME TYPE VALUE
———————————— ———– ——————————
spfile string
SQL> alter diskgroup CRS mount;
alter diskgroup CRS mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup “CRS” cannot be mounted
ORA-15040: diskgroup is incomplete

SQL> !
[grid@srv2 ~]$ asmcmd
ASMCMD> mount all
ORA-15110: no diskgroups mounted (DBD ERROR: OCIStmtExecute)
ASMCMD> exit
[grid@srv2 ~]$ ls -l /dev/mapper/HDD* |wc -l
ls: cannot access /dev/mapper/HDD*: No such file or directory
0
[grid@srv2 ~]$ oakcli show disk
bash: oakcli: command not found
[grid@srv2 ~]$ sudo su –
[sudo] password for grid:
[root@srv2 ~]# oakcli show disk
-bash: oakcli: command not found
[root@srv2 ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@srv2 ~]# /etc/init.d/oracleasm listdisks
CRSDISK1
DATADISK1
FRADISK1
[root@srv2 ~]# sudo su – grid
[grid@srv2 ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.1.0.2.0 Production on Thu Jul 21 20:06:13 2016

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 – 64bit Production
With the Automatic Storage Management option

SQL> shutdown immediate
ORA-15100: invalid or missing diskgroup name

ASM instance shutdown
SQL> startup
ORA-00099: warning: no parameter file specified for ASM instance
ASM instance started

Total System Global Area 1140850688 bytes
Fixed Size                  2933400 bytes
Variable Size            1112751464 bytes
ASM Cache                  25165824 bytes
ORA-15110: no diskgroups mounted

SQL> shutdown
ORA-15100: invalid or missing diskgroup name

ASM instance shutdown
SQL> startup pfile=/u01/test.ora
ASM instance started

Total System Global Area 1140850688 bytes
Fixed Size                  2933400 bytes
Variable Size            1112751464 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
SQL> show parameter spfile

SQL> create spfile from pfile=’/u01/test.ora’;

File created.

SQL> show parameter spfile

NAME TYPE VALUE
———————————— ———– ——————————
spfile string
SQL> shutdown
ORA-15100: invalid or missing diskgroup name

ASM instance shutdown
SQL> startup
ASM instance started

Total System Global Area 1140850688 bytes
Fixed Size                  2933400 bytes
Variable Size            1112751464 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
SQL> select NAME,TOTAL_MB,FREE_MB from v$asm_diskgroup;

NAME                             TOTAL_MB    FREE_MB
—————————— ———- ———-
FRA                                 51199      47878
DATA                                51199      44783
CRS                                     0          0

SQL> alter diskgroup CRS mount;

Diskgroup altered.

SQL> select NAME,TOTAL_MB,FREE_MB from v$asm_diskgroup;

NAME                             TOTAL_MB    FREE_MB
—————————— ———- ———-
FRA                                 51199      47878
DATA                                51199      44783
CRS                                  2047       1985

SQL> create spfile=’+CRS’ from pfile=’/u01/test.ora’;

File created.

SQL> !ps -ef|grep pmon
grid      9401     1 0 20:09 ?        00:00:00 asm_pmon_+ASM
grid     10068 8903 0 20:12 pts/0    00:00:00 /bin/bash -c ps -ef|grep pmon
grid     10070 10068 0 20:12 pts/0    00:00:00 grep pmon

SQL>

*(/etc/host) (/etc/hosts)

Shanoj

My Experiments on Technology

Tag Archives: ASM startup fail with ORA-29701

How to Reconfigure Oracle Restart | ASM startup fail with ORA-29701.