Issue during database upgrade from 11.2.0.1 to 11.2.0.3 on VM

Today, We came across the different issue while upgrade. Database was successfully opened in 11.2.0.1 but while " Startup Upgrade", it was throwing following errors-

ORA-01157: cannot identify/lock data file 359 - see DBWR trace file

ORA-01110: data file 359: 'apps_undots.dbf'
ORA-27048: skgfifi: file header information is invalid

In alert log we found:

ALTER DATABASE OPEN MIGRATE

Direct NFS: attempting to mount /vol/ORADATA_FS01 on filer defined in mtab
Direct NFS: channel config is:
channel id [0] local [] path ]
Direct NFS: mount complete dir /ORADATA_FS01 Errors in file /oracle/admin//diag/rdbms/ts2ecp/dbw0_20560.trc:
ORA-01157: cannot identify/lock data file 1 - see DBWR trace file
ORA-01110: data file 1: '/oracle/oradata/fs01/system_01.dbf'
ORA-17503: ksfdopn:7 Failed to open file /oracle/oradata/fs01/system_01.dbf
ORA-27047: unable to read the header block of file
Linux-x86_64 Error: 11174: Unknown system error
Additional information: 7
Additional information: 1795
Additional information: 0
Direct NFS: attempting to mount /vol/ORADATA_FS02 on filer DB-generic10-vfiler112 defined in mtab
Direct NFS: mount complete dir /vol/ORADATA_FS02 on --vfiler112 mntport 4046 nfsport 2049


To resolve this issue we generated STRACE :
strace -fo

sqlplus /nolog


SQL> conn / as sysdba

SQL> startup migrate

Strace o/p:

 3314 stat("/oracle/oradata/fs01/system_01.dbf", {st_mode=S_IFREG

0644, st_size=9018802176, ...}) = 0


3314 stat("/oracle/oradata//fs01/system_01.dbf", {st_mode=S_IFREG
0644, st_size=9018802176, ...}) = 0


3314 open("/oracle/oradata//fs01/system_01.dbf", O_RDONLY) = 18


3314 ioctl(18, BLKSSZGET, 0x7fffdb90e398) = -1 ENOTTY (Inappropriate ioctl for device)


3314 close(18)

 

Solution:
While upgrade, oracle may update each datafile. But as observed in alert log and Strace, data in datafile reached it's brim, eventually not able to write in the datafiles. In our case, 7 datafiles were listed in alert log for which it was unable to update header.

As a solution, resized all the datafiles listed in the log in order to make some place available for updating header using following query:

alter database datafile 'datafile_name' resize 4800M;

ORA error :

ORA-25228: timeout or end-of-fetch during message dequeue from AQADM.CMRS_INCR_ITEM_DQ


ORA-06512: at "SYS.DBMS_AQ", line 335

Solution:
 
Try to restart the jobs if it doesnt fix the issue, then bounce job_queue_processes and aq_tm_processes.

Oracle database vault

command to check if database vault is enabled the database

GO to $ORACLE_HOME/rdbms/lib and execute following:
ar -t libknlopt.a | grep -c kzvidv.o

If it gives output as 0, database vault is disabled and vice versa.


To Enable:

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk dv_on
make -f ins_rdbms.mk ioracle

To Disable:

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk dv_off
make -f ins_rdbms.mk ioracle

Oracle DB Error:ORA-29702: error occurred in Cluster Group Service operation

Cause:
In case of standalone database,RAC has tobe made off.If its not made off,one can see this error while starting up the database.

Simple Solution:
cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk rac_off
make -f ins_rdbms.mk ioracle