How I solved the ORA-08103: object no longer exists problem

Applies to:

Oracle Enterprise Edition 9.2.0.7.0
Any platform

Problem:

I had migrated an Oracle database from one server to another using the cold backup. The database came up clean. But the user had complaints saying his select count (*) on some tables gives the error ORA-08103: object no longer exists. But the select * for the same tables was giving the rows! There were around 3000 tables out of which 30 tables where giving the error. All these tables belonged to the same tablespace.

I tried the following.

SQL>Select * from test;
NAME
--------------
India
Australia
England
Srilanka
USA
5 rows selected

SQL> Select count (*) from test;
Select count (*) from test
*
ERROR at line 1:
ORA-08103: object no longer exist

Initially I thought there could be some problem with the indexes of these tables. I picked up one of the tables. And I tried validating the indexes by the below command

SQL>Analyze table test validate structure cascade;
Analyze table test validate structure cascade
*
ERROR at line 1:
ORA-08103: object no longer exists


And then I tried skipping the indexes by using the following command.
I used “no index” hint to execute “select count (*) from table”, command failed with the same error.
SQL>Select /*+NO INDEX*/ count (*) from test;
Select /*+NO INDEX*/ count (*) from test
*
ERROR at line 1:
ORA-08103: object no longer exists


Then I did the following.

1) Got the DDL of the table.
SQL>SELECT dbms_metadata.get_ddl(‘TABLE’, ‘TEST’,’USER’) FROM DUAL;

2) Create new table using above DDL

3) Execute the following statement.

SQL>INSERT INTO testr SELECT * FROM test;
INSERT INTO testr SELECT * FROM test;
*
ERROR at line 1:
ORA-08103: object no longer exists

Even this failed with the error ora-08103.
One of the solutions I tried was.

-For a table, truncate or drop it and reload the data from export dump

I dropped the tables having this error and took an export dump from the server (A) and imported the tables back into the server (B).For my surprise this time I faced the same problem with some other tables.

Cause:

Here, I had copied the database files (Cold backup) from Solaris server (A) ->windows server Solaris server (B) in the binary mode. This was done because I had a better network speed this way. This method had worked me for the same kind of restoration of the databases before. But this time it failed with the corruption of the data blocks. This could be because of

- The header block has an invalid block type or the block type inside the block is not expected. i.e. A data block (Type=6) was expected but the actual block information is not a data block.

Or

- The data_object_id (seg/obj) stored in the block is different than the data_object_id stored in the segment header.

Solution:

This time I copied the database files (cold backup) from solaris server (A) to solaris server (B) directly without the windows server in the middle. And I restored the whole database again. I had no problems with any of the tables and the database is running fine now.

The solution was to copy the files directly from the UNIX server to the UNIX server to avoid block corruptions of the data blocks.

References:

Note.268302.1 ORA-8103 Diagnostics and Solution

VNI-4026 : Discovery failed : Error while transferring services.ora from agent : 4009.

I am BACK after a LOOOOOOONG break...!!

I was not able to update my blog all these days but not anymore..Anyways,let me come to the point..

We have been migrating databases from one datacenter to another..Migration of the databases were pretty simple..Just take the cold backup of the database and restore it.We had to build a new server for the migration and the Unix consulatnt had installed the Oracle for us on the new server..Actually,he had not installed Oracle as such..He copied the Oracle home from the old server to the new server.Since the volumes on the server were identical,we had no issues in bringing up the database on the other server.

But when we tried to discover the new server on the OEM,there was this error

VNI-4026 : Discovery failed : Error while transferring services.ora from agent : 4009.

The intelligent agent was up and running.We stopped the agent and started the Intelligent agent after deleting *.q files in $ORACLE_HOME/network/agent/ location.Even then we had the same problem.

The above issue got resolved only after REINSTALLING the entire Oracle Home on the new server.It is recommended not to copy the Oracle Home from one server to another.Some components may work and some may not!!So always go with the fresh installations while taking such kind of activities.

Listener hangs in Oracle 10g and no new connections are allowed

Just when I was thinking how boring my job as DBA was (doing the same routine work and nothing new to implement until you convince everyone and take approvals from them!!) I was made to feel that DBA work is not at all boring if you are at the receiving end :-) . I was suddenly bombarded with mails from the users saying that they are not able to connect to the database. As usual I checked the listener status and replied to some of them saying that there is no problem with the listener as such but would get back to them with a solution. It was a production database, the pressure on me was mounting every second. I was not sure what to do? Reply to the users or solve the problem :-)

The database version was of Oracle 10.2.0.1 running on Linux RHEL 4..The listener seemed to be in hung state and moreover there were no errors in the listener log file.

I don’t know what came to my mind I thought of checking the listener process using the ps command. I was surprised to see a child process forked automatically with the same name of the current listener.

$ ps -ef |grep tns
oracle 2310 1 0 Jul 17 ? 72:00 /oracle/ora10g/db/bin/tnslsnr oprem -inherit
oracle 6573 2310 0 14:19:23 ? 0:00 /oracle/ora10g/db/bin/tnslsnr oprem –inherit

I killed the child process and then reloaded the listener. Phew!!...it worked.The users were able to connect to the database now.

Later I found that this is an Oracle bug (No.4518443).

Some of the possible solutions for the above problem are:

1. Kill the child process using kill command and then reload the listener process.

2. Add the following parameter in the listener.ora file and restart the listener process.

SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER_NAME=OFF

Where listener_name is the listener name (here it would be oprem)

3. Apply Patch.

4. Rename the ons.config file and restart the listener. (path:$ORACLE_HOME/opmn/conf)

This problem is fixed in 10.2.0.3

Note: If you add the above parameter for a listener.ora in RAC setup, then Fast Application Notification (FAN) will not work.

For patch and more details on this refer to metalink note: 340091.1

Hope this was useful to you guys.

Sometimes something goes unnoticed...

ORA-01194: File 5 needs media recovery to be consistent


When you get the above error if you decide that you have to recover the datafile (or database), think twice. Always you may not have to recover the file. I had faced this scenario in one of the client places. Though the initial plan was to restore the datafile from the backup and recover, I had to change my mind.

Now the scenario…I was informed that a guy at the client site had restarted the database (on windows) and they could not open the database because of the datafile inconsistency. After some searching I could find that the problem was with the backup process!!

It happened so that when the hot backup (user-managed) of the database was being taken, that guy had shutdown the database. Now, while restarting the database a particular datafile was asking for media recovery. When checked I found that the datafile was still in backup mode. i.e. the database was shutdown when the backup was going on. Now, the solution is simple. Mount the database and use alter database datafile 5 end backup command and open the database. There is absolutely no need to recover the datafile!! J

So what exactly happened here?!? Well, nothing strange happened. It is a normal oracle behavior, nothing else. When the instance was restarted, the datafile which was in the backup mode will look old because the datafile header is freezed with older SCN. So it will (and it shouldJ) ask for recovery. That’s it!

Generally on unix servers if you issue normal shutdown commands (except abort!!) on the database when the hot backup is going on, shutdown wont happen. It will throw an error saying that the datafile is in backup mode. So there is very less chance of facing this scenario on unix servers.

But in windows, it is general practice to shutdown the database using the OracleService on the services window. In this case even if the backup is happening database goes down with shutdown abort command internally. You won’t come to know that shutdown abort has happened.

You can even face this scenario when the instance crashes during backup (hot backup).You try to restart the instance without knowing that backup was active during the instance crash and end up with the same error.

So, don’t panic… Take it easy!!!