logging into machine and grep the alert log, found the ora-600; database can be opened for a while (about 2 minutes) and then terminate.

Doing block recovery for file 7 block 15312
Block recovery from logseq 147867, block 3707 to scn 3789052035
Recovery of Online Redo Log: Thread 1 Group 6 Seq 147867 Reading mem 0
Mem# 0: /redolog1/dbname/redo06.log
Block recovery stopped at EOT rba 147867.3776.16
Block recovery completed at rba 147867.3776.16, scn 0.3789052035
Thu Dec 29 21:35:26 2011
Completed: alter database open
Thu Dec 29 21:35:29 2011
Starting background process CJQ0
Thu Dec 29 21:35:29 2011
CJQ0 started with pid=45, OS id=192
Thu Dec 29 21:35:29 2011
Sweep Incident[95434]: completed
Errors in file /oracle/diag/diag/rdbms/dbnameg/dbname/trace/dbname_smon_29913.trc:
SVR4 Error: 2: No such file or directory
Additional information: 3
ORACLE Instance dbname (pid = 13) – Error 600 encountered while recovering transaction (203, 27) on object 79208.
Errors in file /oracle/diag/diag/rdbms/dbnameg/dbname/trace/dbname_smon_29913.trc:
ORA-00600: internal error code, arguments: [17182], [0xFFFFFFFF7B81D750], [], [], [], [], [], []
Errors in file /oracle/diag/diag/rdbms/dbnameg/dbname/trace/dbname_smon_29913.trc (incident=95435):
ORA-00600: internal error code, arguments: [KSMFPG2], [0xFFFFFFFF7B81D000], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [17182], [0xFFFFFFFF7B81D750], [], [], [], [], [], []
Incident details in: /oracle/diag/diag/rdbms/dbnameg/dbname/incident/incdir_95435/dbname_smon_29913_i95435.trc
Thu Dec 29 21:35:32 2011
Trace dumping is performing id=[cdmp_20111229213532]
Thu Dec 29 21:35:33 2011
Fatal internal error happened while SMON was doing active transaction recovery.
Errors in file /oracle/diag/diag/rdbms/dbnameg/dbname/trace/dbname_smon_29913.trc:
ORA-00600: internal error code, arguments: [KSMFPG2], [0xFFFFFFFF7B81D000], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [17182], [0xFFFFFFFF7B81D750], [], [], [], [], [], []
SMON (ospid: 29913): terminating the instance due to error 474
Instance terminated by SMON, pid = 29913

from the alert log I found there are corruption at block level, but oracle actualy can recovered it automatically, but it seems when there is undo application for a transaction got stuck due to block corrupt at the other block.

find the object name as informed by alertlog:

select owner,object_name from dba_objects where object_id=79208;

now time to scan all the logical corruption by rman validate command:

run {
allocate channel d1 type disk;
backup check logical validate database;
release channel d1;
}

confirmed there are 2 block corruption on the table:

Error backing up file 7, block 15312: logical corruption
Error backing up file 7, block 18714: logical corruption

another check by dbms_repair to investigate the real corruption, and marked it as corrupt:
declare
begin
dbms_repair.admin_tables(
table_name => ‘REPAIR_TABLE_99′,
table_type => dbms_repair.repair_table,
action => dbms_repair.create_action,
tablespace => ‘USERS’);
end;
/
select owner,object_name,object_type
from dba_objects
where object_name like ‘%REPAIR_TABLE%’;

declare
begin
dbms_repair.admin_tables(
table_name => ‘ORPHAN_REP_TBL_99′,
table_type => dbms_repair.orphan_table,
action => dbms_repair.create_action,
tablespace => ‘USERS’);
end;
/

select owner,object_name,object_type
from dba_objects
where object_name like ‘%ORPHAN%’;

set serveroutput on

declare
rpr_count int;
begin
rpr_count :=0;

dbms_repair.check_object(
schema_name => ‘TEST’,
object_name => ‘TBL1′,
repair_table_name => ‘REPAIR_TABLE_99′,
corrupt_count => rpr_count);

dbms_output.put_line(‘rep cnt : ‘||to_char(rpr_count));
end;
/

select object_name,block_id,corrupt_type,marked_corrupt,corrupt_description,repair_description from repair_table_99;

OBJECT_NAME BLOCK_ID CORRUPT_TYPE MARKED_COR
—————————— ———- ———— ———-
CORRUPT_DESCRIPTION
——————————————————————————–
REPAIR_DESCRIPTION
——————————————————————————–
TBL1 15312 6130 FALSE
Block Checking: DBA = 29375440, Block Type = KTB-managed data block
data header at 0x3d85b007c
kdbchk: fseo(223) < fsbo(282)
mark block software corrupt
TBL1 18714 6110 FALSE
Block Checking: DBA = 29378842, Block Type = KTB-managed data block
data header at 0x3d756e0ac
kdbchk: the amount of space used is not equal to block size
used=7403 fsc=0 avsp=769 dtl=8016
mark block software corrupt

mark those block as corrupted:

declare
l_fix_count int;
begin
l_fix_count:=0;

dbms_repair.fix_corrupt_blocks(
schema_name => ‘TEST’,
object_name => ‘TBL1′,
object_type => dbms_repair.table_object,
repair_table_name => ‘REPAIR_TABLE_99′,
fix_count => l_fix_count);

dbms_output.put_line(‘fix cnt : ‘||to_char(l_fix_count));
end;
/

select object_name,block_id,corrupt_type,marked_corrupt,corrupt_description,repair_description from repair_table_99;

OBJECT_NAME BLOCK_ID CORRUPT_TYPE MARKED_COR
—————————— ———- ———— ———-
CORRUPT_DESCRIPTION
——————————————————————————–
REPAIR_DESCRIPTION
——————————————————————————–
TBL1 15312 6130 TRUE
Block Checking: DBA = 29375440, Block Type = KTB-managed data block
data header at 0x3d85b007c
kdbchk: fseo(223) < fsbo(282)
mark block software corrupt

TBL1 18714 6110 TRUE
Block Checking: DBA = 29378842, Block Type = KTB-managed data block
data header at 0x3d756e0ac
kdbchk: the amount of space used is not equal to block size
used=7403 fsc=0 avsp=769 dtl=8016
mark block software corrupt

now time to recover the table
skip corrupted block from reading, and get the good data:

alter session set event=’10231 trace name context forever, level 10′;
create table test.tmp as select * from test.tbl1;

ok done, now the application guy who have to look into the data and fix them
have a nice sleep then :)

how to execute:
1. to restore & recover until all archivelog in the backup.
./restore_arc.sh <full_path_rman_backup_log>

2. to restore & recover until all archivelog (until cancel).
./restore_arc_last.sh <full_path_rman_backup_log>

has been tested on linux env, you may use this script for your needs and please take your own risk :)

cat restore_arc.sh

#!/bin/bash
. /home/ujang/.bashrc

SCRPATH=/home/ujang/docs/orascript/backup-recovery/rman
LOGPATH=/home/ujang/docs/orascript/backup-recovery/rman
TAG=`date +%Y%m%d`
CMDFILE=${SCRPATH}/restore.txt
LOGFILE=${LOGPATH}/restore_${TAG}.log
RMANLOG=$1
cd $SCRPATH
export ORACLE_HOME=/apps/oracle/product/9.2
export ORACLE_SID=PROD
./restore.pl $RMANLOG spfile=NO lastarc=NO
##$ORACLE_HOME/bin/rman target / nocatalog cmdfile=$CMDFILE msglog $LOGFILE

cat restore_arc_last.sh

#!/bin/bash
. /home/ujang/.bashrc

SCRPATH=/home/ujang/docs/orascript/backup-recovery/rman
LOGPATH=/home/ujang/docs/orascript/backup-recovery/rman
TAG=`date +%Y%m%d`
CMDFILE=${SCRPATH}/restore.txt
LOGFILE=${LOGPATH}/restore_${TAG}.log
RMANLOG=$1
cd $SCRPATH
export ORACLE_HOME=/apps/oracle/product/9.2
export ORACLE_SID=PROD
./restore.pl $RMANLOG spfile=NO lastarc=YES
##$ORACLE_HOME/bin/rman target / nocatalog cmdfile=$CMDFILE msglog $LOGFILE

cat restore.pl

#!/usr/bin/perl
my @array = qq[input archive log thread];
my @match = (“sequence=”,”thread=”,”DBID=”,”piece handle=”,”CONTROL”,”SPFILE”);
my @seqid;
my $thread=0;
my $dbid=0;
my $ctlfile=”XXX”;
my $initfile=”XXX”;
my $orahome=”XXX”;
my $orasid=”X”;
foreach $key (keys(%ENV)) {
if ($orahome eq “XXX”) {
if ($key eq “ORACLE_HOME”) {
$orahome=$ENV{$key};
}
}
if ($orasid eq “X”) {
if ($key eq “ORACLE_SID”) {
$orasid=$ENV{$key};
}
}
}

if ($orahome eq “XXX” || $orasid eq “X”)
{
print “environment variable ORACLE_HOME dan ORACLE_SID harus diset!!\n”;
exit 0;
}

if ($ARGV[0] eq “help=yes” || $ARGV[0] eq “help=y” || $ARGV[0] eq “-h” || $ARGV[0] eq “–h”)
{
print “Cara penggunaan :\n”;
print “1. anda harus berada pada directory dimana file ini dijalankan, misalnya cd /oradata/rman/scr”;
print “2. ketik \n”;
print ” recover spfile=YES|NO\n”;
print ” misal: \n”;
print ” recover /oradata/rman/log/rman_bkp_20110504.log spfile=NO\n”;
exit 0;
}

my $rmanlog=$ARGV[0];
my $spfileloc=$ARGV[1];
my $lastarc=$ARGV[2];
my $spfileloc=substr($spfileloc,index($spfileloc,”=”,0)+1,3);
my $lastarc=substr($lastarc,index($lastarc,”=”,0)+1,3);

if (length($rmanlog)<=0)
{
print “anda harus passing parameter, gunakan restore.sh -h untuk bantuan\n”;
exit 0;
}

open(OLD, “<”, $rmanlog) or die “can’t open $rmanlog”;
while ()
{
my $line = $_;
if ($dbid == 0) {
my $seq = (split /$match[2]/, $line)[1];
$dbid=substr($seq,0,index($seq,”)”,0));
}
if ($ctlfile eq “XXX”) {
my $seq = (split /^$match[3]/, $line)[1];
if($seq =~ m/$match[4]/i) {
$ctlfile=substr($seq,0,index($seq,chr(32),0));
}
}

if ($initfile eq “XXX”) {
my $seq = (split /$match[3]/, $line)[1];
if($seq =~ m/$match[5]/i) {
$initfile=substr($seq,0,index($seq,chr(32),0));
}
}
if ($line =~ /^@array/) {
my $seq = (split /$match[0]/, $line)[1];
push @seqid, substr($seq,0,index($seq,chr(32),0));
if ($thread == 0) {
my $seq = (split /$match[1]/, $line)[1];
$thread=substr($seq,0,index($seq,chr(32),0));
}
}
}
@seqid=sort(@seqid);
my $seqmax = @seqid;
close(OLD);

system(“rm -f restore.txt”);

($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
$year=$year+1900;
$new=”restore.txt”;
open(NEW, “>”, $new) or die “can’t open $new: $!”;
if ($spfileloc eq “YES” or $spfileloc eq “yes”) {
print NEW “host ‘mv $orahome/dbs/spfile$orasid\.ora $orahome/dbs/spfile$orasid\_$year$mon$mday$hour$min\.ora’;\n” or die “can’t write $new: $!”;
}
print NEW “set dbid=$dbid;\n” or die “can’t write $new: $!”;
print NEW “startup force nomount;\n” or die “can’t write $new: $!”;
if ($spfileloc eq “YES” or $spfileloc eq “yes”) {
print NEW “restore spfile to ‘$orahome/dbs/spfile$orasid\.ora’ from ‘$initfile’;\n” or die “can’t write $new: $!”;
}
print NEW “shutdown abort;\n” or die “can’t write $new: $!”;
print NEW “set dbid=$dbid;\n” or die “can’t write $new: $!”;
print NEW “startup nomount;\n” or die “can’t write $new: $!”;
print NEW “restore controlfile from ‘$ctlfile’;\n” or die “can’t write $new: $!”;
print NEW “alter database mount;\n” or die “can’t write $new: $!”;
print NEW “restore database;\n” or die “can’t write $new: $!”;
print NEW “restore archivelog from sequence $seqid[0] until sequence $seqid[$seqmax-1];\n” or die “can’t write $new: $!”;

if ($lastarc eq “YES” or $lastarc eq “yes”) {
print NEW “sql ‘RECOVER DATABASE UNTIL CANCEL USING BACKUP CONTROLFILE;’;\n” or die “can’t write $new: $!”;
} else {
print NEW “recover database until logseq $seqid[$seqmax-1] thread $thread;\n” or die “can’t write $new: $!”;
}
print NEW “sql ‘alter database open resetlogs’;\n” or die “can’t write $new: $!”;
close(NEW) or die “can’t close $new: $!”;

From AWR, the insert statement on a BLOB column can reach 101 seconds, hence makes database call from application server to database slower. Log in to the system and looking for v$session_wait, found a session who execute ‘rollback force’ and hung, I decided to kill this problematic session.

By querying v$session_wait sampled every 3 seconds, found HW – contention from the P3, decode to the file number and block number:

select DBMS_UTILITY.DATA_BLOCK_ADDRESS_FILE(771921779) FILE#,
DBMS_UTILITY.DATA_BLOCK_ADDRESS_BLOCK(771921779) BLOCK#
from dual;

What is the segments which contending on HW:

select owner, segment_type, segment_name
from dba_extents
where file_id = 184
and 169843 between block_id and block_id + blocks - 1; 

Internal application server engine saves state of every transaction to BLOB, and this above query result point to that BLOB. Insertion to this column will wait and finish after 101 seconds. 128 Hash partition on this segments can help at least for a while, and luckily there is no such documentation on the NET about how the best way to partition this table, this table has a column using unix date.

set serverout on size 5555
ALTER SYSTEM SET RESOURCE_MANAGER_PLAN ='';
exec dbms_resource_manager.clear_pending_area();
exec dbms_resource_manager.create_pending_area();
exec dbms_resource_manager.DELETE_PLAN('PLAN_OLTP');
exec dbms_resource_manager.DELETE_CONSUMER_GROUP('GROUP_OLTP');
exec DBMS_RESOURCE_MANAGER.VALIDATE_PENDING_AREA;
exec DBMS_RESOURCE_MANAGER.SUBMIT_PENDING_AREA();
begin
dbms_resource_manager.clear_pending_area();
dbms_resource_manager.create_pending_area();
dbms_resource_manager.create_consumer_group(consumer_group=>'GROUP_OLTP',comment=>'oltp rsrc');
dbms_resource_manager.create_plan(plan=>'PLAN_OLTP',comment=>'batch plan');
dbms_resource_manager.create_plan_directive(plan=>'PLAN_OLTP',group_or_subplan=>'GROUP_OLTP',comment=>'plan dir night',cpu_p1=>80,parallel_degree_limit_p1=>16);
dbms_resource_manager.create_plan_directive(plan=>'PLAN_OLTP',group_or_subplan=>'OTHER_GROUPS',comment=>'plan dir night',cpu_p3=>80,parallel_degree_limit_p1=>8);
dbms_resource_manager.validate_pending_area();
dbms_resource_manager.submit_pending_area();
end;
/
commit
/
begin
dbms_resource_manager_privs.grant_switch_consumer_group(grantee_name=>'SCOTT',consumer_group=>'GROUP_OLTP',grant_option=>false);
dbms_resource_manager_privs.grant_switch_consumer_group(grantee_name=>'SH',consumer_group=>'GROUP_OLTP',grant_option=>false);
dbms_resource_manager.set_initial_consumer_group(user=>'SCOTT',consumer_group=>'GROUP_OLTP');
dbms_resource_manager.set_initial_consumer_group(user=>'SH',consumer_group=>'GROUP_OLTP');
end;
/
alter system set resource_manager_plan=PLAN_OLTP scope=both
/
alter session set resource_manager_plan=PLAN_OLTP scope=both
/

set lines 200
COL consumer_group      FORMAT A24  HEADING 'Consumer Group'
COL cpu_method          FORMAT A18  HEADING 'CPU Method'
COL status              FORMAT A10  HEADING 'Status'
COL mandatory           FORMAT A06  HEADING 'Manda-|tory?'
COL comments            FORMAT A32  HEADING 'Comments'
SELECT
     consumer_group
    ,cpu_method
    ,status
    ,mandatory
    ,comments
  FROM dba_rsrc_consumer_groups;

COL plan                      FORMAT A18  HEADING 'Resource Plan'
COL num_plan_directives       FORMAT 9999 HEADING '# of|Plan|Dirs'
COL cpu_method                FORMAT A18  HEADING 'CPU Method'
COL active_sess_pool_mth      FORMAT A32  HEADING 'Active|Session|Pool|Method'
COL parallel_degree_limit_mth FORMAT A32  HEADING 'Parallel|Limit|Method'
COL queueing_mth              FORMAT A18  HEADING 'Queueing|Method'
COL status                    FORMAT A10  HEADING 'Status'
COL mandatory                 FORMAT A06  HEADING 'Manda-|tory?'
COL comments                  FORMAT A32  HEADING 'Comments'
SELECT
     plan
    ,num_plan_directives
    ,cpu_method
    ,active_sess_pool_mth
    ,parallel_degree_limit_mth
    ,queueing_mth
    ,status
    ,mandatory
  FROM dba_rsrc_plans
;

COL plan                      FORMAT A18  HEADING 'Resource Plan'
COL group_or_subplan          FORMAT A12  HEADING 'Group or|SubPlan'
COL type                      FORMAT A15  HEADING 'Type'
COL cpu_p1                    FORMAT 999  HEADING 'CPU|1 %'
COL cpu_p2                    FORMAT 999  HEADING 'CPU|2 %'
COL cpu_p3                    FORMAT 999  HEADING 'CPU|3 %'
COL status                    FORMAT A08  HEADING 'Status'
COL mandatory                 FORMAT A06  HEADING 'Manda-|tory?'
COL comments                  FORMAT A24  HEADING 'Comments'

SELECT
     plan
    ,group_or_subplan
    ,type
    ,cpu_p1
    ,cpu_p2
    ,cpu_p3
    ,status
  FROM dba_rsrc_plan_directives
 WHERE plan = 'PLAN_OLTP'
;

COL initial_rsrc_consumer_group  FORMAT A24  HEADING 'Resource|Consumer|Group'
COL username                     FORMAT A12  HEADING 'User Name'
SELECT
	 initial_rsrc_consumer_group,
    username
  FROM dba_users
 ORDER BY 1,2;

SELECT se.sid sess_id, co.name consumer_group,
 se.state, se.consumed_cpu_time cpu_time, se.cpu_wait_time, se.queued_time
 FROM v$rsrc_session_info se, v$rsrc_consumer_group co
 WHERE se.current_consumer_group_id = co.id;

select username,resource_CONSUMER_GROUP,count(*) from v$session group by username,resource_CONSUMER_GROUP;

The query is really simple just querying a table filtered by combination of columns, the table has 2 indexes (each 1 column) which is as part of where clause. from the plan, oracle seems to deal with just only cost, ya it is true cost is cheaper than by scanning only 1 index, but IO cost would likely double (scanning 2 indexes, bitmap and operation and than conver bitmap to rowid).

----------------------------------------------------------------------------------------------------
| Id  | Operation                        | Name            | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                 |                 |     1 |   112 |    66   (2)| 00:00:01 |
|*  1 |  TABLE ACCESS BY INDEX ROWID     | T_4             |     1 |   112 |    66   (2)| 00:00:01 |
|   2 |   BITMAP CONVERSION TO ROWIDS    |                 |       |       |            |          |
|   3 |    BITMAP AND                    |                 |       |       |            |          |
|   4 |     BITMAP CONVERSION FROM ROWIDS|                 |       |       |            |          |
|*  5 |      INDEX RANGE SCAN            | IDX_1           | 25237 |       |     7   (0)| 00:00:01 |
|   6 |     BITMAP CONVERSION FROM ROWIDS|                 |       |       |            |          |
|*  7 |      INDEX RANGE SCAN            | IDX_0           | 25237 |       |    40   (0)| 00:00:01 |
----------------------------------------------------------------------------------------------------

here by force to use index 1,

-----------------------------------------------------------------------------------------------
| Id  | Operation                   | Name            | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |                 |     1 |   112 |   501   (1)| 00:00:07 |
|*  1 |  TABLE ACCESS BY INDEX ROWID| T_4             |     1 |   112 |   501   (1)| 00:00:07 |
|*  2 |   INDEX RANGE SCAN          | IDX_1           |  3801 |       |     7   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------------

it seems the cost is higher than bitmap conversion, but from the real execution time the second one is really faster. it seems cost calculation on the first plan is misleading.

got error that capture builder has error, OK well done.
I started manually the capture process and after 10 minutes
PMON always kill capture builder process.

PMON has hidden parameter by default 10 minutes
to scan dead process.

no chance to do workaround as of metalink,
the note will state that database should be upgraded
to atleast 10.2.0.3…phiew

dong sync manualy from the latest applied SCN
and then skip capture SCN to start from
the last sync SCN :(

exec dbms_capture_adm.stop_capture(capture_name => 'AQ_CAPTURE');
delete from system.logmnr_restart_ckpt$ where spare1 < nnnn ;
exec dbms_capture_adm.alter_capture(capture_name => 'AQ_CAPTURE', first_scn => nnnn, start_scn => nnnn);
exec dbms_capture_adm.start_capture(capture_name => 'AQ_CAPTURE');

by looking at the alert log, luckily only objects belongs to sysman schema
got warning…which those are not captured.

- log buffer space issue, due to lack of log_buffer memory setting – free buffer waits, due to lack of db_cache_size memory setting

- buffer busy waits, this issue related to concurrency and transactions at the same table which being inserted, also sometimes too many indexes or indexes which has high clustering factors will lead to this wait event issue.

in fact if we doing direct path inserts, some issues can be avoided, such as free buffer waits, but wait event that related to redo is intact. in 10.2 there is a feature called asynchronous commit, which can alleviate redo wait event issue, but index issue still carryon even we using direct path.

there is trick approach which depends on your application behavior to boost insert performance.

- temporary table

- asynchronous commit (but be aware of recovery and data consistency)

- direct path

I believe that after configuring oracle streams environment 99% we have to monitor for the future ‘fun’ :)

1. Configuration

Configure oracle streams environment is quite handy and easy for us. Oracle given us a package to generate necessary scripts that needed for configuration to take place. (I have used this facility in oracle 10gR2). Streams also can communicate to cross platform with cross oracle version…this is the feature that really useful for my life :)

2. The game starting now

After configuration has been done. Error queue always happening at destination database, when user updating data let say master form, even when application server which point to destination database was killed, sadly synchronization was not happened, it always give me error message “ORA-01403: no data found”.

3. Printing error

Look at stream documentation, we can find script to print all error message with particular data either by specifying local transaction ID or can grab all from the DBA_APPY_ERROR. The script  simply extract and print LCR vertically.

4. Resolve error

Let me check each error and do manual resolution to know the pattern of the erroneous  by updating data on destination, using this method until my hand cannot accept again and bore to press the keyboard :)

5. Manual error handling

Based on the pattern that I got from previous step, I developed pl/sql package to handle the error in the streams destination due to apply process is in the destination. This package simply extract LCR and find the row in the particular table’s row updating it to match with the OLD LCR’s value. Interestingly this approach can handle for some tables (actually table with few column). For a lot of column’s tables this solution cannot accomplished.

6. Trace apply error queue

To trace streams apply error to see what is going on:

alter session set events=’10308 trace name context forever, level 8′ ;
alter session set events=’26700 trace name context forever, level 15999′;
alter session set events=’1403 trace name errorstack level 1′;

Find interesting for the tables which has lot of column, which is not get printed by print_lcr as of point 3, event if I changed my packaged to accommodate the hidden column, it always skipped by print_lcr (my pl/sql also utilize print_lcr to extract LCR), so… is the game over? :)

7. Final solution

Finally by searching document for the comparability between OLD value in the LCR and the existing record at the destination’s table found interesting package to do it…..

DECLARE
cols DBMS_UTILITY.LNAME_ARRAY;
BEGIN
cols(1):=’COLUMN1′;
cols(2):=’COLUMN2′;
cols(3):=’COLUMN3′;
DBMS_APPLY_ADM.COMPARE_OLD_VALUES(
object_name  => ‘SCHEMA.TABLENAME’,
column_table => cols,
operation    => ‘*’,
compare      => false);
END;

The above package really safe my life :), it simply will not performs comparation for the listed columns, since streams when applying LCR will compare all columns even PK has been added to the supplemental log. This behavior as I understand aimed to do synchronization for tables without PK, Unique indexes.

now the game paused…waiting for next fun :)

Ujang| Indonesia | Jakarta

from the 10046 trace raw contents, we actually can do analysis, but internal recursive sql sometimes makes our live harder :)

metalink note 224270.1 give us a tool to analyze the trace easily, but again got problem when I need to know the history by mean of step-by-step application make database calls. I cannot see control transaction on the trace file.

we can only find “XCTEND rlbk=<number>, rd_only=<number>”, based on metalink note 371984.1:

rlbk 1 if a rollback was performed, 0 if no rollback (commit).
rd_only 1 if transaction was read only, 0 if changes occurred.
the harder part of the raw data of trace 10046 those transaction control is reside at the middle of cursor parsing :(

FETCH #10:c=0,e=35,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,tim=49665367470
WAIT #10: nam=’SQL*Net message from client’ ela= 472 driver id=1952673792 #bytes=1 p3=0 obj#=-1 tim=49665367990
STAT #10 id=1 cnt=1 pid=0 pos=1 obj=0 op=’FAST DUAL  (cr=0 pr=0 pw=0 time=1 us)’
WAIT #12: nam=’SQL*Net message to client’ ela= 1 driver id=1952673792 #bytes=1 p3=0 obj#=-1 tim=49665368337
WAIT #12: nam=’SQL*Net message from client’ ela= 376 driver id=1952673792 #bytes=1 p3=0 obj#=-1 tim=49665368739
XCTEND rlbk=0, rd_only=1
WAIT #0: nam=’SQL*Net message to client’ ela= 1 driver id=1952673792 #bytes=1 p3=0 obj#=-1 tim=49665368872
*** 2008-08-14 12:06:51.980
WAIT #0: nam=’SQL*Net message from client’ ela= 53635836 driver id=1952673792 #bytes=1 p3=0 obj#=-1 tim=49719004740
WAIT #0: nam=’SQL*Net message to client’ ela= 3 driver id=1952673792 #bytes=1 p3=0 obj#=-1 tim=49719005053
WAIT #0: nam=’SQL*Net message from client’ ela= 6340 driver id=1952673792 #bytes=1 p3=0 obj#=-1 tim=49719011426

even when I tried to make a trick by generating new cursor parsing by replacing the XCTEND bla..bla…bla with something like this:

:%s/XCTEND rlbk=0, rd_only=1/^M=====================^MPARSING IN CURSOR #10 len=27 dep=0 uid=57 oct=3 lid=57 tim=49665366604 hv=824003789 ad=’ca1c6268′^Mcommit

———————————————

Note to get ^M use following
Hold down the Ctrl, and press v m
DO NOT USE Shift+6 and Shift+M
———————————————

still from the trace analyzer output give me a confusing (the order of transaction control still doen’t make any sense – it totally wrong order).

anyway good tools to analyze execution plan, due to the parameter that makes CBO work are in one place with the explain plan gathered.

Ujang| Indonesia | Jakarta