linux oracle rac oprocd reboot

2010.02.07 7:14 下午 »Author: bosonmaster »
最近在客户升级10.2.0.4外加，CRS PSU2 DB PSU3，升级完后，系统不定时发生重启，发生重启期间都有大文件操作，根据OSW记录的系统状态情况，当时内存剩余比较少，但是又有某些重启时刻系统重启时，内存也有不少剩余，每次发生系统重启的时候，系统收到的信息都是SysRq:reseting ，然后系统就重启了，中间测试了很多种情况，包括写在PSU2 压力测试都是如此
10.2.0.4比10.2.0.3在监控节点间的状态的时候，特别LINUX平台，多了一个oprocd进程，用于检测节点状态，如果发生如下情况，就会重启系统
A problem detected by the OPROCD process. This can be caused by 4 things:
 
1) An OS scheduler problem.
2) The OS is getting locked up in a driver or hardware.
3) Excessive amounts of load on the machine, thus preventing the scheduler from
behaving reasonably.
4) An Oracle bug.
 
并且如果是OPROCD进程导致系统重启，那么会看到SysRq:reseting信息，在如下位置有oprocd日志
/etc/oracle/oprocd or /var/opt/oracle/oprocd
oprocd默认启动oprocd run -t 1000 -m 500 单位为毫秒，默认允许延迟时间为1S，如果1S内没有响应，那么0.5秒后重启系统，也就说OPROCD检测问题后，允许的延迟时间是1S，1S后 延迟后，0.5秒后OPROCD进程就会掉用脚本重启系统，我们这次遇到的都是SysRq:reseting，所以是OPROCD进程重启的系统，我们测试了很多次。
10.2.0.4以前LINUX 上是通过hangcheck timer模块来检测的，默认的延时
    *  9i: Assuming the default setting of "oracm misscount" is set to 220 seconds:
      hangcheck_tick=30 hangcheck_margin=180 hangcheck_reboot=1
    * 10g/11g: Assuming the default setting of "CSS misscount" is set to either 30 or 60 seconds:
      hangcheck_tick=1 hangcheck_margin=10 hangcheck_reboot=1
还有以下几个进程可能重启系统
 
1 ocssd进程
 
ocssd进程引起节点重启的时候，会有类似如下日志：
Rebooted for Cluster Integrity 在系统日志里，比如linux /var/log/message hp-ux syslog,CRS日志里有如下类似记录
Sun: /var/adm/messages
HP-UX: /var/adm/syslog/syslog.log
Tru64: /var/adm/messages
Linux: /var/log/messages
IBM: /bin/errpt -a > messages.out
 
 
Network failure or latency between nodes. It would take at least 30 consecutive
missed checkins to cause a reboot, where heartbeats are issued once per second.
 
Example of missed checkins in the CSS log:
 
WARNING: clssnmPollingThread: node <node> (1) at 50% heartbeat fatal, eviction in 29.100 seconds
WARNING: clssnmPollingThread: node <node> (1) at 75% heartbeat fatal, eviction in 14.960 seconds
WARNING: clssnmPollingThread: node <node> (1) at 75% heartbeat fatal, eviction in 13.950 seconds
 
The first thing to do is find out if the missed checkins ARE the problem or are a
result of the node going down due to other reasons. Check the messages file to see
what exact time the node went down and compare it to the time of the missed checkins.
 
- If the messages file reboot time < missed checkin time then the node eviction was
likely not due to these missed checkins.
 
- If the messages file reboot time > missed checkin time then the node eviction was
likely a result of the missed checkins.
 
 
- Problems writing to or reading from the CSS voting disk.
 
Example of a voting disk problem in the CSS log:
 
ERROR: clssnmDiskPingMonitorThread: voting device access hanging (160008 miliseconds)
 
- Lack of CPU resources. There are some situations which will appear to be missed
heartbeat issues, however turn out to be caused by a user running a high
sustained load average. When a machine gets too heavily loaded, the scheduling
reliability can be bad. This could cause CSS to not get scheduled in time and
thus CSS cannot get its work done. If this happens, the node is declared
not-viable for cluster work and is evicted.
 
- A problem with the executables (for example, removing CRS Home files)
 
- Misconfiguration of CRS. Possible misconfigurations:
 
- Wrong network selected as the private network for CRS (confirm with CSS log,
/etc/hosts, and ifconfig output). Make sure it is not the public or VIP
address. Look in the CSS log for strings like...
clsc_listen: (*) Listening on
(ADDRESS=(PROTOCOL=tcp)(HOST=dlsun2046)(PORT=61196))
 
- Putting the CSS vote file on a Netapp that's shared over some kind of public
network or otherwise excessively loaded/unreliable network. If this is the
case, you are likely to see the following message in the CSS logfile:
 
ERROR: clssnmDiskPingThread(): Large disk IO timeout * seconds.
 
If you ever see this error, then it's important to investigate why the disk
subsystem is unresponsive.
 
See section 3.2 for information on how to correct common misconfiguration
problems.
 
- Killing the "init.cssd fatal" process or "ocssd" process.
 
- An unexpected failure of the OCSSD process, this can be caused by any of the
above issues.
 
- An Oracle bug. Known bugs that can cause CSS reboots:
 
 
3 第三个进程如下 oclsomon
A problem detected by the OCLSOMON process. This can be caused by 4 things:
 
1) A thread(s) within the CSS daemon hung.
2) An OS scheduler problem.
3) Excessive amounts of load on the machine, thus preventing the scheduler from
behaving reasonably.
4) An Oracle bug.
 
更加信息的资料可以看文档：265769.1 726833.1 395878.1

oracle No Comments

vip gateway 10.2.0.3 10.2.0.4

2010.01.01 9:11 下午 »Author: bosonmaster »
在升级数据库从10.2.0.3到10.2.0.4 后，VIP服务死活启动不起来，检查了半天也没发现问题，最后发现没有设置缺省网关，因为这个库安装的比较早，所以没设，而且10.2.0.3运行也什么问题，设置缺省网关后就好了，VIP启动起来。看来10.2.0.3和10.2.0.4 RACGVIP脚本改动还是有关的，还是那句话，越是认为不会出问题的地方，越是出问题，新年第一天。还不错运气

oracle No Comments

vip &ipc

2010.01.01 9:08 下午 »Author: bosonmaster »
今天在客户这升级数据库从10.2.0.3到10.2.0.4，升级过程基本很顺利，可是在测试拔网线，VIP切换时速度比较慢，去METALINK搜索了下，发现如下提示：
Cause
This problem is caused by the first address in the listener.ora configuration being an address that uses the TCP protocol.
 
In this circumstance, when a network cable is pulled, "lsnrctl stop" listener has to wait for TCP timeout before it can check next address. On the Solaris platform, TCP timeout is defined by tcp_ip_abort_cinterval with a default value of 180000 (3 minutes).   That is why shutting down listener almost took 3.5 minutes. (TCP timeout on other platforms may vary).  The error message "Solaris Error: 145: Connection timed out" in ora.node1.LISTENER_NODE1.lsnr.log also indicates it is waiting for tcp timeout.
 
The listener.ora in this scenario is defined as:
 
 
 
[LISTENER_NODE1 =
 (DESCRIPTION_LIST =
   (DESCRIPTION =
     (ADDRESS_LIST =
       (ADDRESS = (PROTOCOL = TCP)(HOST = node1vip)(PORT = 1521)(IP = FIRST))
     )
     (ADDRESS_LIST =
       (ADDRESS = (PROTOCOL = TCP)(HOST = 10.1.10.100)(PORT = 1521)(IP = FIRST))
     )
     (ADDRESS_LIST =
       (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
     )
   )
 )
Solution
To prevent this, move the IPC address to be the first address for the listener in the listener.ora, eg:
 
LISTENER_NODE1 =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
       (ADDRESS_LIST =
          (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
       )
       (ADDRESS_LIST =
          (ADDRESS = (PROTOCOL = TCP)(HOST = node1vip)(PORT = 1521)(IP = FIRST))
        )
       (ADDRESS_LIST =
           (ADDRESS = (PROTOCOL = TCP)(HOST = 10.1.10.100)(PORT = 1521)(IP = FIRST))
        )
     )
  )
 
 
When lsnrctl tries to stop the listener, it will now connect to the IPC address first, which is available during that time. It will not have to wait for tcp timeout.
 
After the above change, the VIP failover only takes 48 to 50 seconds to complete regardless of the tcp_ip_abort_cinterval setting.
 
Please note, listener.ora files newly created from 10.2.0.3 to 11.1.0.7 should have the IPC protocol as the first address in listener.ora in most cases.  However, if you have upgraded from a previous release, or manually modified/copied over a listener.ora from a previous install, you may not have the IPC protocol as the first address, regardless of your version. Manual modification is required to move IPC protocol to be the first address to avoid the problem described in this note.
 
也就说 IPC协议需要放在监听地址第一列，修改后，我们在测试，从原来2分钟缩减到20多秒，符合应用切换的要求

oracle No Comments

回顾2009年，展望2010年

2009.12.28 9:33 下午 »Author: bosonmaster »
今天收到公司发的生日邮件，发现2009年马上就要过去了，这一年，说忙碌吧，也还可以，技术这一年基本没啥长进，银子也没攒几个，一直梦想的房子也离自己越来越远，这房价就像做了火箭，一个劲的往上冲，国家老是说控制房价，也不知道最后控制到哪去了，地王一个一个接一个，这房价能控制住吗？去年10月和朋友看的一套房子，当时9.6K，现在二期15500，还售罄，今天还看到新闻，说2009年中国农民纯收入超5000，我不知道这是咋算出来，可以去问问农民伯伯们，你们真的纯收入5000了吗？不
管怎么说，日子还要过下去，房子还要继续租下去，不知道在未来的2010年，房价能否下来，先做个梦吧，希望在2010年自己的技术能进步一点，O(∩_∩)O~，也希望自己在个人方面也有所突破。
 
最后祝大家：
 
         元旦快乐，新的一年里健康幸福！

bosn

随笔 2 Comments

kcbz_check_objd_typ_3

2009.12.27 10:41 上午 »Author: bosonmaster »
客户系统linux4 x86_64 10.2.0.3，近日有如下报错：
Errors in file /u01/app/oracle/admin/lzrac/bdump/lzrac2_m001_26960.trc:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [0], [0], [1], [], [], [], []
查看相关METALINK和产生的TRACE文件
TRACLE文件部分内容：
 
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [0], [0], [1], [], [], [], []
Current SQL statement for this session:
SELECT count(*) over () as total_count,        sd_xe_ash_nm.event_name,        sd_xe_ash_nm.event_id,        sd_xe_ash_nm.parameter1 as p1text,        (CASE WHEN (sd_xe_ash_nm.parameter1 is NULL                    OR                    sd_xe_ash_nm.parameter1 = '0')              THEN 0              ELSE 1         END) as p1valid,        sd_xe_ash_nm.parameter2 as p2text,        (CASE WHEN (sd_xe_ash_nm.parameter2 is NULL                    OR                    sd_xe_ash_nm.parameter2 = '0')              THEN 0              ELSE 1         END) as p2valid,        sd_xe_ash_nm.parameter3 as p3text,        (CASE WHEN (sd_xe_ash_nm.parameter3 is NULL                    OR                    sd_xe_ash_nm.parameter3 = '0')              THEN 0              ELSE 1         END) as p3valid,        sd_xe_ash_nm.keh_evt_id,        nvl(xc.class#, 0) as class_num,        sd_xe_ash_nm.wait_class_id,        nvl(xc.keh_id, 0) as keh_ecl_id,        sd_xe_ash_nm.ash_cnt,        sd_xe_ash_nm.tot_wts_diff,        sd_xe_ash_nm.tot_tmo_diff,       sd_xe_ash_nm.tim_wait_diff FROM   ( SELECT sd_xe_ash.*,        evtname.event_name, evtname.wait_class_id,        evtname.parameter1, evtname.parameter2, evtname.parameter3 FROM   ( SELECT sd_xe.*, nvl(ash.cnt, 0) as ash_cnt FROM   ( SELECT nvl(xe.keh_id, 0) as keh_evt_id,        nvl(sd.event_id, xe.event_hash) as event_id,        nvl(sd.tot_wts_diff, 0) as tot_wts_diff,        nvl(sd.tot_tmo_diff, 0) as tot_tmo_diff,        nvl(sd.tim_wait_diff, 0) as tim_wait_diff FROM   ( SELECT endsn.event_id as event_id,        (endsn.total_waits - nvl(begsn.total_waits,0)) as tot_wts_diff,        (endsn.total_timeouts - nvl(begsn.total_timeouts,0))        as tot_tmo_diff,        (endsn.time_waited_micro - nvl(begsn.time_waited_micro,0))        as tim_wait_diff FROM   ( SELECT end_snap.*          FROM  (SELECT t1.* FROM WRH$_SYSTEM_EVENT t1, WRM$_SNAPSHOT s1   WHERE  t1.dbid = s1.dbid AND t1.instance_number = s1.instance_number     AND  t1.snap_id = s1.snap_id AND s1.bl_moved = 0   UNION ALL   SELECT t2.* FROM WRH$_SYSTEM_EVENT_BL t2, WRM$_SNAPSHOT s2   WHERE  t2.dbid = s2.dbid AND t2.instance_number = s2.instance_number     AND  t2.snap_id = s2.snap_id AND s2.bl_moved <> 0) end_snap          WHERE  end_snap.dbid            = :dbid            and  end_snap.instance_number = :instance_number            and  end_snap.snap_id         = :end_snap ) endsn        LEFT OUTER JOIN        ( SELECT beg_snap.*          FROM  (SELECT t1.* FROM WRH$_SYSTEM_EVENT t1, WRM$_SNAPSHOT s1   WHERE  t1.dbid = s1.dbid AND t1.instance_number = s1.instance_number     AND  t1.snap_id = s1.snap_id AND s1.bl_moved = 0   UNION ALL   SELECT t2.* FROM WRH$_SYSTEM_EVENT_BL t2, WRM$_SNAPSHOT s2   WHERE  t2.dbid = s2.dbid AND t2.instance_number = s2.instance_number     AND  t2.snap_id = s2.snap_id AND s2.bl_moved <> 0) beg_snap          WHERE  beg_snap.dbid            = :dbid            and  beg_snap.instance_number = :instance_number            and  beg_snap.snap_id         = :beg_snap ) begsn        ON endsn.event_id = begsn.event_id  ) sd        FULL OUTER JOIN        X$KEHEVTMAP xe        ON sd.event_id = xe.event_hash  ) sd_xe        LEFT OUTER JOIN        (SELECT a.event_id,                count(*) as cnt         FROM  (SELECT t1.* FROM WRH$_ACTIVE_SESSION_HISTORY t1, WRM$_SNAPSHOT s1   WHERE  t1.dbid = s1.dbid AND t1.instance_number = s1.instance_number     AND  t1.snap_id = s1.snap_id AND s1.bl_moved = 0   UNION ALL   SELECT t2.* FROM WRH$_ACTIVE_SESSION_HISTORY_BL t2, WRM$_SNAPSHOT s2   WHERE  t2.dbid = s2.dbid AND t2.instance_number = s2.instance_number     AND  t2.snap_id = s2.snap_id AND s2.bl_moved <> 0) a         WHERE  a.dbid            =  :dbid           and  a.instance_number =  :instance_number           and  a.snap_id         >  :beg_snap           and  a.snap_id         <= :end_snap           and  a.wait_time       =  0         GROUP BY a.event_id) ash        ON sd_xe.event_id = ash.event_id  ) sd_xe_ash,  WRH$_EVENT_NAME evtname WHERE  evtname.event_id = sd_xe_ash.event_id   and  evtname.event_id > 0   and  evtname.dbid     = :dbid  ) sd_xe_ash_nm,        X$KEHECLMAP xc WHERE  sd_xe_ash_nm.wait_class_id = xc.class_hash ORDER  BY sd_xe_ash_nm.wait_class_id,           sd_xe_ash_nm.tim_wait_diff DESC,           sd_xe_ash_nm.event_id
----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0x111fc7790        10  package body SYS.PRVT_HDM
0x10eaf25d0        16  SYS.WRI$_ADV_HDM_T
0x119d97690      1535  package body SYS.PRVT_ADVISOR
0x119d97690      1618  package body SYS.PRVT_ADVISOR
0x111fc7790       106  package body SYS.PRVT_HDM
描述过程如下：
Oracle Server - Enterprise Edition - Version: 10.2.0.2 to 10.2.0.3
This problem can occur on any platform.
 
Symptoms
Segment Advisor is being used.
 
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [4], [0], [15], [], [], [], []
 
With a stack trace similar to:
 
kgerinv kgeasnmierr kcbassertbd3 kcbz_check_objd_typ kcbzib kcbgtcr ktrget kdirfrs
 
ORA-600 [kcbnew_3] may be reported instead.
 
The PLSQL stack, if there is one, may have SYS.PRVT_ADVISOR or SYS.DBMS_SPACE near the top.
 
 
 
Cause
The cause of this problem has been identified and verified in an Unpublished Bug 4430244.
 
It is caused by the Segment Advisor code which can load blocks for dropped objects into cache as CURRENT leading to subsequent operations seeing an incorrect (old) version of a block.
 
Solution
This bug is fixed in our 10.2.0.4 patchset and 11g Release 1.
 
You can check if a patch is available for your patchset release and O/S environment  Patch 4430244
 
To obtain a patch from MetaLink:
1) Click on Patches.
2) Click on Simple Search.
3) Enter your Patch number : 4430244
4) Select your platform
5) Click Go.
6) Read any applicable notes before downloading, then click the Download button.
 
Note: Please review the Readme file for instructions on how to install the patchset.
 
To avoid the issue in the short-term, turn off Segment Advisor by disabling 'Automatic Segment Advisor Job'.  This is an advisory job so there will be no harm caused to your database by turning it off.
 
If you are encountering this bug, flush the buffer cache with the following command:
SQL> alter system flush buffer_cache;
 
to remove old copies of the block.
If you are using RAC then this will need to be done on all instances.

oracle 1 Comment

Pages (64): 1 2 3 4 » ... Last »

CopyRight ©2007 oralife.cn, All rights reserved.
联系站长bosonmaster(at)gmail.com|QQ联系:859358
Powered by WordPress
京ICP备07026883号