梦想有多远

linux ipcs max sharememory

bosonmaster — Tue, 09 Mar 2010 06:29:38 +0000

最近同事遇到一个LINUX上共享内存段的问题，一般我们会根据系统内存的大小来定义系统支持的最大内存段大小，kernel.shmmax ，检查了这些设置都没问题，系统版本也没啥问题，最后经老熊提醒确认为NUMA的问题
ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 98304 gdm 600 393216 2 dest
0x00000000 1310721 oracle 640 1543503872 31
0x00000000 1343490 oracle 640 2835349504 31
0x00000000 1376259 oracle 640 2835349504 31
0x00000000 1409028 oracle 640 2852126720 31
0x00000000 1441797 oracle 640 2835349504 31
0x1714b88c 1474566 oracle 640 2097152 31
设置了如下参数后就解决了次问题
alter system set "_enable_numa_optimization"=false scope=spfile;
有关NUMA的问题，可以看文档：759565.1

linux oracle rac oprocd reboot

bosonmaster — Sun, 07 Feb 2010 11:14:03 +0000

最近在客户升级10.2.0.4外加，CRS PSU2 DB PSU3，升级完后，系统不定时发生重启，发生重启期间都有大文件操作，根据OSW记录的系统状态情况，当时内存剩余比较少，但是又有某些重启时刻系统重启时，内存也有不少剩余，每次发生系统重启的时候，系统收到的信息都是SysRq:reseting ，然后系统就重启了，中间测试了很多种情况，包括写在PSU2 压力测试都是如此
10.2.0.4比10.2.0.3在监控节点间的状态的时候，特别LINUX平台，多了一个oprocd进程，用于检测节点状态，如果发生如下情况，就会重启系统
A problem detected by the OPROCD process. This can be caused by 4 things:

1) An OS scheduler problem.
2) The OS is getting locked up in a driver or hardware.
3) Excessive amounts of load on the machine, thus preventing the scheduler from
behaving reasonably.
4) An Oracle bug.

并且如果是OPROCD进程导致系统重启，那么会看到SysRq:reseting信息，在如下位置有oprocd日志
/etc/oracle/oprocd or /var/opt/oracle/oprocd
oprocd默认启动oprocd run -t 1000 -m 500 单位为毫秒，默认允许延迟时间为1S，如果1S内没有响应，那么0.5秒后重启系统，也就说OPROCD检测问题后，允许的延迟时间是1S，1S后延迟后，0.5秒后OPROCD进程就会掉用脚本重启系统，我们这次遇到的都是SysRq:reseting，所以是OPROCD进程重启的系统，我们测试了很多次。
10.2.0.4以前LINUX 上是通过hangcheck timer模块来检测的，默认的延时
   * 9i: Assuming the default setting of "oracm misscount" is set to 220 seconds:
   hangcheck_tick=30 hangcheck_margin=180 hangcheck_reboot=1
   * 10g/11g: Assuming the default setting of "CSS misscount" is set to either 30 or 60 seconds:
   hangcheck_tick=1 hangcheck_margin=10 hangcheck_reboot=1
还有以下几个进程可能重启系统

1 ocssd进程

ocssd进程引起节点重启的时候，会有类似如下日志：
Rebooted for Cluster Integrity 在系统日志里，比如linux /var/log/message hp-ux syslog,CRS日志里有如下类似记录
Sun: /var/adm/messages
HP-UX: /var/adm/syslog/syslog.log
Tru64: /var/adm/messages
Linux: /var/log/messages
IBM: /bin/errpt -a > messages.out

Network failure or latency between nodes. It would take at least 30 consecutive
missed checkins to cause a reboot, where heartbeats are issued once per second.

Example of missed checkins in the CSS log:

WARNING: clssnmPollingThread: node <node> (1) at 50% heartbeat fatal, eviction in 29.100 seconds
WARNING: clssnmPollingThread: node <node> (1) at 75% heartbeat fatal, eviction in 14.960 seconds
WARNING: clssnmPollingThread: node <node> (1) at 75% heartbeat fatal, eviction in 13.950 seconds

The first thing to do is find out if the missed checkins ARE the problem or are a
result of the node going down due to other reasons. Check the messages file to see
what exact time the node went down and compare it to the time of the missed checkins.

- If the messages file reboot time < missed checkin time then the node eviction was
likely not due to these missed checkins.

- If the messages file reboot time > missed checkin time then the node eviction was
likely a result of the missed checkins.

- Problems writing to or reading from the CSS voting disk.

Example of a voting disk problem in the CSS log:

ERROR: clssnmDiskPingMonitorThread: voting device access hanging (160008 miliseconds)

- Lack of CPU resources. There are some situations which will appear to be missed
heartbeat issues, however turn out to be caused by a user running a high
sustained load average. When a machine gets too heavily loaded, the scheduling
reliability can be bad. This could cause CSS to not get scheduled in time and
thus CSS cannot get its work done. If this happens, the node is declared
not-viable for cluster work and is evicted.

- A problem with the executables (for example, removing CRS Home files)

- Misconfiguration of CRS. Possible misconfigurations:

- Wrong network selected as the private network for CRS (confirm with CSS log,
/etc/hosts, and ifconfig output). Make sure it is not the public or VIP
address. Look in the CSS log for strings like...
clsc_listen: (*) Listening on
(ADDRESS=(PROTOCOL=tcp)(HOST=dlsun2046)(PORT=61196))

- Putting the CSS vote file on a Netapp that's shared over some kind of public
network or otherwise excessively loaded/unreliable network. If this is the
case, you are likely to see the following message in the CSS logfile:

ERROR: clssnmDiskPingThread(): Large disk IO timeout * seconds.

If you ever see this error, then it's important to investigate why the disk
subsystem is unresponsive.

See section 3.2 for information on how to correct common misconfiguration
problems.

- Killing the "init.cssd fatal" process or "ocssd" process.

- An unexpected failure of the OCSSD process, this can be caused by any of the
above issues.

- An Oracle bug. Known bugs that can cause CSS reboots:

3 第三个进程如下 oclsomon
A problem detected by the OCLSOMON process. This can be caused by 4 things:

1) A thread(s) within the CSS daemon hung.
2) An OS scheduler problem.
3) Excessive amounts of load on the machine, thus preventing the scheduler from
behaving reasonably.
4) An Oracle bug.

更加信息的资料可以看文档：265769.1 726833.1 395878.1

vip gateway 10.2.0.3 10.2.0.4

bosonmaster — Fri, 01 Jan 2010 13:11:33 +0000

在升级数据库从10.2.0.3到10.2.0.4 后，VIP服务死活启动不起来，检查了半天也没发现问题，最后发现没有设置缺省网关，因为这个库安装的比较早，所以没设，而且10.2.0.3运行也什么问题，设置缺省网关后就好了，VIP启动起来。看来10.2.0.3和10.2.0.4 RACGVIP脚本改动还是有关的，还是那句话，越是认为不会出问题的地方，越是出问题，新年第一天。还不错运气

vip &ipc

bosonmaster — Fri, 01 Jan 2010 13:08:17 +0000

今天在客户这升级数据库从10.2.0.3到10.2.0.4，升级过程基本很顺利，可是在测试拔网线，VIP切换时速度比较慢，去METALINK搜索了下，发现如下提示：
Cause
This problem is caused by the first address in the listener.ora configuration being an address that uses the TCP protocol.

In this circumstance, when a network cable is pulled, "lsnrctl stop" listener has to wait for TCP timeout before it can check next address. On the Solaris platform, TCP timeout is defined by tcp_ip_abort_cinterval with a default value of 180000 (3 minutes). That is why shutting down listener almost took 3.5 minutes. (TCP timeout on other platforms may vary). The error message "Solaris Error: 145: Connection timed out" in ora.node1.LISTENER_NODE1.lsnr.log also indicates it is waiting for tcp timeout.

The listener.ora in this scenario is defined as:

[LISTENER_NODE1 =
(DESCRIPTION_LIST =
   (DESCRIPTION =
   (ADDRESS_LIST =
   (ADDRESS = (PROTOCOL = TCP)(HOST = node1vip)(PORT = 1521)(IP = FIRST))
   )
   (ADDRESS_LIST =
   (ADDRESS = (PROTOCOL = TCP)(HOST = 10.1.10.100)(PORT = 1521)(IP = FIRST))
   )
   (ADDRESS_LIST =
   (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
   )
   )
)
Solution
To prevent this, move the IPC address to be the first address for the listener in the listener.ora, eg:

LISTENER_NODE1 =
(DESCRIPTION_LIST =
   (DESCRIPTION =
   (ADDRESS_LIST =
   (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
   )
   (ADDRESS_LIST =
   (ADDRESS = (PROTOCOL = TCP)(HOST = node1vip)(PORT = 1521)(IP = FIRST))
   )
   (ADDRESS_LIST =
   (ADDRESS = (PROTOCOL = TCP)(HOST = 10.1.10.100)(PORT = 1521)(IP = FIRST))
   )
   )
)

When lsnrctl tries to stop the listener, it will now connect to the IPC address first, which is available during that time. It will not have to wait for tcp timeout.

After the above change, the VIP failover only takes 48 to 50 seconds to complete regardless of the tcp_ip_abort_cinterval setting.

Please note, listener.ora files newly created from 10.2.0.3 to 11.1.0.7 should have the IPC protocol as the first address in listener.ora in most cases. However, if you have upgraded from a previous release, or manually modified/copied over a listener.ora from a previous install, you may not have the IPC protocol as the first address, regardless of your version. Manual modification is required to move IPC protocol to be the first address to avoid the problem described in this note.

也就说 IPC协议需要放在监听地址第一列，修改后，我们在测试，从原来2分钟缩减到20多秒，符合应用切换的要求

回顾2009年，展望2010年

bosonmaster — Mon, 28 Dec 2009 13:33:50 +0000

今天收到公司发的生日邮件，发现2009年马上就要过去了，这一年，说忙碌吧，也还可以，技术这一年基本没啥长进，银子也没攒几个，一直梦想的房子也离自己越来越远，这房价就像做了火箭，一个劲的往上冲，国家老是说控制房价，也不知道最后控制到哪去了，地王一个一个接一个，这房价能控制住吗？去年10月和朋友看的一套房子，当时9.6K，现在二期15500，还售罄，今天还看到新闻，说2009年中国农民纯收入超5000，我不知道这是咋算出来，可以去问问农民伯伯们，你们真的纯收入5000了吗？不
管怎么说，日子还要过下去，房子还要继续租下去，不知道在未来的2010年，房价能否下来，先做个梦吧，希望在2010年自己的技术能进步一点，O(∩_∩)O~，也希望自己在个人方面也有所突破。

最后祝大家：

元旦快乐，新的一年里健康幸福！

kcbz_check_objd_typ_3

bosonmaster — Sun, 27 Dec 2009 02:41:28 +0000

客户系统linux4 x86_64 10.2.0.3，近日有如下报错：
Errors in file /u01/app/oracle/admin/lzrac/bdump/lzrac2_m001_26960.trc:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [0], [0], [1], [], [], [], []
查看相关METALINK和产生的TRACE文件
TRACLE文件部分内容：

ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [0], [0], [1], [], [], [], []
Current SQL statement for this session:
SELECT count(*) over () as total_count, sd_xe_ash_nm.event_name, sd_xe_ash_nm.event_id, sd_xe_ash_nm.parameter1 as p1text, (CASE WHEN (sd_xe_ash_nm.parameter1 is NULL OR sd_xe_ash_nm.parameter1 = '0') THEN 0 ELSE 1 END) as p1valid, sd_xe_ash_nm.parameter2 as p2text, (CASE WHEN (sd_xe_ash_nm.parameter2 is NULL OR sd_xe_ash_nm.parameter2 = '0') THEN 0 ELSE 1 END) as p2valid, sd_xe_ash_nm.parameter3 as p3text, (CASE WHEN (sd_xe_ash_nm.parameter3 is NULL OR sd_xe_ash_nm.parameter3 = '0') THEN 0 ELSE 1 END) as p3valid, sd_xe_ash_nm.keh_evt_id, nvl(xc.class#, 0) as class_num, sd_xe_ash_nm.wait_class_id, nvl(xc.keh_id, 0) as keh_ecl_id, sd_xe_ash_nm.ash_cnt, sd_xe_ash_nm.tot_wts_diff, sd_xe_ash_nm.tot_tmo_diff, sd_xe_ash_nm.tim_wait_diff FROM ( SELECT sd_xe_ash.*, evtname.event_name, evtname.wait_class_id, evtname.parameter1, evtname.parameter2, evtname.parameter3 FROM ( SELECT sd_xe.*, nvl(ash.cnt, 0) as ash_cnt FROM ( SELECT nvl(xe.keh_id, 0) as keh_evt_id, nvl(sd.event_id, xe.event_hash) as event_id, nvl(sd.tot_wts_diff, 0) as tot_wts_diff, nvl(sd.tot_tmo_diff, 0) as tot_tmo_diff, nvl(sd.tim_wait_diff, 0) as tim_wait_diff FROM ( SELECT endsn.event_id as event_id, (endsn.total_waits - nvl(begsn.total_waits,0)) as tot_wts_diff, (endsn.total_timeouts - nvl(begsn.total_timeouts,0)) as tot_tmo_diff, (endsn.time_waited_micro - nvl(begsn.time_waited_micro,0)) as tim_wait_diff FROM ( SELECT end_snap.* FROM (SELECT t1.* FROM WRH$_SYSTEM_EVENT t1, WRM$_SNAPSHOT s1 WHERE t1.dbid = s1.dbid AND t1.instance_number = s1.instance_number AND t1.snap_id = s1.snap_id AND s1.bl_moved = 0 UNION ALL SELECT t2.* FROM WRH$_SYSTEM_EVENT_BL t2, WRM$_SNAPSHOT s2 WHERE t2.dbid = s2.dbid AND t2.instance_number = s2.instance_number AND t2.snap_id = s2.snap_id AND s2.bl_moved <> 0) end_snap WHERE end_snap.dbid = :dbid and end_snap.instance_number = :instance_number and end_snap.snap_id = :end_snap ) endsn LEFT OUTER JOIN ( SELECT beg_snap.* FROM (SELECT t1.* FROM WRH$_SYSTEM_EVENT t1, WRM$_SNAPSHOT s1 WHERE t1.dbid = s1.dbid AND t1.instance_number = s1.instance_number AND t1.snap_id = s1.snap_id AND s1.bl_moved = 0 UNION ALL SELECT t2.* FROM WRH$_SYSTEM_EVENT_BL t2, WRM$_SNAPSHOT s2 WHERE t2.dbid = s2.dbid AND t2.instance_number = s2.instance_number AND t2.snap_id = s2.snap_id AND s2.bl_moved <> 0) beg_snap WHERE beg_snap.dbid = :dbid and beg_snap.instance_number = :instance_number and beg_snap.snap_id = :beg_snap ) begsn ON endsn.event_id = begsn.event_id ) sd FULL OUTER JOIN X$KEHEVTMAP xe ON sd.event_id = xe.event_hash ) sd_xe LEFT OUTER JOIN (SELECT a.event_id, count(*) as cnt FROM (SELECT t1.* FROM WRH$_ACTIVE_SESSION_HISTORY t1, WRM$_SNAPSHOT s1 WHERE t1.dbid = s1.dbid AND t1.instance_number = s1.instance_number AND t1.snap_id = s1.snap_id AND s1.bl_moved = 0 UNION ALL SELECT t2.* FROM WRH$_ACTIVE_SESSION_HISTORY_BL t2, WRM$_SNAPSHOT s2 WHERE t2.dbid = s2.dbid AND t2.instance_number = s2.instance_number AND t2.snap_id = s2.snap_id AND s2.bl_moved <> 0) a WHERE a.dbid = :dbid and a.instance_number = :instance_number and a.snap_id > :beg_snap and a.snap_id <= :end_snap and a.wait_time = 0 GROUP BY a.event_id) ash ON sd_xe.event_id = ash.event_id ) sd_xe_ash, WRH$_EVENT_NAME evtname WHERE evtname.event_id = sd_xe_ash.event_id and evtname.event_id > 0 and evtname.dbid = :dbid ) sd_xe_ash_nm, X$KEHECLMAP xc WHERE sd_xe_ash_nm.wait_class_id = xc.class_hash ORDER BY sd_xe_ash_nm.wait_class_id, sd_xe_ash_nm.tim_wait_diff DESC, sd_xe_ash_nm.event_id
----- PL/SQL Call Stack -----
object line object
handle number name
0x111fc7790 10 package body SYS.PRVT_HDM
0x10eaf25d0 16 SYS.WRI$_ADV_HDM_T
0x119d97690 1535 package body SYS.PRVT_ADVISOR
0x119d97690 1618 package body SYS.PRVT_ADVISOR
0x111fc7790 106 package body SYS.PRVT_HDM
描述过程如下：
Oracle Server - Enterprise Edition - Version: 10.2.0.2 to 10.2.0.3
This problem can occur on any platform.

Symptoms
Segment Advisor is being used.

ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_3], [4], [0], [15], [], [], [], []

With a stack trace similar to:

kgerinv kgeasnmierr kcbassertbd3 kcbz_check_objd_typ kcbzib kcbgtcr ktrget kdirfrs

ORA-600 [kcbnew_3] may be reported instead.

The PLSQL stack, if there is one, may have SYS.PRVT_ADVISOR or SYS.DBMS_SPACE near the top.

Cause
The cause of this problem has been identified and verified in an Unpublished Bug 4430244.

It is caused by the Segment Advisor code which can load blocks for dropped objects into cache as CURRENT leading to subsequent operations seeing an incorrect (old) version of a block.

Solution
This bug is fixed in our 10.2.0.4 patchset and 11g Release 1.

You can check if a patch is available for your patchset release and O/S environment Patch 4430244

To obtain a patch from MetaLink:
1) Click on Patches.
2) Click on Simple Search.
3) Enter your Patch number : 4430244
4) Select your platform
5) Click Go.
6) Read any applicable notes before downloading, then click the Download button.

Note: Please review the Readme file for instructions on how to install the patchset.

To avoid the issue in the short-term, turn off Segment Advisor by disabling 'Automatic Segment Advisor Job'. This is an advisory job so there will be no harm caused to your database by turning it off.

If you are encountering this bug, flush the buffer cache with the following command:
SQL> alter system flush buffer_cache;

to remove old copies of the block.
If you are using RAC then this will need to be done on all instances.

qksfroFXTStatsLoc

bosonmaster — Sun, 27 Dec 2009 02:32:58 +0000

做在客户处，OS从5305升级到了5309SP5，然后数据库从10.2.0.3升级到10.2.0.4，升级时间很快，启动数据库后也没报错，就撤了，早上发现alert里有如下报错
Errors in file /u01/app/oracle/admin/dzcadb/bdump/dzcadb_ora_139828.trc:
ORA-00600: internal error code, arguments: [qksfroFXTStatsLoc() - unknown KQ], [0], [], [], [], [], [], []
Sat Dec 26 04:00:17 2009
Errors in file /u01/app/oracle/admin/dzcadb/bdump/dzcadb_ora_139828.trc:
ORA-00600: internal error code, arguments: [qksfroFXTStatsLoc() - unknown KQ], [0], [], [], [], [], [], []
Sat Dec 26 04:00:18 2009
Errors in file /u01/app/oracle/admin/dzcadb/bdump/dzcadb_ora_139828.trc:
ORA-00600: internal error code, arguments: [qksfroFXTStatsLoc() - unknown KQ], [0], [], [], [], [], [], []
WARNING: Oracle executable binary mismatch detected.
Binary of new process does not match binary which started instance
issue alter system set "_disable_image_check" = true to disable these messages
WARNING: Oracle executable binary mismatch detected.
Binary of new process does not match binary which started instance
issue alter system set "_disable_image_check" = true to disable these messages
Sat Dec 26 04:21:10 2009
Errors in file /u01/app/oracle/admin/dzcadb/bdump/dzcadb_ora_139872.trc:
ORA-07445: exception encountered: core dump [lstclo+0030] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFFF] [] []

一看报错就知道，是ORACLE的执行文件出问题了，因为升级了操作需要重新relink ORACLE的执行文件

解决过程就很简单了，关闭监听，数据库，slibclean 下，在ORACLE用户下执行下relink all就解决

linux 5.4 for oracle rac vipca bug

bosonmaster — Sat, 19 Dec 2009 04:12:08 +0000

在LINUX AS 5 UP4安装10G RAC的时候，在节点2运行root.sh的时候，最后会报如下错误：
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
/home/oracle/crs/oracle/product/10/crs/jdk/jre//bin/java: error while loading
shared libraries: libpthread.so.0: cannot open shared object file:
No such file or directory
在执行vipca的时候会报如下错：
# vipca
Error 0(Native: listNetInterfaces:[3])
[Error 0(Native: listNetInterfaces:[3])]

通过查询METALINK是ORACLE的一个BUG
解决方法：就是修改vipca和srvctl ，搜索LD_ASSUME_KERNEL
在下方添加如下
unset LD_ASSUME_KERNEL
然后在配置PUBLIC和心跳网络

<CRS_HOME>/bin # ./oifcfg setif -global eth0/192.168.1.0:public
<CRS_HOME>/bin # ./oifcfg setif -global eth1/10.10.10.0:cluster_interconnect
然后在VIPCA就OK了

rac datagurad相关的几篇不错文档

bosonmaster — Sun, 01 Nov 2009 03:00:44 +0000

http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_DataGuardNetworkBestPractices.pdf
http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10g_RACPrimarySingleInstancePhysicalStandby.pdf
http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_RACPrimaryRACLogicalStandby.pdf

(Warn) Failed To Find Arch For Message (Message:0×1)

bosonmaster — Sun, 01 Nov 2009 02:51:57 +0000

产生该问题的原因就是log_archive_max_processes 参数设置太小引起的，默认的是2，在10GR最大可以增加到30，我修改为10，问题就解决了，在METALINK：375229.1，文档描述如下
Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.1.0.6
This problem can occur on any platform.
"Checked for relevance on 27-Jan-2009"
Symptoms
- After starting the database receive the following errors in trace file:>
tkcrrsarc: (WARN) Failed to find ARCH for message (message:0x1)
tkcrrpa: (WARN) Failed initial attempt to send ARCH message (message:0x1)
<<
- After manual performing log switches the following messages occur in the alert.log:>
Shutting down archive processes
Wed Oct 19 14:13:22 2005
ARCH shutting down
ARC2: Archival stopped
<<
Cause
Bug 4704334 - ''TKCRRSARC: (WARN) FAILED TO FIND ARCH FOR MESSAGE (MESSAGE:0X1)''

These messages are warning messages and are a consequence of the internal archive process spawning and releasing.

There can be different reasons for this message to appear, but it is in fact related to internal work.

Solution
These are only diagnostic messages and can be safely ignored.
However, there are two options:

Option 1

Ignore the messages.

Option 2

Increase the log_archive_max_processes equal to/or greater than 10

*Testing information about Option 2

log_archive_max_processes=2 may still cause messages, and 4 may cause less.
Recommended setting is 10 or higher as testing has verified that messages disappeared completely at this setting.