Grid Infrastructure 12.1.0.1.0 Cluster Health Monitor – A Deconstruction

While digging around Grid Infrastructure logs, I came across this new feature with 12c called Cluster Health Monitor (CHM) – I knew the MGMTDB database was good for something when I opted to install it even though it is not required.

From Oracle’s Documentation

“The Cluster Health Monitor (CHM) detects and analyzes operating system and cluster resource-related degradation and failures. CHM stores real-time operating system metrics in the Oracle Grid Infrastructure Management Repository that you can use for later triage with the help of My Oracle Support should you have cluster issues.”

Consisting of three components (see below), the CHM collects and stores data for later review on the cluster’s over-all health.

System Services Monitor (osysmond) 

Where every node in the cluster contains this process, it is responsible for up-to-date monitoring and metric collection service at the Operating System level.

Cluster Logger Service (ologgerd)

This process is what actually retrieves data from the osysmond and writes it to the repository

Grid Infrastructure Management Respository

An Oracle instance which stores the collected data from osysmond. This will only run on a single (hub) node in a cluster, and by design will fail-over to another node should the one its on be unavailable. Interestingly enough, the data files for the instance are located on the same disk group as the OCR and Voting files. Oracle Docs do not talk about any specific sizing, but the onclumon utility is responsible for retention of the stored data.

Let’s take a look at all processes associated with a Grid Infrastructure setup.

NewImage

[root@flex1 ~]# ps -ef | grep root | grep grid
root 2210 1 1 21:07 ? 00:00:57 /u01/app/12.1.0.1/grid/bin/ohasd.bin reboot
root 2516 1 0 21:07 ? 00:00:06 /u01/app/12.1.0.1/grid/bin/orarootagent.bin
root 2729 1 0 21:07 ? 00:00:02 /u01/app/12.1.0.1/grid/bin/cssdmonitor
root 2743 1 0 21:07 ? 00:00:02 /u01/app/12.1.0.1/grid/bin/cssdagent
root 4809 1 0 21:08 ? 00:00:20 /u01/app/12.1.0.1/grid/bin/octssd.bin reboot
root 5699 1 1 21:08 ? 00:01:04 /u01/app/12.1.0.1/grid/bin/osysmond.bin
root 5705 1 0 21:08 ? 00:00:39 /u01/app/12.1.0.1/grid/bin/crsd.bin reboot
root 5969 1 0 21:08 ? 00:00:22 /u01/app/12.1.0.1/grid/bin/orarootagent.bin
root 19405 1 0 22:10 ? 00:00:01 /u01/app/12.1.0.1/grid/bin/gnsd.bin -trace-level 1 -ip-address 192.168.78.244 -startup-endpoint ipc://GNS_flex1.muscle_5969_a598858b344350d1
root 20713 1 0 22:13 ? 00:00:01 /u01/app/12.1.0.1/grid/bin/ologgerd -M -d /u01/app/12.1.0.1/grid/crf/db/flex1

Diagnostics Collection

The most convenient method to query the data in the CHM Repository, is by executing the oclumon utility. 

To collect diagnostic information, preferably all nodes in a cluster, you can run the diagcollection.pl script located in the $GRID_HOME/bin. There are options with this script to collected either all, or specific CRS daemon process logs.

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/diagcollection.pl --help
Production Copyright 2004, 2010, Oracle. All rights reserved

Cluster Ready Services (CRS) diagnostic collection tool

diagcollection
--collect
[--crs] For collecting crs diagnostic information
[--adr] For collecting diagnostic information for ADR; specify ADR location
[--chmos] For collecting Cluster Health Monitor (OS) data
[--acfs] Unix only. For collecting ACFS diagnostic information
[--all] Default.For collecting all diagnostic information.
[--core] UNIX only. Package core files with CRS data
[--afterdate] UNIX only. Collects archives from the specified date. Specify in mm/dd/yyyy format
[--aftertime] Supported with -adr option. Collects archives after the specified time. Specify in YYYYMMDDHHMISS24 format
[--beforetime] Supported with -adr option. Collects archives before the specified date. Specify in YYYYMMDDHHMISS24 format
[--crshome] Argument that specifies the CRS Home location
[--incidenttime] Collects Cluster Health Monitor (OS) data from the specified time. Specify in MM/DD/YYYYHH24:MM:SS format
If not specified, Cluster Health Monitor (OS) data generated in the past 24 hours are collected
[--incidentduration] Collects Cluster Health Monitor (OS) data for the duration after the specified time. Specify in HH:MM format.
If not specified, all Cluster Health Monitor (OS) data after incidenttime are collected
NOTE:
1. You can also do the following
diagcollection.pl --collect --crs --crshome
--clean cleans up the diagnosability
information gathered by this script
--coreanalyze UNIX only. Extracts information from core files
and stores it in a text file

1. First off, we need to find out which node the OLOGGERD service is currently running.

[root@flex1 bin]# /u01/app/12.1.0.1/grid/bin/oclumon manage -get master

Master = flex1

2. Good, it happens to run on the same node I am currently on. Next, we can invoke the diagcollection.pl script to collect the data in the repository.

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/diagcollection.pl --collect
Production Copyright 2004, 2010, Oracle. All rights reserved
Cluster Ready Services (CRS) diagnostic collection tool
The following CRS diagnostic archives will be created in the local directory.
crsData_flex1_20140206_2335.tar.gz -> logs,traces and cores from CRS home. Note: core files will be packaged only with the --core option.
ocrData_flex1_20140206_2335.tar.gz -> ocrdump, ocrcheck etc
coreData_flex1_20140206_2335.tar.gz -> contents of CRS core files in text format

osData_flex1_20140206_2335.tar.gz -> logs from Operating System
Collecting crs data
/bin/tar: log/flex1/cssd/ocssd.log: file changed as we read it
Collecting OCR data
Collecting information from core files
No corefiles found
The following diagnostic archives will be created in the local directory.
acfsData_flex1_20140206_2335.tar.gz -> logs from acfs log.
Collecting acfs data
Collecting OS logs
Collecting sysconfig data

3. It generates a few tar balls, and a text file.

[root@flex1 tmp]# ls -lhtr
total 25M
-rw-r--r-- 1 root root 25M Feb 6 23:36 crsData_flex1_20140206_2335.tar.gz
-rw-r--r-- 1 root root 57K Feb 6 23:37 ocrData_flex1_20140206_2335.tar.gz
-rw-r--r-- 1 root root 927 Feb 6 23:37 acfsData_flex1_20140206_2335.tar.gz
-rw-r--r-- 1 root root 329K Feb 6 23:37 osData_flex1_20140206_2335.tar.gz
-rw-r--r-- 1 root root 31K Feb 6 23:37 sysconfig_flex1_20140206_2335.txt

4. You could limit the data that collected by using date fields

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/diagcollection.pl --collect --afterdate 02/04/2014

5. I was curious, so I untar’d the crsData_flex1_20140206_2335.tar.gz file, and found that the logs from the following locations in the $GRID_HOME directory.

install/
log/flex1/
log/flex1/crfmond/
log/flex1/mdnsd/
log/flex1/gpnpd/
log/flex1/gipcd/
log/flex1/cvu/cvutrc/
log/flex1/cvu/cvulog/
log/flex1/racg/
log/flex1/crflogd/
log/flex1/cssd/
log/flex1/ohasd/
log/flex1/acfs/kernel/
log/flex1/ctssd/
log/flex1/gnsd/
log/flex1/crsd/
log/flex1/client/
log/flex1/agent/ohasd/oracssdmonitor_root/
log/flex1/agent/crsd/oraagent_oracle/
log/flex1/evmd/
cfgtoollogs/
cfgtoollogs/cfgfw/
cfgtoollogs/crsconfig/
cfgtoollogs/oui/
cfgtoollogs/mgmtca/
oc4j/j2ee/home/log/
oc4j/j2ee/home/log/wsmgmt/auditing/
oc4j/j2ee/home/log/wsmgmt/logging/
oc4j/j2ee/home/log/oc4j/
oc4j/j2ee/home/log/dbwlm/auditing/
oc4j/j2ee/home/log/dbwlm/logging/

OCLUMON

Now that we have dispensed with the logs, let’s see what this fancy OCLUMON can do.

1. First off, we need to set the logging level for the daemon we’d like to monitor.

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/oclumon debug log osysmond CRFMOND:3

2. Next, start the process with the dumpnodeview

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/oclumon dumpnodeview -n flex1

----------------------------------------

Node: flex1 Clock: '14-02-07 00.02.08' SerialNo:2081

----------------------------------------

SYSTEM:

#pcpus: 1 #vcpus: 1 cpuht: N chipname: Intel(R) cpu: 26.27 cpuq: 2 physmemfree: 141996 physmemtotal: 4055420 mcache: 2017648 swapfree: 3751724 swaptotal: 4063228 hugepagetotal: 0 hugepagefree: 0 hugepagesize: 2048 ior: 105 iow: 248 ios: 55 swpin: 0 swpout: 0 pgin: 105 pgout: 182 netr: 47.876 netw: 24.977 procs: 302 rtprocs: 12 #fds: 24800 #sysfdlimit: 6815744 #disks: 9 #nics: 3 nicErrors: 0

TOP CONSUMERS:

topcpu: 'apx_vktm_+apx1(7240) 3.40' topprivmem: 'java(19943) 138464' topshm: 'ora_mman_sport(14493) 223920' topfd: 'ocssd.bin(2778) 341' topthread: 'console-kit-dae(1973) 64'



----------------------------------------

Node: flex1 Clock: '14-02-07 00.02.13' SerialNo:2082

----------------------------------------

SYSTEM:

#pcpus: 1 #vcpus: 1 cpuht: N chipname: Intel(R) cpu: 46.13 cpuq: 17 physmemfree: 110400 physmemtotal: 4055420 mcache: 2027560 swapfree: 3751708 swaptotal: 4063228 hugepagetotal: 0 hugepagefree: 0 hugepagesize: 2048 ior: 7714 iow: 210 ios: 399 swpin: 0 swpout: 6 pgin: 7212 pgout: 190 netr: 19.810 netw: 17.537 procs: 303 rtprocs: 12 #fds: 24960 #sysfdlimit: 6815744 #disks: 9 #nics: 3 nicErrors: 0


TOP CONSUMERS:

topcpu: 'apx_vktm_+apx1(7240) 3.00' topprivmem: 'java(19943) 138464' topshm: 'ora_mman_sport(14493) 223920' topfd: 'ocssd.bin(2778) 341' topthread: 'console-kit-dae(1973) 64'

This will regularly dump an output similar to a “top” command in linux. As with the diagcollection.pl script, there are date duration parameters for oclumon as well.

3. The data (in the MGMTDB instance) is stored in the CHM schema.

SQL> select table_name from dba_tables where owner = 'CHM';

TABLE_NAME
--------------------------------------------------------------------------------
CHMOS_SYSTEM_SAMPLE_INT_TBL
CHMOS_SYSTEM_CONFIG_INT_TBL
CHMOS_SYSTEM_PERIODIC_INT_TBL
CHMOS_SYSTEM_MGMTDB_CONFIG_TBL
CHMOS_CPU_INT_TBL
CHMOS_PROCESS_INT_TBL
CHMOS_DEVICE_INT_TBL
CHMOS_NIC_INT_TBL
CHMOS_FILESYSTEM_INT_TBL
CHMOS_ASM_CONFIG


10 rows selected.

4. As mentioned earlier, you can also manage the repository retention period from oclumon.

4.1 To find out the current settings, we can issue the -get parameter.

4.1.1 Find the repository size, in bytes.

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/oclumon manage -get repsize

CHM Repository Size = 136320

4.1.2 Find the repository data file location

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/oclumon manage -get reppath

CHM Repository Path = +DATA/_MGMTDB/DATAFILE/sysmgmtdata.260.835192031

4.1.3 Find the master and logger nodes

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/oclumon manage -get master

Master = flex1
[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/oclumon manage -get alllogger

Loggers = flex1,
[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/oclumon manage -get mylogger

Logger = flex1

4.2 To set parameters, follow some of the examples below.

4.2.1 The changeretentiontime is merely an indicator for how much longer the underlying tablespace can accommodate the collected data. The value is (I believe) in seconds.

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/oclumon manage -repos changeretentiontime 1000

The Cluster Health Monitor repository can support the desired retention for 2 hosts

4.2.2 Change the repository’s tablespace size (in MB). This also changes the retention period.

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/oclumon manage -repos changerepossize 6000
The Cluster Health Monitor repository was successfully resized.The new retention is 399240 seconds.

The alert.log for -MGMTDB shows a simple ALTER TABLESPACE command.

Fri Feb 07 00:28:11 2014
ALTER TABLESPACE SYSMGMTDATA RESIZE 6000 M
Completed: ALTER TABLESPACE SYSMGMTDATA RESIZE 6000 M

The size of the instance has obviously increased as well.

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/oclumon manage -get repsize

CHM Repository Size = 399240

5. And last, but not least, the version check!

[root@flex1 tmp]# /u01/app/12.1.0.1/grid/bin/oclumon version

Cluster Health Monitor (OS), Version 12.1.0.1.0 - Production Copyright 2007, 2013 Oracle. All rights reserved.

Well, I hope this has been an insightful post on the new CHM feature in the 12c release of Grid Infrastructure. If anything, the diagcollection.pl will be a nice replacement to the RDA that Support might request. I haven’t had to troubleshoot any clusteware issues on 12c, but I plan to break this environment and use the oclumon utility to debug the processes at a later date.

Cheers!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Musings

Things I see and learn!

Thoughts from James H. Lui

If you Care a Little More, Things Happen. Bees can be dangerous. Always wear protective clothing when approaching or dealing with bees. Do not approach or handle bees without proper instruction and training.

bdt's oracle blog

Sharing experience (by Bertrand Drouvot)

Frits Hoogland Weblog

IT Technology; Oracle, linux, TCP/IP and other stuff I find interesting

Vishal desai's Oracle Blog

Just another WordPress.com weblog

%d bloggers like this: