Inherited Problems on an #EM12c with Admin Server Woes

Today, I had the pleasure of deploying a couple of plugins for Application Management Pack (eBusiness Suite and Siebel) when I came across a nasty error during the actual deployment phase where the OMS is shutdown and restarted.

2014-05-16 09:07:27,576 [main] INFO commands.EmbeddedWLSTRunner flush.112 - Currently connected to Node Manager to monitor the domain GCDomain.
2014-05-16 09:07:27,576 [main] INFO commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:07:27,576 [main] INFO commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:07:27,579 [main] INFO commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:07:27,579 [main] INFO commands.EmbeddedWLSTRunner flush.112 - FAILED_NOT_RESTARTABLE
2014-05-16 09:07:27,579 [main] INFO commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:07:27,580 [main] INFO commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:07:27,580 [main] INFO commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:07:27,580 [main] INFO commands.EmbeddedWLSTRunner checkServerStatus.312 - EMGC_ADMINSERVER server status is FAILED_NOT_RESTARTABLE
2014-05-16 09:07:27,581 [main] INFO commands.EmbeddedWLSTRunner flush.112 - Successfully disconnected from Node Manager.
2014-05-16 09:07:27,581 [main] INFO commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:07:27,582 [main] INFO commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:07:27,593 [main] DEBUG commands.StatusCommand printWebTierStatus.107 - Checking status of WebTier. WebTier host: greenoms01.planets.com upload port: 4889
2014-05-16 09:07:27,709 [main] DEBUG commands.StatusCommand printWebTierStatus.118 - rspCode when accessing over webtier upload port: 200

Umm, what the %#$@? When I checked the emctl.log, it seemed to me that the deployment process had not completed, except that it was restarting OMS. Upon tailing the log, I saw this error message. Having never seen this before, and not being able to find anything on MOS about it, I went ahead and tried to restart OMS but fortunately it was still up and functional. One note (which I can’t find again) mentioned a similar issue where the MOS plugin deployment failed and the fix was a force shutdown of all OMS resources and then a restart.

That was too easy…

Did the plugin even get deployed?

[oracle@greenoms01 ~]$ emcli list_plugins_on_server
OMS name is greenoms01.planets.com:4889_Management_Service
Plug-in Name                                 Plugin-id                     Version [revision]
 
Oracle Engineered System Healthchecks        oracle.em.sehc                12.1.0.4.0
Oracle Ops Center Infrastructure stack       oracle.em.sooc                12.1.0.2.0
Oracle GoldenGate                            oracle.fmw.gg                 12.1.0.1.0
Oracle Database                              oracle.sysman.db              12.1.0.5.0
Oracle Fusion Middleware                     oracle.sysman.emas            12.1.0.5.0
Oracle MOS (My Oracle Support)               oracle.sysman.mos             12.1.0.5.0
Oracle Virtualization                        oracle.sysman.vt              12.1.0.5.0
Oracle Exadata                               oracle.sysman.xa              12.1.0.5.0

Nope, it sure didn’t. Since OMS was up and running, and I didn’t want any more surprises before I continued, I took an export of the configuration – by the way I already had the repository backups.

[oracle@greenoms01 ~]$ emctl exportconfig oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
ExportConfig started...
Machine is Admin Server host. Performing Admin Server backup...
Exporting emoms properties...
Exporting secure properties...
 
Export has determined that the OMS is not fronted
by an SLB. The local hostname was NOT exported.
The exported data can be imported on any host but
resecure of all agents will be required. Please
see the EM Advanced Configuration Guide for more
details.
 
Exporting configuration for pluggable modules...
Preparing archive file...
Backup has been written to file: /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1/sysman/backup/opf_ADMIN_20140516_091322.bka
 
The export file contains sensitive data.
You must keep it secure.
 
ExportConfig completed successfully!

I checked the status once more, and got an interesting error regarding the Admin Server.

[oracle@greenoms01 ~]$ emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : greenoms01.planets.com
HTTP Console Port          : 7788
HTTPS Console Port         : 7802
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4903
EM Instance Home           : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://greenoms01.planets.com:7802/em
Upload URL: https://greenoms01.planets.com:4903/empbs/upload
 
WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : greenoms01.planets.com
Admin Server HTTPS Port: 7102
Admin Server is FAILED_NOT_RESTARTABLE
Admin Server Port is occupied by another process. Please stop process to free port.
 
Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: greenoms01.planets.com
WebTier is Up
Oracle Management Server is Up

Interesting, so most likely there’s a process holding a lock on a file or it simply is the port that is being used again. The emctl.log showed me a “Connection refused” to the NodeManager.

<pre class="brush:text">[oracle@greenoms01 ~]$ vi /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log
2014-05-16 09:04:41,491 [main] INFO  util.EmctlUtil logp.251 - Output messages of the command :
opmnctl stopall: opmn is not running.

2014-05-16 09:04:41,492 [main] INFO  util.EmctlUtil logp.251 - error messages of the command :

2014-05-16 09:04:41,698 [main] INFO  mas.CredStoreUtil logp.251 - getCredential : Got creds for mapName -EM keyName - ENCR_DETAILS
2014-05-16 09:04:42,153 [main] DEBUG oms.PrintOMSSetupInfo main.142 - OMS is not configured with SLB or virtual hostname
2014-05-16 09:04:43,327 [main] INFO  commands.EmbeddedWLSTRunner flush.112 - Not connected to Node Manager
2014-05-16 09:04:43,328 [main] INFO  commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:04:43,328 [main] INFO  commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:04:43,328 [main] INFO  oms.AdminCredsWalletUtil setInstanceHome.177 - Getting credentials from wallet
2014-05-16 09:04:43,361 [main] INFO  oms.AdminCredsWalletUtil setInstanceHome.192 - Read the credentials from wallet
2014-05-16 09:04:43,363 [main] INFO  commands.EmbeddedWLSTRunner flush.112 - Connecting to Node Manager ...
2014-05-16 09:04:43,363 [main] INFO  commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:04:43,364 [main] INFO  commands.EmbeddedWLSTRunner flush.112 -
2014-05-16 09:04:43,429 [main] ERROR commands.EmbeddedWLSTRunner getServerStatus.442 - Exception caught: Traceback (innermost last):
  File "", line 1, in ?
  File "", line 123, in nmConnect
  File "", line 648, in raiseWLSTException
WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Connection refused. Could not connect to NodeManager. Check that it is running at greenoms01.planets.com:7403.
Use dumpStack() to view the full stacktrace
</pre>

The nodemanager.log shows me there was an error while starting the Admin Server.

[oracle@greenoms01 ~]$ vi /u01/app/oracle/12.1.0.3.0/gc_inst/NodeManager/emnodemanager/nodemanager.log
<May 16, 2014 9:04:38 AM> <INFO> <GCDomain> <EMGC_OMS1> <Server was killed>
<May 16, 2014 9:04:50 AM> <INFO> <Loading domains file: /u01/app/oracle/12.1.0.3.0/gc_inst/NodeManager/emnodemanager/nodemanager.domains>
<May 16, 2014 9:04:51 AM> <INFO> <Loading identity key store: FileName=/u01/app/oracle/12.1.0.3.0/middleware/wlserver_10.3/server/lib/DemoIdentity.jks, Type=jks, PassPhraseUsed=true>
<May 16, 2014 9:04:51 AM> <INFO> <Loaded node manager configuration properties from '/u01/app/oracle/12.1.0.3.0/gc_inst/NodeManager/emnodemanager/nodemanager.properties'>
<May 16, 2014 9:04:51 AM> <INFO> <GCDomain> <EMGC_OMS1> <Startup configuration properties loaded from "/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/startup.properties">
<May 16, 2014 9:04:51 AM> <INFO> <GCDomain> <BIP> <Startup configuration properties loaded from "/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/BIP/data/nodemanager/startup.properties">
<May 16, 2014 9:04:52 AM> <INFO> <Secure socket listener started on port 7403, host greenoms01.planets.com/172.20.211.39>
<May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Rotated server output log to "/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.out00014">
<May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Server error log also redirected to server log>
<May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Starting WebLogic server with command line: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/bin/startEMServer.sh >
<May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Working directory is '/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain'>
<May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Rotated server output log to "/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.out00015">
<May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Server error log also redirected to server log>
<May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Server output log file is '/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.out'>
<May 16, 2014 9:05:39 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Server failed during startup so will not be restarted>
<May 16, 2014 9:05:39 AM> <WARNING> <Exception while starting server 'EMGC_ADMINSERVER'>
java.io.IOException: Server failed to start up. See server output log for more details.
 at weblogic.nodemanager.server.AbstractServerManager.start(AbstractServerManager.java:196)
 at weblogic.nodemanager.server.ServerManager.start(ServerManager.java:23)
 at weblogic.nodemanager.server.Handler.handleStart(Handler.java:609)
 at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:119)
 at weblogic.nodemanager.server.Handler.run(Handler.java:71)
 at java.lang.Thread.run(Thread.java:662)

So what does the Admin Servers log show? An error grabbing a lock: “Could not obtain an exclusive lock for directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles”

 vi /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.out

<May 16, 2014 9:04:57 AM EDT> <Info> <Management> <BEA-141107> <Version: WebLogic Server Temporary Patch for 13349651 Thu May 24 10:18:28 IST 2012
WebLogic Server Temporary Patch for BUG14482558 Fri Jan 11 11:20:16 IST 2013
WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050 >
<May 16, 2014 9:04:58 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<May 16, 2014 9:04:58 AM EDT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<May 16, 2014 9:04:58 AM EDT> <Notice> <Log Management> <BEA-170019> <The server log file /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.log is opened. All server side log events will be written to this file.>
<May 16, 2014 9:04:59 AM EDT> <Warning>  <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>
<May 16, 2014 9:05:09 AM EDT> <Warning>  <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>
<May 16, 2014 9:05:19 AM EDT> <Warning>  <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>
<May 16, 2014 9:05:29 AM EDT> <Warning>  <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>
<May 16, 2014 9:05:39 AM EDT> <Error>  <BEA-171519> <Could not obtain an exclusive lock to the embedded LDAP data files directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles because another WebLogic Server is already using this directory. Ensure that the first WebLogic Server is completely shutdown and restart the server.>
<May 16, 2014 9:05:39 AM EDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: Could not obtain an exclusive lock to the embedded LDAP data files directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles because another WebLogic Server is already using this directory. Ensure that the first WebLogic Server is completely shutdown and restart the server.>
<May 16, 2014 9:05:39 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<May 16, 2014 9:05:39 AM EDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<May 16, 2014 9:05:39 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
<May 16, 2014 9:05:39 AM> <FINEST> <NodeManager> <Waiting for the process to die: 30273>
<May 16, 2014 9:05:39 AM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>
<May 16, 2014 9:05:39 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

Nothing on MOS about it, but I did find a good post by Kumar here (see #3). Let’s try to move the lock file and see if that helps.

[oracle@greenoms01 logs]$ cd /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles
[oracle@greenoms01 ldapfiles]$ ls -lhtr
total 6.7M
-rw-r----- 1 oracle oinstall 2.2K Apr  1 14:35 EmbeddedLDAP.index
-rw-r----- 1 oracle oinstall 1.5K Apr  1 14:35 EmbeddedLDAP.delete
-rw-r----- 1 oracle oinstall    0 May 16 09:04 EmbeddedLDAP.lok
-rw-r----- 1 oracle oinstall    8 May 16 09:55 EmbeddedLDAP.twpos
-rw-r----- 1 oracle oinstall    8 May 16 09:55 EmbeddedLDAP.trpos
-rw-r----- 1 oracle oinstall 741K May 16 09:55 EmbeddedLDAP.tran
-rw-r----- 1 oracle oinstall 486K May 16 09:55 EmbeddedLDAP.data
-rw-r----- 1 oracle oinstall  35K May 16 09:55 changelog.index
-rw-r----- 1 oracle oinstall 5.4M May 16 09:55 changelog.data
 
[oracle@greenoms01 logs]$ mv EmbeddedLDAP.lok EmbeddedLDAP.lok.05.16.2014
[oracle@greenoms01 logs]$ rm EmbeddedLDAP.lok
[oracle@greenoms01 ldapfiles]$ ls -lhtr
total 6.7M
-rw-r----- 1 oracle oinstall 2.2K Apr  1 14:35 EmbeddedLDAP.index
-rw-r----- 1 oracle oinstall 1.5K Apr  1 14:35 EmbeddedLDAP.delete
-rw-r----- 1 oracle oinstall    0 May 16 09:04 EmbeddedLDAP.lok.05.16.2014
-rw-r----- 1 oracle oinstall    8 May 16 09:55 EmbeddedLDAP.twpos
-rw-r----- 1 oracle oinstall    8 May 16 09:55 EmbeddedLDAP.trpos
-rw-r----- 1 oracle oinstall 741K May 16 09:55 EmbeddedLDAP.tran
-rw-r----- 1 oracle oinstall 486K May 16 09:55 EmbeddedLDAP.data
-rw-r----- 1 oracle oinstall  35K May 16 09:55 changelog.index
-rw-r----- 1 oracle oinstall 5.4M May 16 09:55 changelog.data

After I restarted OMS, I still came across the same issue, i.e. “FAILED_NOT_RESTARTABLE”.

[oracle@greenoms01 ldapfiles]$ emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
[oracle@greenoms01 ldapfiles]$ emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : greenoms01.planets.com
HTTP Console Port          : 7788
HTTPS Console Port         : 7802
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4903
EM Instance Home           : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://greenoms01.planets.com:7802/em
Upload URL: https://greenoms01.planets.com:4903/empbs/upload
 
WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : greenoms01.planets.com
Admin Server HTTPS Port: 7102
Admin Server is FAILED_NOT_RESTARTABLE
Admin Server Port is occupied by another process. Please stop process to free port.
 
Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: greenoms01.planets.com
WebTier is Up
Oracle Management Server is Up

Since I ran out of options, I did a force shutdown of OMS.

[oracle@greenoms01 ldapfiles]$ emctl stop oms -all -force
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
AdminServer Already Stopped
AdminServer Port is occupied by another process. Please stop process to free port.
Oracle Management Server is Down

And checked for any OMS processes (recommended by the MOS note I cannot find).

[oracle@greenoms01 ldapfiles]$ ps -ef | grep oracle
oracle    3335     1  0 May01 ?        00:00:00 /bin/sh /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/bin/startEMServer.sh
oracle    3338  3335  0 May01 ?        00:00:00 /bin/sh /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/bin/startWebLogic.sh production
oracle    3401  3338  0 May01 ?        00:36:27 /u01/app/oracle/12.1.0.3.0/middleware/jdk16/jdk/bin/java -server -Xms256m -Xmx512m -XX:MaxPermSize=512m -Dweblogic.Name=EMGC_ADMINSERVER -Djava.security.policy=/u01/app/oracle/12.1.0.3.0/middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.system.BootIdentityFile=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/security/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Djava.security.egd=file:///dev/./urandom -Dweblogic.debug.DebugWebAppSecurity=true -Dweblogic.SSL.LoginTimeoutMillis=300000 -Djps.auth.debug=true -Djps.authz=ACC -Djps.combiner.optimize.lazyeval=true -Djps.combiner.optimize=true -Djps.policystore.hybrid.mode=false -Djps.subject.cache.key=5 -Djps.subject.cache.ttl=600000 -Xbootclasspath/p:/u01/app/oracle/12.1.0.3.0/middleware/oms/sysman/jlib/diagpatch_bug11725986.jar -Djdkpatchlog=/u01/app/oracle/12.1.0.3.0/middleware/oms/sysman/log/diagpatch_bug11725986.log -Doracle.apm.home=/u01/app/oracle/12.1.0.3.0/middleware/oms/apm/ -DAPM_HELP_FILENAME=oesohwconfig.xml -Dweblogic.ssl.JSSEEnabled=true -da -Dplatform.home=/u01/app/oracle/12.1.0.3.0/middleware/wlserver_10.3 -Dwls.home=/u01/app/oracle/12.1.0.3.0/middleware/wlserver_10.3/server -Dweblogic.home=/u01/app/oracle/12.1.0.3.0/middleware/wlserver_10.3/server -Dcommon.components.home=/u01/app/oracle/12.1.0.3.0/middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain -Djrockit.optfile=/u01/app/oracle/12.1.0.3.0/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.server.config.dir=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/servers/EMGC_ADMINSERVER -Doracle.domain.config.dir=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/fmwconfig -Digf.arisidbeans.carmlloc=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/carml -Digf.arisidstack.home=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/app/oracle/12.1.0.3.0/middleware/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/app/oracle/12.1.0.3.0/middleware/oracle_common/modules/oracle.oamprovider_11.1.1,/u01/app/oracle/12.1.0.3.0/middleware/oms/sysman/jlib -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.jdbc.remoteEnabled=false -Doracle.apm.home=/u01/app/oracle/12.1.0.3.0/middleware/oms/apm/ -DAPM_HELP_FILENAME=oesohwconfig.xml -Dxdo.server.config.dir=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/bipublisher -DXDO_FONT_DIR=/u01/app/oracle/12.1.0.3.0/middleware/Oracle_BI1/common/fonts -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/app/oracle/12.1.0.3.0/middleware/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server

Kill em’ all!!

[oracle@greenoms01 ldapfiles]$ kill -9 3335 3338

And restart OMS.

[oracle@greenoms01 ldapfiles]$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
[oracle@greenoms01 ldapfiles]$ emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : greenoms01.planets.com
HTTP Console Port          : 7788
HTTPS Console Port         : 7802
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4903
EM Instance Home           : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://greenoms01.planets.com:7802/em
Upload URL: https://greenoms01.planets.com:4903/empbs/upload
 
WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : greenoms01.planets.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING
 
Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: greenoms01.planets.com
WebTier is Up
Oracle Management Server is Up

Bingo!

Now, I do not know what caused the Admin Server to remain up even though I stopped it manually. I know one can start it manually, but not sure of whether the OMS server can be started manually (outside of “emctl start oms”. In any case, this made of an interesting 2 hours in the morning.

If you come across this, I hope you find it helped.

Cheers!

5 comments

  1. Carol D. · · Reply

    Maaz, The same thing just happened to me. Thanks for the post because it saved me a lot of time! Take care and great work!

    1. Hi Carol,

      Glad to hear it was helpful – this took me two to three hours to figure out when I came across it.

      Cheers,
      Maaz

  2. You saved me a lot of grief… The scenario I came across is not quite the same, but the logic of checking various logs sure helped.

  3. you solved my issue.

    thanks a lot for share your knowledge with us.

    have a nice day.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Musings

Things I see and learn!

Thoughts from James H. Lui

If you Care a Little More, Things Happen. Bees can be dangerous. Always wear protective clothing when approaching or dealing with bees. Do not approach or handle bees without proper instruction and training.

bdt's oracle blog

Sharing experience (by Bertrand Drouvot)

Frits Hoogland Weblog

IT Technology; Oracle, linux, TCP/IP and other stuff I find interesting

Vishal desai's Oracle Blog

Just another WordPress.com weblog

%d bloggers like this: