Today, I had the pleasure of deploying a couple of plugins for Application Management Pack (eBusiness Suite and Siebel) when I came across a nasty error during the actual deployment phase where the OMS is shutdown and restarted.
2014-05-16 09:07:27,576 [main] INFO commands.EmbeddedWLSTRunner flush.112 - Currently connected to Node Manager to monitor the domain GCDomain. 2014-05-16 09:07:27,576 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:07:27,576 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:07:27,579 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:07:27,579 [main] INFO commands.EmbeddedWLSTRunner flush.112 - FAILED_NOT_RESTARTABLE 2014-05-16 09:07:27,579 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:07:27,580 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:07:27,580 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:07:27,580 [main] INFO commands.EmbeddedWLSTRunner checkServerStatus.312 - EMGC_ADMINSERVER server status is FAILED_NOT_RESTARTABLE 2014-05-16 09:07:27,581 [main] INFO commands.EmbeddedWLSTRunner flush.112 - Successfully disconnected from Node Manager. 2014-05-16 09:07:27,581 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:07:27,582 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:07:27,593 [main] DEBUG commands.StatusCommand printWebTierStatus.107 - Checking status of WebTier. WebTier host: greenoms01.planets.com upload port: 4889 2014-05-16 09:07:27,709 [main] DEBUG commands.StatusCommand printWebTierStatus.118 - rspCode when accessing over webtier upload port: 200
Umm, what the %#$@? When I checked the emctl.log, it seemed to me that the deployment process had not completed, except that it was restarting OMS. Upon tailing the log, I saw this error message. Having never seen this before, and not being able to find anything on MOS about it, I went ahead and tried to restart OMS but fortunately it was still up and functional. One note (which I can’t find again) mentioned a similar issue where the MOS plugin deployment failed and the fix was a force shutdown of all OMS resources and then a restart.
That was too easy…
Did the plugin even get deployed?
[oracle@greenoms01 ~]$ emcli list_plugins_on_server OMS name is greenoms01.planets.com:4889_Management_Service Plug-in Name Plugin-id Version [revision] Oracle Engineered System Healthchecks oracle.em.sehc 12.1.0.4.0 Oracle Ops Center Infrastructure stack oracle.em.sooc 12.1.0.2.0 Oracle GoldenGate oracle.fmw.gg 12.1.0.1.0 Oracle Database oracle.sysman.db 12.1.0.5.0 Oracle Fusion Middleware oracle.sysman.emas 12.1.0.5.0 Oracle MOS (My Oracle Support) oracle.sysman.mos 12.1.0.5.0 Oracle Virtualization oracle.sysman.vt 12.1.0.5.0 Oracle Exadata oracle.sysman.xa 12.1.0.5.0
Nope, it sure didn’t. Since OMS was up and running, and I didn’t want any more surprises before I continued, I took an export of the configuration – by the way I already had the repository backups.
[oracle@greenoms01 ~]$ emctl exportconfig oms Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : ExportConfig started... Machine is Admin Server host. Performing Admin Server backup... Exporting emoms properties... Exporting secure properties... Export has determined that the OMS is not fronted by an SLB. The local hostname was NOT exported. The exported data can be imported on any host but resecure of all agents will be required. Please see the EM Advanced Configuration Guide for more details. Exporting configuration for pluggable modules... Preparing archive file... Backup has been written to file: /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1/sysman/backup/opf_ADMIN_20140516_091322.bka The export file contains sensitive data. You must keep it secure. ExportConfig completed successfully!
I checked the status once more, and got an interesting error regarding the Admin Server.
[oracle@greenoms01 ~]$ emctl status oms -details Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Console Server Host : greenoms01.planets.com HTTP Console Port : 7788 HTTPS Console Port : 7802 HTTP Upload Port : 4889 HTTPS Upload Port : 4903 EM Instance Home : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1 OMS Log Directory Location : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1/sysman/log OMS is not configured with SLB or virtual hostname Agent Upload is locked. OMS Console is locked. Active CA ID: 1 Console URL: https://greenoms01.planets.com:7802/em Upload URL: https://greenoms01.planets.com:4903/empbs/upload WLS Domain Information Domain Name : GCDomain Admin Server Host : greenoms01.planets.com Admin Server HTTPS Port: 7102 Admin Server is FAILED_NOT_RESTARTABLE Admin Server Port is occupied by another process. Please stop process to free port. Managed Server Information Managed Server Instance Name: EMGC_OMS1 Managed Server Instance Host: greenoms01.planets.com WebTier is Up Oracle Management Server is Up
Interesting, so most likely there’s a process holding a lock on a file or it simply is the port that is being used again. The emctl.log showed me a “Connection refused” to the NodeManager.
<pre class="brush:text">[oracle@greenoms01 ~]$ vi /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log 2014-05-16 09:04:41,491 [main] INFO util.EmctlUtil logp.251 - Output messages of the command : opmnctl stopall: opmn is not running. 2014-05-16 09:04:41,492 [main] INFO util.EmctlUtil logp.251 - error messages of the command : 2014-05-16 09:04:41,698 [main] INFO mas.CredStoreUtil logp.251 - getCredential : Got creds for mapName -EM keyName - ENCR_DETAILS 2014-05-16 09:04:42,153 [main] DEBUG oms.PrintOMSSetupInfo main.142 - OMS is not configured with SLB or virtual hostname 2014-05-16 09:04:43,327 [main] INFO commands.EmbeddedWLSTRunner flush.112 - Not connected to Node Manager 2014-05-16 09:04:43,328 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:04:43,328 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:04:43,328 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.177 - Getting credentials from wallet 2014-05-16 09:04:43,361 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.192 - Read the credentials from wallet 2014-05-16 09:04:43,363 [main] INFO commands.EmbeddedWLSTRunner flush.112 - Connecting to Node Manager ... 2014-05-16 09:04:43,363 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:04:43,364 [main] INFO commands.EmbeddedWLSTRunner flush.112 - 2014-05-16 09:04:43,429 [main] ERROR commands.EmbeddedWLSTRunner getServerStatus.442 - Exception caught: Traceback (innermost last): File "", line 1, in ? File "", line 123, in nmConnect File "", line 648, in raiseWLSTException WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Connection refused. Could not connect to NodeManager. Check that it is running at greenoms01.planets.com:7403. Use dumpStack() to view the full stacktrace </pre>
The nodemanager.log shows me there was an error while starting the Admin Server.
[oracle@greenoms01 ~]$ vi /u01/app/oracle/12.1.0.3.0/gc_inst/NodeManager/emnodemanager/nodemanager.log <May 16, 2014 9:04:38 AM> <INFO> <GCDomain> <EMGC_OMS1> <Server was killed> <May 16, 2014 9:04:50 AM> <INFO> <Loading domains file: /u01/app/oracle/12.1.0.3.0/gc_inst/NodeManager/emnodemanager/nodemanager.domains> <May 16, 2014 9:04:51 AM> <INFO> <Loading identity key store: FileName=/u01/app/oracle/12.1.0.3.0/middleware/wlserver_10.3/server/lib/DemoIdentity.jks, Type=jks, PassPhraseUsed=true> <May 16, 2014 9:04:51 AM> <INFO> <Loaded node manager configuration properties from '/u01/app/oracle/12.1.0.3.0/gc_inst/NodeManager/emnodemanager/nodemanager.properties'> <May 16, 2014 9:04:51 AM> <INFO> <GCDomain> <EMGC_OMS1> <Startup configuration properties loaded from "/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/startup.properties"> <May 16, 2014 9:04:51 AM> <INFO> <GCDomain> <BIP> <Startup configuration properties loaded from "/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/BIP/data/nodemanager/startup.properties"> <May 16, 2014 9:04:52 AM> <INFO> <Secure socket listener started on port 7403, host greenoms01.planets.com/172.20.211.39> <May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Rotated server output log to "/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.out00014"> <May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Server error log also redirected to server log> <May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Starting WebLogic server with command line: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/bin/startEMServer.sh > <May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Working directory is '/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain'> <May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Rotated server output log to "/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.out00015"> <May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Server error log also redirected to server log> <May 16, 2014 9:04:55 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Server output log file is '/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.out'> <May 16, 2014 9:05:39 AM> <INFO> <GCDomain> <EMGC_ADMINSERVER> <Server failed during startup so will not be restarted> <May 16, 2014 9:05:39 AM> <WARNING> <Exception while starting server 'EMGC_ADMINSERVER'> java.io.IOException: Server failed to start up. See server output log for more details. at weblogic.nodemanager.server.AbstractServerManager.start(AbstractServerManager.java:196) at weblogic.nodemanager.server.ServerManager.start(ServerManager.java:23) at weblogic.nodemanager.server.Handler.handleStart(Handler.java:609) at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:119) at weblogic.nodemanager.server.Handler.run(Handler.java:71) at java.lang.Thread.run(Thread.java:662)
So what does the Admin Servers log show? An error grabbing a lock: “Could not obtain an exclusive lock for directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles”
vi /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.out <May 16, 2014 9:04:57 AM EDT> <Info> <Management> <BEA-141107> <Version: WebLogic Server Temporary Patch for 13349651 Thu May 24 10:18:28 IST 2012 WebLogic Server Temporary Patch for BUG14482558 Fri Jan 11 11:20:16 IST 2013 WebLogic Server 10.3.6.0 Tue Nov 15 08:52:36 PST 2011 1441050 > <May 16, 2014 9:04:58 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING> <May 16, 2014 9:04:58 AM EDT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool> <May 16, 2014 9:04:58 AM EDT> <Notice> <Log Management> <BEA-170019> <The server log file /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/EMGC_ADMINSERVER.log is opened. All server side log events will be written to this file.> <May 16, 2014 9:04:59 AM EDT> <Warning> <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.> <May 16, 2014 9:05:09 AM EDT> <Warning> <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.> <May 16, 2014 9:05:19 AM EDT> <Warning> <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.> <May 16, 2014 9:05:29 AM EDT> <Warning> <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.> <May 16, 2014 9:05:39 AM EDT> <Error> <BEA-171519> <Could not obtain an exclusive lock to the embedded LDAP data files directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles because another WebLogic Server is already using this directory. Ensure that the first WebLogic Server is completely shutdown and restart the server.> <May 16, 2014 9:05:39 AM EDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: Could not obtain an exclusive lock to the embedded LDAP data files directory: /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles because another WebLogic Server is already using this directory. Ensure that the first WebLogic Server is completely shutdown and restart the server.> <May 16, 2014 9:05:39 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED> <May 16, 2014 9:05:39 AM EDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down> <May 16, 2014 9:05:39 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN> <May 16, 2014 9:05:39 AM> <FINEST> <NodeManager> <Waiting for the process to die: 30273> <May 16, 2014 9:05:39 AM> <INFO> <NodeManager> <Server failed during startup so will not be restarted> <May 16, 2014 9:05:39 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>
Nothing on MOS about it, but I did find a good post by Kumar here (see #3). Let’s try to move the lock file and see if that helps.
[oracle@greenoms01 logs]$ cd /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles [oracle@greenoms01 ldapfiles]$ ls -lhtr total 6.7M -rw-r----- 1 oracle oinstall 2.2K Apr 1 14:35 EmbeddedLDAP.index -rw-r----- 1 oracle oinstall 1.5K Apr 1 14:35 EmbeddedLDAP.delete -rw-r----- 1 oracle oinstall 0 May 16 09:04 EmbeddedLDAP.lok -rw-r----- 1 oracle oinstall 8 May 16 09:55 EmbeddedLDAP.twpos -rw-r----- 1 oracle oinstall 8 May 16 09:55 EmbeddedLDAP.trpos -rw-r----- 1 oracle oinstall 741K May 16 09:55 EmbeddedLDAP.tran -rw-r----- 1 oracle oinstall 486K May 16 09:55 EmbeddedLDAP.data -rw-r----- 1 oracle oinstall 35K May 16 09:55 changelog.index -rw-r----- 1 oracle oinstall 5.4M May 16 09:55 changelog.data [oracle@greenoms01 logs]$ mv EmbeddedLDAP.lok EmbeddedLDAP.lok.05.16.2014
[oracle@greenoms01 logs]$ rm EmbeddedLDAP.lok
[oracle@greenoms01 ldapfiles]$ ls -lhtr total 6.7M -rw-r----- 1 oracle oinstall 2.2K Apr 1 14:35 EmbeddedLDAP.index -rw-r----- 1 oracle oinstall 1.5K Apr 1 14:35 EmbeddedLDAP.delete -rw-r----- 1 oracle oinstall 0 May 16 09:04 EmbeddedLDAP.lok.05.16.2014 -rw-r----- 1 oracle oinstall 8 May 16 09:55 EmbeddedLDAP.twpos -rw-r----- 1 oracle oinstall 8 May 16 09:55 EmbeddedLDAP.trpos -rw-r----- 1 oracle oinstall 741K May 16 09:55 EmbeddedLDAP.tran -rw-r----- 1 oracle oinstall 486K May 16 09:55 EmbeddedLDAP.data -rw-r----- 1 oracle oinstall 35K May 16 09:55 changelog.index -rw-r----- 1 oracle oinstall 5.4M May 16 09:55 changelog.data
After I restarted OMS, I still came across the same issue, i.e. “FAILED_NOT_RESTARTABLE”.
[oracle@greenoms01 ldapfiles]$ emctl status oms Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. WebTier is Up Oracle Management Server is Up [oracle@greenoms01 ldapfiles]$ emctl status oms -details Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Console Server Host : greenoms01.planets.com HTTP Console Port : 7788 HTTPS Console Port : 7802 HTTP Upload Port : 4889 HTTPS Upload Port : 4903 EM Instance Home : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1 OMS Log Directory Location : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1/sysman/log OMS is not configured with SLB or virtual hostname Agent Upload is locked. OMS Console is locked. Active CA ID: 1 Console URL: https://greenoms01.planets.com:7802/em Upload URL: https://greenoms01.planets.com:4903/empbs/upload WLS Domain Information Domain Name : GCDomain Admin Server Host : greenoms01.planets.com Admin Server HTTPS Port: 7102 Admin Server is FAILED_NOT_RESTARTABLE Admin Server Port is occupied by another process. Please stop process to free port. Managed Server Information Managed Server Instance Name: EMGC_OMS1 Managed Server Instance Host: greenoms01.planets.com WebTier is Up Oracle Management Server is Up
Since I ran out of options, I did a force shutdown of OMS.
[oracle@greenoms01 ldapfiles]$ emctl stop oms -all -force Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Stopping WebTier... WebTier Successfully Stopped Stopping Oracle Management Server... Oracle Management Server Successfully Stopped AdminServer Already Stopped AdminServer Port is occupied by another process. Please stop process to free port. Oracle Management Server is Down
And checked for any OMS processes (recommended by the MOS note I cannot find).
[oracle@greenoms01 ldapfiles]$ ps -ef | grep oracle oracle 3335 1 0 May01 ? 00:00:00 /bin/sh /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/bin/startEMServer.sh oracle 3338 3335 0 May01 ? 00:00:00 /bin/sh /u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/bin/startWebLogic.sh production oracle 3401 3338 0 May01 ? 00:36:27 /u01/app/oracle/12.1.0.3.0/middleware/jdk16/jdk/bin/java -server -Xms256m -Xmx512m -XX:MaxPermSize=512m -Dweblogic.Name=EMGC_ADMINSERVER -Djava.security.policy=/u01/app/oracle/12.1.0.3.0/middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.system.BootIdentityFile=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/security/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Djava.security.egd=file:///dev/./urandom -Dweblogic.debug.DebugWebAppSecurity=true -Dweblogic.SSL.LoginTimeoutMillis=300000 -Djps.auth.debug=true -Djps.authz=ACC -Djps.combiner.optimize.lazyeval=true -Djps.combiner.optimize=true -Djps.policystore.hybrid.mode=false -Djps.subject.cache.key=5 -Djps.subject.cache.ttl=600000 -Xbootclasspath/p:/u01/app/oracle/12.1.0.3.0/middleware/oms/sysman/jlib/diagpatch_bug11725986.jar -Djdkpatchlog=/u01/app/oracle/12.1.0.3.0/middleware/oms/sysman/log/diagpatch_bug11725986.log -Doracle.apm.home=/u01/app/oracle/12.1.0.3.0/middleware/oms/apm/ -DAPM_HELP_FILENAME=oesohwconfig.xml -Dweblogic.ssl.JSSEEnabled=true -da -Dplatform.home=/u01/app/oracle/12.1.0.3.0/middleware/wlserver_10.3 -Dwls.home=/u01/app/oracle/12.1.0.3.0/middleware/wlserver_10.3/server -Dweblogic.home=/u01/app/oracle/12.1.0.3.0/middleware/wlserver_10.3/server -Dcommon.components.home=/u01/app/oracle/12.1.0.3.0/middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain -Djrockit.optfile=/u01/app/oracle/12.1.0.3.0/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.server.config.dir=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/servers/EMGC_ADMINSERVER -Doracle.domain.config.dir=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/fmwconfig -Digf.arisidbeans.carmlloc=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/carml -Digf.arisidstack.home=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/app/oracle/12.1.0.3.0/middleware/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/app/oracle/12.1.0.3.0/middleware/oracle_common/modules/oracle.oamprovider_11.1.1,/u01/app/oracle/12.1.0.3.0/middleware/oms/sysman/jlib -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.jdbc.remoteEnabled=false -Doracle.apm.home=/u01/app/oracle/12.1.0.3.0/middleware/oms/apm/ -DAPM_HELP_FILENAME=oesohwconfig.xml -Dxdo.server.config.dir=/u01/app/oracle/12.1.0.3.0/gc_inst/user_projects/domains/GCDomain/config/bipublisher -DXDO_FONT_DIR=/u01/app/oracle/12.1.0.3.0/middleware/Oracle_BI1/common/fonts -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/app/oracle/12.1.0.3.0/middleware/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server
Kill em’ all!!
[oracle@greenoms01 ldapfiles]$ kill -9 3335 3338
And restart OMS.
[oracle@greenoms01 ldapfiles]$ emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Starting Oracle Management Server... Starting WebTier... WebTier Successfully Started Oracle Management Server Successfully Started Oracle Management Server is Up [oracle@greenoms01 ldapfiles]$ emctl status oms -details Oracle Enterprise Manager Cloud Control 12c Release 3 Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Console Server Host : greenoms01.planets.com HTTP Console Port : 7788 HTTPS Console Port : 7802 HTTP Upload Port : 4889 HTTPS Upload Port : 4903 EM Instance Home : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1 OMS Log Directory Location : /u01/app/oracle/12.1.0.3.0/gc_inst/em/EMGC_OMS1/sysman/log OMS is not configured with SLB or virtual hostname Agent Upload is locked. OMS Console is locked. Active CA ID: 1 Console URL: https://greenoms01.planets.com:7802/em Upload URL: https://greenoms01.planets.com:4903/empbs/upload WLS Domain Information Domain Name : GCDomain Admin Server Host : greenoms01.planets.com Admin Server HTTPS Port: 7102 Admin Server is RUNNING Managed Server Information Managed Server Instance Name: EMGC_OMS1 Managed Server Instance Host: greenoms01.planets.com WebTier is Up Oracle Management Server is Up
Bingo!
Now, I do not know what caused the Admin Server to remain up even though I stopped it manually. I know one can start it manually, but not sure of whether the OMS server can be started manually (outside of “emctl start oms”. In any case, this made of an interesting 2 hours in the morning.
If you come across this, I hope you find it helped.
Cheers!
Maaz Anjum's Blog 
Maaz, The same thing just happened to me. Thanks for the post because it saved me a lot of time! Take care and great work!
Hi Carol,
Glad to hear it was helpful – this took me two to three hours to figure out when I came across it.
Cheers,
Maaz
You saved me a lot of grief… The scenario I came across is not quite the same, but the logic of checking various logs sure helped.
Cheers!!
you solved my issue.
thanks a lot for share your knowledge with us.
have a nice day.