Litmus Horizons

Practical insights on AI, Oracle Cloud, and enterprise architecture.

OVMRU_005007E … Reasons: Server: el01cnXX.mycompany.com is not running … Don't Panic!

OVMRU_005007E … Reasons: Server: el01cnXX.mycompany.com is not running … Don't Panic! (title card)

Alright, after each power down and/or Exalogic control-stack restart there may be this situation where some vServers come up just fine and others puke as follows:

08/16/2018 10:35:13 AM CDT ERROR Start of VirtualMachine 0004fb00000600009c5756098cedaf81 failed: OVMRU_005007E Virtual Machine: random-vserver, cannot be started/resumed on Server: el01cnXX.mycompany.com. Reasons:
Server: el01cnXX.mycompany.com is not running

The above error is found while starting vServers using EMOC, and if we drill down the logs you’ll see it.

Basically, what happened here is that even if the CN is actually up and running, and the OCFS2/O2CB cluster is healthy… for some reason/bug (or feature…lol) the OVMM is not updated with the correct CN/server status, thus OVMM still believes the CN is still down causing the OVMRU_005007E error while trying to start vServers on these “off-line” CN/servers.

You will see the following at the OVMM Console:

OVMM Console showing the problem

Also at the OVMM Servers and VMs Tab, switch the perspective to Events and you will find the OVMM clue about the server not being online:

“Ignoring discover request for server: el01cn14.mycompany.com. It is offline“
Alright, with the issue properly identified now comes the simple workaround:
  1. Go to the OVMM Console and then go to the Servers and VMs tab.
  2. Expand all of the Server Pools
  3. Find the “not running/offline” servers inside the Server Pools
  4. On each of the “not running/offline” servers, right click on it and then “Rediscover Server”.
  5. Wait for the OVMM jobs to complete and check the status again. It should show your servers online.
The workaround is to rediscover the servers.
After all of the affected CN/servers have been rediscovered at the OVMM Console, then go to EMOC Console and start your vServers.

Sergio

Comments

Leave a Reply