Tuesday, March 26, 2019

OVM_STATISTIC Table Growing abnormally on OVM Manager 3.4, Unable to Login to "OVM Manager", OVM manager login failed and repeatedly asking login credentials.

OVM_STATISTIC Table Growing abnormally on OVM Manager 3.4. OVM Manager Stopped Working.



Symptoms:

OVM Manager login is very slow.
OVM manager login failed and repeatedly asking login credentials.
OVM Manager "Server and VM Statistics" menu not responding.



AdminServer.log showing event,

####<2017-12-07T08:16:39.005-0500> <Warning> <com.oracle.odof.util.PersistUtils> <ovmmgr.<domain.name> <AdminServer> <[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <a5fed4f7-6dfb-4b3c-befd-ed62a6f7823c-0000055a> <1478524599005><BEA-000000> <Error executing query: select c from CertificateEntity c where c.fingerprint = :fingerprint

javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Error Code: 0
Call: SELECT ID, CERTIFICATE, FINGERPRINT, VERSION, USER FROM OVM_USER_CERTIFICATE WHERE (FINGERPRINT = ?)
bind => [1 parameter bound]
Query: ReadAllQuery(referenceClass=CertificateEntity sql="SELECT ID, CERTIFICATE, FINGERPRINT, VERSION, USER FROM OVM_USER_CERTIFICATE WHERE
(FINGERPRINT = ?)")
at org.eclipse.persistence.internal.jpa.QueryImpl.getDetailedException(QueryImpl.java:378)
at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:260)
at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:469)

at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2280)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2196)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1575)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
Caused By: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Error Code: 0
Call: SELECT ID, CERTIFICATE, FINGERPRINT, VERSION, USER FROM OVM_USER_CERTIFICATE WHERE (FINGERPRINT = ?)
bind => [1 parameter bound]
Query: ReadAllQuery(referenceClass=CertificateEntity sql="SELECT ID, CERTIFICATE, FINGERPRINT, VERSION, USER FROM OVM_USER_CERTIFICATE WHERE (FINGERPRINT = ?)")

Reason:

Disk is full on OVM manager Server, and OVM_STATISTIC.ibd of OVMM database is growing too large

# du -hs /u01/app/oracle/mysql/data/ovs/OVM_STATISTIC*
16K /u01/app/oracle/mysql/data/ovs/OVM_STATISTIC.frm
62G /u01/app/oracle/mysql/data/ovs/OVM_STATISTIC.ibd <--- this file

# mysql ovs -u ovs -p -S /u01/app/oracle/mysql/data/mysqld.sock
mysql> select count(*) from OVM_STATISTIC;
+----------+
| count(*) |
+----------+
| 97127984 |
+----------+


Solution:

Truncate the huge OVM_STATISTIC table as workaround. 

# mysql ovs -u ovs -p -S /u01/app/oracle/mysql/data/mysqld.sock
mysql> truncate table OVM_STATISTIC;


there is no reported/known issue or side effect for truncating the table.

But need to analyze the root cause of abnormal DB growth if this issue is repeating. 



1 comment:

  1. Hello, Has any known issue or side effect been noticed after publication?

    ReplyDelete

Enter Comments...