NetApp DataFabric / OnCommand Unified Manager 5.x - DFM Sybase ASA Database service won't start
Symptoms:
You won't be able to log into DFM...but the web server is running. When attempting to log in you get the following error:
Could not connect to host "SERVERNAME" on port 8488.
=== CLIENT ===
Version=3.3.0.8036
Build=3.3P2
OS Name=Windows Server 2008 R2
OS Arch=x86
OS Version=6.1
=== ERROR ===
com.netapp.nmf.main.ConnectionRefusedException
=== MESSAGE ===
Could not connect to host "SERVERNAME" on port 8488.
=== DETAILS ===
The server refused a connection on the indicated port. This means that there is no network service listening on that port.
The default server ports for DataFabric Manager server are 8088 for HTTP and 8488 for HTTPS.
=== CORRECTIVE ACTION ===
Check that the port number you entered is correct. If it is correct, make sure the service is started on the server for the port you specified.
=== STACK TRACE ===
com.netapp.nmf.main.ConnectionRefusedException: Could not connect to host "accs001mgt01.accs.int" on port 8488.
at com.netapp.nmf.main.ConnectionException.createException(ConnectionException.java:60)
at com.netapp.nmf.main.DialogApiManager.displayErrorDialog(DialogApiManager.java:720)
at com.netapp.nmf.main.DialogApiManager.showLoginDialog(DialogApiManager.java:221)
at com.netapp.nmf.main.SumoMainFrame.doInitialLogin(SumoMainFrame.java:179)
at com.netapp.nmf.main.SumoMainFrame.<init>(SumoMainFrame.java:149)
at com.netapp.nmf.main.SumoMainComponent.initFrame(SumoMainComponent.java:21)
at com.netapp.nmf.main.MainPlugin.initFrame(MainPlugin.java:637)
at com.netapp.nmf.main.MainPlugin.createAndShowGUI(MainPlugin.java:237)
at com.netapp.nmf.main.MainPlugin$2.run(MainPlugin.java:715)
at java.awt.event.InvocationEvent.dispatch(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$000(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
Caused by: netapp.sumo.toolkit.zephyr.NaIOException: java.net.ConnectException: Connection refused: connect
at netapp.sumo.toolkit.zephyr.ZephyrServer.invoke(ZephyrServer.java:277)
at netapp.sumo.toolkit.zephyr.SimpleApiManager.getResult(SimpleApiManager.java:25)
at netapp.sumo.toolkit.zephyr.LoggingApiManager.getResult(LoggingApiManager.java:116)
at netapp.sumo.toolkit.zephyr.SharedCredentialApiManager.getResult(SharedCredentialApiManager.java:79)
at com.netapp.nmf.main.DialogApiManager.access$401(DialogApiManager.java:48)
at com.netapp.nmf.main.DialogApiManager$RequestContext.callApiSuper(DialogApiManager.java:881)
at com.netapp.nmf.main.DialogApiManager.getServerInfo(DialogApiManager.java:469)
at com.netapp.nmf.main.DialogApiManager.access$000(DialogApiManager.java:48)
at com.netapp.nmf.main.DialogApiManager$1.sendRequest(DialogApiManager.java:434)
at com.netapp.nmf.main.DialogApiManager$1.sendRequest(DialogApiManager.java:431)
at com.netapp.nmf.userinterface.util.task.ServerTask$1.run(ServerTask.java:47)
at foxtrot.AbstractWorkerThread$1.run(AbstractWorkerThread.java:39)
at java.security.AccessController.doPrivileged(Native Method)
at foxtrot.AbstractWorkerThread.runTask(AbstractWorkerThread.java:35)
at foxtrot.workers.SingleWorkerThread.run(SingleWorkerThread.java:196)
at foxtrot.workers.SingleWorkerThread.run(SingleWorkerThread.java:179)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.<init>(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl.createSocket(Unknown Source)
at netapp.manage.http.HTTPClient.<init>(HTTPClient.java:556)
at netapp.manage.NaServer.getSession(NaServer.java:335)
at netapp.manage.NaServer.invokeHTTP(NaServer.java:868)
at netapp.manage.NaServer.invokeElemForXML(NaServer.java:716)
at netapp.manage.NaServer.invokeElem(NaServer.java:635)
at netapp.sumo.toolkit.zephyr.ZephyrServer.invoke(ZephyrServer.java:265)
... 16 more
You also will have these Event IDs on the DFM server:
EventID: 1
Source: SQLANY64 16.0
The description for Event ID 1 from source SQLANY64 16.0 cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
DFMSybase
Could not start server
Cause
After discovering that it was the database service that wouldn't start (since the Event IDs are useless) I found the database log here:
C:\Program Files\NetApp\DataFabric Manager\DFM\log\sybase.log
After scrolling around a bit, I found the following error:
I. 10/07 14:46:42. Starting database "monitordb" (C:\Program Files\NetApp\DataFabric Manager\DFM\data\monitordb.db) at Tue Oct 07 2014 14:46
I. 10/07 14:46:42. Performance warning: Database file "C:\Program Files\NetApp\DataFabric Manager\DFM\data\monitordb.db" consists of 70 disk fragments
I. 10/07 14:46:43. Transaction log: monitordb.log
E. 10/07 14:46:43. Error: Database cannot be started -- monitordb.log is an invalid transaction log
E. 10/07 14:46:43. Database cannot be started -- monitordb.log is an invalid transaction log
I. 10/07 14:46:43. Database server shutdown due to startup error
I. 10/07 14:46:43. Database server stopped at Tue Oct 07 2014 14:46
Resolution
The bad transaction log file was located here:
C:\Program Files\NetApp\DataFabric Manager\DFM\data\monitordb.log
I renamed monitordb.log to monitordb.old
And then I attempted to start the Sybase database service again - and it worked!
net start DFMSybase
So I started the rest of the DFM services:
net start DFMServer
net start DFMWatchDog
net start DFMWebUI
net start DFMScheduler
net start DFMMonitor
net start DFMApache
net start DFMEvent
You may have a lot a little data in the database file, but you've lost more with teh services being stopped, so don't fret. Delete the bad transaction log file and move on.
Symptoms:
You won't be able to log into DFM...but the web server is running. When attempting to log in you get the following error:
Could not connect to host "SERVERNAME" on port 8488.
=== CLIENT ===
Version=3.3.0.8036
Build=3.3P2
OS Name=Windows Server 2008 R2
OS Arch=x86
OS Version=6.1
=== ERROR ===
com.netapp.nmf.main.ConnectionRefusedException
=== MESSAGE ===
Could not connect to host "SERVERNAME" on port 8488.
=== DETAILS ===
The server refused a connection on the indicated port. This means that there is no network service listening on that port.
The default server ports for DataFabric Manager server are 8088 for HTTP and 8488 for HTTPS.
=== CORRECTIVE ACTION ===
Check that the port number you entered is correct. If it is correct, make sure the service is started on the server for the port you specified.
=== STACK TRACE ===
com.netapp.nmf.main.ConnectionRefusedException: Could not connect to host "accs001mgt01.accs.int" on port 8488.
at com.netapp.nmf.main.ConnectionException.createException(ConnectionException.java:60)
at com.netapp.nmf.main.DialogApiManager.displayErrorDialog(DialogApiManager.java:720)
at com.netapp.nmf.main.DialogApiManager.showLoginDialog(DialogApiManager.java:221)
at com.netapp.nmf.main.SumoMainFrame.doInitialLogin(SumoMainFrame.java:179)
at com.netapp.nmf.main.SumoMainFrame.<init>(SumoMainFrame.java:149)
at com.netapp.nmf.main.SumoMainComponent.initFrame(SumoMainComponent.java:21)
at com.netapp.nmf.main.MainPlugin.initFrame(MainPlugin.java:637)
at com.netapp.nmf.main.MainPlugin.createAndShowGUI(MainPlugin.java:237)
at com.netapp.nmf.main.MainPlugin$2.run(MainPlugin.java:715)
at java.awt.event.InvocationEvent.dispatch(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$000(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
Caused by: netapp.sumo.toolkit.zephyr.NaIOException: java.net.ConnectException: Connection refused: connect
at netapp.sumo.toolkit.zephyr.ZephyrServer.invoke(ZephyrServer.java:277)
at netapp.sumo.toolkit.zephyr.SimpleApiManager.getResult(SimpleApiManager.java:25)
at netapp.sumo.toolkit.zephyr.LoggingApiManager.getResult(LoggingApiManager.java:116)
at netapp.sumo.toolkit.zephyr.SharedCredentialApiManager.getResult(SharedCredentialApiManager.java:79)
at com.netapp.nmf.main.DialogApiManager.access$401(DialogApiManager.java:48)
at com.netapp.nmf.main.DialogApiManager$RequestContext.callApiSuper(DialogApiManager.java:881)
at com.netapp.nmf.main.DialogApiManager.getServerInfo(DialogApiManager.java:469)
at com.netapp.nmf.main.DialogApiManager.access$000(DialogApiManager.java:48)
at com.netapp.nmf.main.DialogApiManager$1.sendRequest(DialogApiManager.java:434)
at com.netapp.nmf.main.DialogApiManager$1.sendRequest(DialogApiManager.java:431)
at com.netapp.nmf.userinterface.util.task.ServerTask$1.run(ServerTask.java:47)
at foxtrot.AbstractWorkerThread$1.run(AbstractWorkerThread.java:39)
at java.security.AccessController.doPrivileged(Native Method)
at foxtrot.AbstractWorkerThread.runTask(AbstractWorkerThread.java:35)
at foxtrot.workers.SingleWorkerThread.run(SingleWorkerThread.java:196)
at foxtrot.workers.SingleWorkerThread.run(SingleWorkerThread.java:179)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.<init>(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl.createSocket(Unknown Source)
at netapp.manage.http.HTTPClient.<init>(HTTPClient.java:556)
at netapp.manage.NaServer.getSession(NaServer.java:335)
at netapp.manage.NaServer.invokeHTTP(NaServer.java:868)
at netapp.manage.NaServer.invokeElemForXML(NaServer.java:716)
at netapp.manage.NaServer.invokeElem(NaServer.java:635)
at netapp.sumo.toolkit.zephyr.ZephyrServer.invoke(ZephyrServer.java:265)
... 16 more
You also will have these Event IDs on the DFM server:
EventID: 1
Source: SQLANY64 16.0
The description for Event ID 1 from source SQLANY64 16.0 cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
DFMSybase
Could not start server
Event ID: 124
Source: OnCommandHostSvc
Communication failure between HostAgent and DFM server: Could not connect to https://servername:8488/apis/soap/v1. TCP error code 10061: No connection could be made because the target machine actively refused it IPADDRESS:8488. Please make sure DFM server is up running.
Cause
After discovering that it was the database service that wouldn't start (since the Event IDs are useless) I found the database log here:
C:\Program Files\NetApp\DataFabric Manager\DFM\log\sybase.log
After scrolling around a bit, I found the following error:
I. 10/07 14:46:42. Starting database "monitordb" (C:\Program Files\NetApp\DataFabric Manager\DFM\data\monitordb.db) at Tue Oct 07 2014 14:46
I. 10/07 14:46:42. Performance warning: Database file "C:\Program Files\NetApp\DataFabric Manager\DFM\data\monitordb.db" consists of 70 disk fragments
I. 10/07 14:46:43. Transaction log: monitordb.log
E. 10/07 14:46:43. Error: Database cannot be started -- monitordb.log is an invalid transaction log
E. 10/07 14:46:43. Database cannot be started -- monitordb.log is an invalid transaction log
I. 10/07 14:46:43. Database server shutdown due to startup error
I. 10/07 14:46:43. Database server stopped at Tue Oct 07 2014 14:46
Resolution
The bad transaction log file was located here:
C:\Program Files\NetApp\DataFabric Manager\DFM\data\monitordb.log
I renamed monitordb.log to monitordb.old
And then I attempted to start the Sybase database service again - and it worked!
net start DFMSybase
So I started the rest of the DFM services:
net start DFMServer
net start DFMWatchDog
net start DFMWebUI
net start DFMScheduler
net start DFMMonitor
net start DFMApache
net start DFMEvent
You may have a lot a little data in the database file, but you've lost more with teh services being stopped, so don't fret. Delete the bad transaction log file and move on.
No comments:
Post a Comment