NetApp Sysstat CP Type

The CP type will give you important information about the health of the ability of the system to write.

Basically, you want to avoid Bs and bs.  B means bad.  B means back to back CP, which means the disks can't keep up with the write workload.

B = Bad

b = even more bad

If you have a B or a b, you need more disks in the aggregate to be able to keep up with the writes.

CP Ty : Consistency Point (CP) type is the reason that a CP started in that interval. The CP types are as follows:

• - No CP started during sampling interval (no writes happened to disk at this point of time)
• number Number of CPs started during sampling interval
• B Back to back CPs (CP generated CP) (The filer is having a tough time keeping up with writes)
• b Deferred back to back CPs (CP generated CP) (the back to back condition is getting worse)
• F CP caused by full NVLog (one half of the nvram log was full, and so was flushed)
• H CP caused by high water mark (rare to see this. The filer was at half way full on one side of the nvram logs, so decides to write on disk).
• L CP caused by low water mark
• S CP caused by snapshot operation
• T CP caused by timer (every 10 seconds filer data is flushed to disk)
• U CP caused by flush
• : continuation of CP from previous interval (means, A cp is still going on, during 1 second intervals)

The type character is followed by a second character which indicates the phase of the CP at the end of the sampling interval. If the CP completed during the sampling interval, this second character will be blank. The phases are as follows:

• 0 Initializing
• n Processing normal files
• s Processing special files
• f Flushing modified data to disk
• v Flushing modified superblock to disk

CP util : The Consistency Point (CP) utilization, the % of time spent in a CP.  100% time in CP is a good thing. It means, the amount of time, used out of the cpu, that was dedicated to writing data, 100% of it was used. 75% means, that only 75% of the time allocated to writing data was utilized, which means we wasted 25% of that time. A good CP percentage has to be at or near 100%.

NetApp Create a junction path on a CDOT vserver / SVM

Create a junction path manually on a NetApp Cluster-Mode/C-Mode/CDOT System

If you use the GUI to set up the SVM, it does not prompt you to create a data volume the same way as the CLI.  Therefore, the "/vol" directory is missing and you are unable to mount volumes on a junction path /vol.

This is as simple as creating a directory, unfortunately it's not not easy.

The only way I've found to do this is to re-run the vserver setup wizard on the command line, use the same vserver name (don't worry it won't affect the running vceter) and then accept all the defaults, and create a data volume as /vol/volname.

Once it has created the volume, CTRL-C out of the wizard, delete the temporary volume, and mount your volumes on the "/vol/" junction path!

cluster::> vserver setup
Welcome to the Vserver Setup Wizard, which will lead you through 
the steps to create a virtual storage server that serves data to clients.

You can enter the following commands at any time:
"help" or "?" if you want to have a question clarified,
"back" if you want to change your answers to previous questions, and 
"exit" if you want to quit the Vserver Setup Wizard. Any changes 
you made before typing "exit" will be applied.

You can restart the Vserver Setup Wizard by typing "vserver setup". To accept a default 
or omit a question, do not enter a value.

Vserver Setup wizard creates and configures only data Vservers.
If you want to create a Vserver with Infinite Volume use the vserver create command.


Step 1. Create a Vserver.
You can type "back", "exit", or "help" at any question.

Enter the Vserver name: [CURRENT-VSERVER-NAME]
Choose the Vserver data protocols to be configured {nfs, cifs, fcp, iscsi, ndmp} [nfs,cifs,fcp,iscsi]: [HIT ENTER]
Choose the Vserver client services to be configured {ldap, nis, dns}: [HIT ENTER]
Vserver SVMNAME's allowed protocol list has been modified to nfs,cifs,fcp,iscsi


Step 2: Create a data volume 
You can type "back", "exit", or "help" at any question.

Do you want to create a data volume? {yes, no} [yes]: [HIT ENTER]
Enter the volume name [vol9]: [HIT ENTER]
Enter the name of the aggregate to contain this volume {AGGR1, AGG2} [AGGR2]: [HIT ENTER]
Enter the volume size: [SOME SMALL SIZE LIKE 2G]
Enter the volume junction path [/vol/vol9]: [HIT ENTER]
It can take up to a minute to create a volume...


Volume vol9 of size 2GB created on aggregate AGGR2 successfully.
Do you want to create an additional data volume? {yes, no} [no]:[HIT CTRL-C] 

Exiting the vserver setup wizard. Any changes you have made so far have been saved.

Warning: You have exited the vserver setup wizard before completing all
of the tasks. The vserver is not configured completely. You can complete setup
by typing "vserver setup" in the command line interface.

cluster::> vol mount -volume VOLNAME -junction-path /vol/volname
  (volume mount)

NetApp DataFabric / OnCommand Unified Manager 5.x - DFM Sybase ASA Database service won't start

NetApp DataFabric / OnCommand Unified Manager 5.x - DFM Sybase ASA Database service won't start

Symptoms:

You won't be able to log into DFM...but the web server is running.  When attempting to log in you get the following error:

Could not connect to host "SERVERNAME" on port 8488.

=== CLIENT ===
Version=3.3.0.8036
Build=3.3P2
OS Name=Windows Server 2008 R2
OS Arch=x86
OS Version=6.1

=== ERROR ===
com.netapp.nmf.main.ConnectionRefusedException

=== MESSAGE ===
Could not connect to host "SERVERNAME" on port 8488.

=== DETAILS ===
The server refused a connection on the indicated port.  This means that there is no network service listening on that port.

The default server ports for DataFabric Manager server are 8088 for HTTP and 8488 for HTTPS.

=== CORRECTIVE ACTION ===
Check that the port number you entered is correct.  If it is correct, make sure the service is started on the server for the port you specified.

=== STACK TRACE ===
com.netapp.nmf.main.ConnectionRefusedException: Could not connect to host "accs001mgt01.accs.int" on port 8488.
at com.netapp.nmf.main.ConnectionException.createException(ConnectionException.java:60)
at com.netapp.nmf.main.DialogApiManager.displayErrorDialog(DialogApiManager.java:720)
at com.netapp.nmf.main.DialogApiManager.showLoginDialog(DialogApiManager.java:221)
at com.netapp.nmf.main.SumoMainFrame.doInitialLogin(SumoMainFrame.java:179)
at com.netapp.nmf.main.SumoMainFrame.<init>(SumoMainFrame.java:149)
at com.netapp.nmf.main.SumoMainComponent.initFrame(SumoMainComponent.java:21)
at com.netapp.nmf.main.MainPlugin.initFrame(MainPlugin.java:637)
at com.netapp.nmf.main.MainPlugin.createAndShowGUI(MainPlugin.java:237)
at com.netapp.nmf.main.MainPlugin$2.run(MainPlugin.java:715)
at java.awt.event.InvocationEvent.dispatch(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$000(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.awt.EventQueue$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
Caused by: netapp.sumo.toolkit.zephyr.NaIOException: java.net.ConnectException: Connection refused: connect
at netapp.sumo.toolkit.zephyr.ZephyrServer.invoke(ZephyrServer.java:277)
at netapp.sumo.toolkit.zephyr.SimpleApiManager.getResult(SimpleApiManager.java:25)
at netapp.sumo.toolkit.zephyr.LoggingApiManager.getResult(LoggingApiManager.java:116)
at netapp.sumo.toolkit.zephyr.SharedCredentialApiManager.getResult(SharedCredentialApiManager.java:79)
at com.netapp.nmf.main.DialogApiManager.access$401(DialogApiManager.java:48)
at com.netapp.nmf.main.DialogApiManager$RequestContext.callApiSuper(DialogApiManager.java:881)
at com.netapp.nmf.main.DialogApiManager.getServerInfo(DialogApiManager.java:469)
at com.netapp.nmf.main.DialogApiManager.access$000(DialogApiManager.java:48)
at com.netapp.nmf.main.DialogApiManager$1.sendRequest(DialogApiManager.java:434)
at com.netapp.nmf.main.DialogApiManager$1.sendRequest(DialogApiManager.java:431)
at com.netapp.nmf.userinterface.util.task.ServerTask$1.run(ServerTask.java:47)
at foxtrot.AbstractWorkerThread$1.run(AbstractWorkerThread.java:39)
at java.security.AccessController.doPrivileged(Native Method)
at foxtrot.AbstractWorkerThread.runTask(AbstractWorkerThread.java:35)
at foxtrot.workers.SingleWorkerThread.run(SingleWorkerThread.java:196)
at foxtrot.workers.SingleWorkerThread.run(SingleWorkerThread.java:179)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.<init>(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl.createSocket(Unknown Source)
at netapp.manage.http.HTTPClient.<init>(HTTPClient.java:556)
at netapp.manage.NaServer.getSession(NaServer.java:335)
at netapp.manage.NaServer.invokeHTTP(NaServer.java:868)
at netapp.manage.NaServer.invokeElemForXML(NaServer.java:716)
at netapp.manage.NaServer.invokeElem(NaServer.java:635)
at netapp.sumo.toolkit.zephyr.ZephyrServer.invoke(ZephyrServer.java:265)
... 16 more

You also will have these Event IDs on the DFM server:

EventID:   1
Source:     SQLANY64 16.0

The description for Event ID 1 from source SQLANY64 16.0 cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

DFMSybase
Could not start server

Event ID:  124

Source:      OnCommandHostSvc

Communication failure between HostAgent and DFM server: Could not connect to https://servername:8488/apis/soap/v1. TCP error code 10061: No connection could be made because the target machine actively refused it IPADDRESS:8488.   Please make sure DFM server is up running.

Cause

After discovering that it was the database service that wouldn't start (since the Event IDs are useless) I found the database log here:

C:\Program Files\NetApp\DataFabric Manager\DFM\log\sybase.log

After scrolling around a bit, I found the following error:


I. 10/07 14:46:42. Starting database "monitordb" (C:\Program Files\NetApp\DataFabric Manager\DFM\data\monitordb.db) at Tue Oct 07 2014 14:46
I. 10/07 14:46:42. Performance warning: Database file "C:\Program Files\NetApp\DataFabric Manager\DFM\data\monitordb.db" consists of 70 disk fragments
I. 10/07 14:46:43. Transaction log: monitordb.log
E. 10/07 14:46:43. Error: Database cannot be started -- monitordb.log is an invalid transaction log

E. 10/07 14:46:43. Database cannot be started -- monitordb.log is an invalid transaction log
I. 10/07 14:46:43. Database server shutdown due to startup error
I. 10/07 14:46:43. Database server stopped at Tue Oct 07 2014 14:46


Resolution

The bad transaction log file was located here:

C:\Program Files\NetApp\DataFabric Manager\DFM\data\monitordb.log

I renamed monitordb.log to monitordb.old

And then I attempted to start the Sybase database service again - and it worked!

net start DFMSybase

So I started the rest of the DFM services:

net start DFMServer
net start DFMWatchDog
net start DFMWebUI
net start DFMScheduler
net start DFMMonitor
net start DFMApache
net start DFMEvent

You may have a lot a little data in the database file, but you've lost more with teh services being stopped, so don't fret.  Delete the bad transaction log file and move on.

NetApp: LUN Migration - 7-Mode to C-Mode/C-DOT

CDOT/C-Mode LUN/block FC/FCoE/iSCSI storage migration

NetApp made a huge improvement in their software moving to C-Mode, however they did not make the process to move block storage LUNs easy.  Of course, they would like you to buy their DTA2800 appliance, which is licensed per TB migrated, and/or use their professional services.

Like most of us, we more than a couple TB to move, and I don't feel I should have to pay for the migration since I bought both systems in the first place.  The good news is you don't have to....if you can deal with a little downtime.

The solution is to use the NetApp Workflow Automation tool or "WFA".  You can do offline and online block LUN replication, and with the online mode - the cutover period can be very short.

Unfortunately, the pieces needed to easily complete this task are not readily compiled and comprehensively documented.

So far, the best document is NetApp TR 4314 which is not publicly accessible - except from this scribd post.  I will not post the actual PDF - but if you have access to the NetApp field portal - you can get EVERYTHING you need to complete the migrations.

Resources:

• A minimum of a 7-Mode System and a C-DOT system
• This is your source and destination system, if you don't have these, why are you reading this article? ;-)?
• An intermediary / staging system
• Ideally another 7-mode system with a bunch of disk to handle the intermediary work, although technically this could be handled by the second half of your origin HA pair.
• A fast network
• Ideally, 10GbE to minimize the bottleneck
• A working installation of NetApp OnCommand Workflow Automation (WFA) 2.2RC1 
• WFA is great but going over the installation process is beyond the scope of this document
• http://mysupport.netapp.com/NOW/download/software/ocwfa/2.2.1RC1/
• OnCommand Unified Manager 6.1 for C-Mode
• You should already have this, hopefully
• http://mysupport.netapp.com/NOW/download/software/oncommand_cdot/6.1/
• OnCommand Unified Manager 5.2 for 7-Mode
• You should already have this, hopefully
• http://mysupport.netapp.com/NOW/cgi-bin/software?product=OnCommand+Unified+Manager+Core+Package&platform=Windows
• A copy of NetApp TR 4314
• This is available via the NetApp Field Portal
• https://fieldportal.netapp.com/Core/DownloadDoc.aspx?documentID=130681&contentID=249731
• A copy of lunmigrate.dar Workflows
• This is available via the NetApp Field Portal
• Release Notes https://fieldportal.netapp.com/Core/DownloadDoc.aspx?documentID=130979&contentID=250745
• Workflow Archive https://fieldportal.netapp.com/Core/DownloadDoc.aspx?documentID=130977&contentID=250744
• A copy of NFShardlink
• This is available via the NetApp Field Portal
• https://fieldportal.netapp.com/Core/DownloadDoc.aspx?documentID=131057&contentID=251066

Instructions:

1. Install OnCommand Workflow Automater
2. Acquire all the items listed above
3. Follow the instructions in TR 4314 to set up WFA
4. Build a test LUN on your 7-Mode System
5. Wait a minute for OCUM to update, and...

Recommendations/Notes:

• Make sure the administrative interfaces, intercluster interfaces, and replication IPs are all on 10-gigabit networks so the network is not your bottleneck in the migration process.
• If you have a spare NetApp, use this system as a staging system between the source and destination systems.  I happened to have a spare system with a couple hundred 15k drives and multiple 10GbE interfaces which worked great as an intermediary.
• There are a bunch of workflows, but don't be overwhelmed.  You really only need one for the initial setup, and three for most migrations.

Steps to Migrate:

Set up the igroups.  You should only need to run this once per source controller/HA pair if your igroups were set up correctly to begin with.

1. Run the "7toC LUN Migrate igroups" workflow
2. Select the source array
3. Select the target cluster
4. Select the target SVM
5. Click Execute
6. Wait for the workflow to execute, make sure it completes successfully
7. Check the CDOT system

Pre-Migration - "Baseline"

1. Run the "7toC LUN Migrate Job Baseline" workflow
2. Select the source system (your 7-Mode controller)
3. Select the volume, when selecting you can see how many LUNs are on that volume.
4. Select the replication IP - this IP needs to be on the same VLAN or routeable to the staging and target systems.  Hopefully it's a 10GbE interface as well.
5. If there is more than one LUN per volume, you can use a regex to describe the LUN(s) you wish to migrate.  If you want to do one LUN at a time (I recommend this) just put the LUN name in the "LUN Pattern" field.
6. Select the staging system, replication IP, vfiler and NFS access IP.
7. Select the target cluster, SVM, aggregate
8. Enter the volume name...I like to name my block volumes LUNNAME_vol but your mileage may vary
9. Click Execute
10. Wait a while - if you can, keep the workflow execution status window up so you can familiarize yourself with the process
11. Hopefully it completes successfully.  If not look at what step of the workflow failed and troubleshoot it

Migration - "Cutover"

IMPORTANT: Leave all the checkboxes at default unless you are going to read the documentation and see what they all do.

1. Run the "7toC LUN Migrate Job Update + Cutover - Rapid" workflow
2. Leave the offline source LUNs box checked
3. Select the target cluster
4. Select the target SVM
5. Select the volume - in the dropbox, it will show you the status of the volume - it should be in the state"baselined"
6. Leave the remap LUNs box checked
7. Shut down the source system (if this is a boot LUN) or stop all applications using the LUN(s)
8. Unmount the disks in the source OS
9. Click the "Execute" button
10. The workflow will typically take about five minutes to execute on a fast system - unless there is a substantial amount of change since the initial baseline.
11. If the LUN migrated was a boot LUN, make sure to update the boot targets in the BIOS (or boot policy in UCS)
12. If the LUN migrated was a data/application LUN, mount the LUN in the OS and test the application

Post-Migration - "Cleanup"

IMPORTANT: Leave all the checkboxes at default unless you are going to read the documentation and see what they all do.

1. Run the "7toC LUN Migrate Job Cleanup" workflow
2. Select the target cluster
3. Select the target SVM
4. Select the target volume - the state should be "cutover"
5. Click Execute

Manual Post-Migration Steps

The original 7-mode LUN should be offline, and the LUN and volume will be intact.  Once you are satisfied with the migration, you may remove the source data.

1. Delete the source LUN
2. Offline the source LUN volume/container (make sure it's the only LUN in the volume)
3. Delete the source LUN container

NetApp: Unable to resize volume - error: Volume has the fixed filesystem size option set

Sometimes you are unable to resize a NetApp volume.  Usually this is because the volume used to be a snapmirror destination, and it was broken off - but it also could have been some cursed sysadmin that preceded you.  At any rate, when you attempt to resize the volume, you're given this error:

"Volume has the fixed filesystem size option set."

Well, NetApp doesn't have a checkbox for that in the GUI.  But, you should be using the command line anyway :)

Here is the command to fix it:

Cluster Mode:

cluster::> volume modify -filesys-size-fixed false -volume <volumename>

7-Mode:

vol options <volumename> fs_size_fixed off

NetApp Error: hw_assist: bind failed to port 4444 on IP address - How to fix

I've run across many filers - set up by NetApp "authorized" systems integrators throw this error.  I've actually have four NetApp HA pairs installed by the professional services group - and the installers always miss this step.

cf.hwassist.socBindFailed:error]: hw_assist: bind failed to port 4444 on IP address X.X.X.X. Error 49

I've actually opened tickets with NetApp support - they told me to reboot the SP and then closed the ticket.  The problem is, the error kept coming back.

The Cause
After some research, this is the high-availability portion of the NetApp attempting to communicate from the SP to the partner controller.  By checking the partner's hardware (rather than waiting for a heartbeat) - the system can do a takeover more quickly in the event of a true failure.

How hardware-assisted takeover speeds up takeover

The IP address it is complaining about in the error should be the IP address of the partner's management interface (not the SP/RLM IP)

Resolution
On each node of the HA pair - set the IP address of the partner's management IP in the correct option.

First, verify that the setting is in fact incorrect.  The IP address in the configuration should NOT be the IP address of the partner's SP/RLM.

Check both nodes.

NETAPP-A> options cf.hw_assist
cf.hw_assist.enable          on         
cf.hw_assist.partner.address 192.163.1.2 
cf.hw_assist.partner.port    4444       

NETAPP-B> options cf.hw_assist
cf.hw_assist.enable          on         
cf.hw_assist.partner.address 192.163.1.3
cf.hw_assist.partner.port    4444       

Yeah - somehow those IP address were on my NetApp.  What noob uses 192.163...what? That's not even RFC1918...wtf?

Next, adjust the settings on each node of the HA pair to the correct IP address for the partner node.   You should use the IP address on the e0m or another management port you may have designated.  Note - do not point this to the SP/RLM IP address - it won't work.

NETAPP-A> options cf.hw_assist.partner.address <CONTROLLER-B-IP>
Validating the new hw-assist configuration. Please wait... 
NETAPP-A> 

NETAPP-B> options cf.hw_assist.partner.address <CONTROLLER-A-IP>
Validating the new hw-assist configuration. Please wait... 
NETAPP-B> 

If you get an error, go back and double check your network - make sure that the correct ports are being used, and that you are putting in the correct IP.  Remember, it needs to be the management address of the partner controller.

Once it's configured correctly, you should see this message on each controller:

[NETAPP-A:cf.hwassist.hwasstActive:info]: hw_assist: hw_assist functionality is active on IP address: <CONTROLLER-A-IP> port: 4444

[NETAPP-B:cf.hwassist.hwasstActive:info]: hw_assist: hw_assist functionality is active on IP address: <CONTROLLER-B-IP> port: 4444 

You can validate the configuration this way:

NETAPP-A> cf hw_assist status 

The output will look like:
                          
Local Node(NETAPP-A) Status:
        Active: NETAPP-Amonitoring alerts from partner(NETAPP-B)
        port 4444 IP address 10.160.99.40
Partner Node(NETAPP-B) Status:
        Active: NETAPP-Bmonitoring alerts from partner(NETAPP-A)
        port 4444 IP address 10.160.99.41

After a few minutes of it being configured correctly, you should see this message in the system log of each controller:

[NETAPP-A:cf.hwassist.recvKeepAlive:info]: hw_assist: Received hw_assist KeepAlive alert from partner(NETAPP-B).  

[NETAPP-B:cf.hwassist.recvKeepAlive:info]: hw_assist: Received hw_assist KeepAlive alert from partner(NETAPP-A).  

NetApp Service Processor (SP) crashing when connecting using putty

I recently brought up a four node 8040 cluster.  Unfortunately, the service processors (SPs) crashed putty every time I attempted to SSH into them.  Openssh on Linux works, however.

After an upgrade to firmware 3.0.1P1 this is no longer an issue.

This appears to be a bug with the firmware version on 3.0.1, which (as of this writing) comes from the factory on Clustered Data ONTAP 8.2.1 on all FAS 8000 systems (8020, 8040, 8060)

To see what version you are running, type the following command:

cluster::> system health controller sp config show

               Firmware Booted               SP       Link
Node           Version  Version   Configured Status   Status Status
-------------- -------- --------- ---------- -------- ------ ---------
node-01
               3.0.1P1  primary   true       online   up     ok
node-02
               3.0.1P1  primary   true       online   up     ok
node-03
               3.0.1P1  primary   true       online   up     ok
node-04
               3.0.1    primary   true       online   up     ok
4 entries were displayed.

As you can see, I have already updated 3 of my four nodes, verifying this is hte fix.

You can download the latest version here:

http://mysupport.netapp.com/NOW/cgi-bin/fw

Alternatively, you can upgrade to the latest version of Clustered Data ONTAP which includes the 3.0.1P1 version as of this writing.

NetApp Cluster-Mode Takeover of epsilon node

If you attempt a manual takeover of a node marked as epsilon, you will receive the following warning:

warning: Takeover of the node holding epsilon can result in data serving failure of the entire cluster. Should an additional node go out of service while the node holding epsilon is out of service, a cluster-wide data serving failure will cause client disruption.

This isn't as scary as it seems...it's important to do correctly, but it's also easy.  To change the node marked as epsilon, do the following:

Change to advanced privilege level:

cluster::> set -privilege advanced

Change to advanced privilege level:

cluster::*> cluster show -epsilon * 

Node            Health  Eligibility   Epsilon
--------------- ------- ------------  ------------
node-01         true    true          false
node-02         true    true          false
node-03         true    true          false
node-04         true    true          true

4 entries were displayed.

Next, mark the current node as false
cluster::*> cluster modify -epsilon false -node CURRENTNODE  

Lastly, mark the new node as true
cluster::*> cluster modify -epsilon true -node NEWNODE

Verify that the change worked:

cluster::*> cluster show -epsilon * 

Node            Health  Eligibility   Epsilon
--------------- ------- ------------  ------------
node-01         true    true          false
node-02         true    true          false
node-03         true    true          true
node-04         true    true          false

4 entries were displayed.

After this, you should be able to take over the node previously holding the epsilon.

OnCommand Unified Manager - Active Directory Integration Error

OnCommand Unified Manager - Active Directory Integration Error

One of the first things you want to do when setting up the NetApp OnCommand Unified Manager is set up AD integrated authentication.

Unfortunately, this doesn't work out of the box, with the correct settings.

Upon testing, you receive the following error:

"Unable to communicate with the authentication server because the authentication server is not configured properly. Verify your authentication server configuration."

Yeah, I know how to set up LDAP.  I read the documentation on this, and I'm following all the rules.

So, after some experimentation, I discovered that removing the extra attributes from the "member" field, and just leaving "member" - as well as using the CORRECT LDAP syntax (the documentation is wrong) fixes the issue and allows you to successfully authenticate.

Well, now it works!

As I was saying, the authentication documentation is wrong.  They state that the OU should be referenced by DC=OU, but this is incorrect.  It should be OU=OUNAME,DC=DOMAIN,DC=TLD.

NetApp CDOT/C-Mode: Domain Authentication

NetApp CDOT / C-Mode / Cluster Mode: Domain Authentication

There are several long guides on how to link cluster-level authentication to AD, however if most of you already have the most basic prerequisites satisfied, you can just bang out a few commands.

You need the following items completed already:

• SVM / VServer created
• CIFS licensed
• CIFS set up and joined to domain on SVM / VServer

Once that is complete, you only need to run a few commands:

1. Create a domain security tunnel

cluster::> security login domain-tunnel create -vserver SVMNAME

2. Allow SSH connection (for us leet folk)

cluster::> security login create -vserver CLUSTERNAME -username DOMAIN\USERNAME -application ssh -authmethod domain -role admin

3. Allow HTTP connection (for all the n00bs)

cluster::> security login create -vserver CLUSTERNAME -username DOMAIN\USERNAME -application http -authmethod domain -role admin

4. Allow ONTAP API connection - most people forget this from their guides...it's for NetApp System Manager

cluster::> security login create -vserver CLUSTERNAME -username DOMAIN\USERNAME -application ontapi -authmethod domain -role admin

5. If you want a list of security permissions, here's the command and sample output:

cluster::> security login show

Vserver: SVMNAME

Vserver: CLUSTERNAME
                             Authentication                  Acct
UserName         Application Method         Role Name        Locked
---------------- ----------- -------------- ---------------- ------
DOM\n00b1        http        domain         admin            -
DOM\n00b1        ontapi      domain         admin            -
DOM\n00b2        http        domain         admin            -
DOM\n00b2        ontapi      domain         admin            -
DOM\l33tadmin    http        domain         admin            -
DOM\l33tadmin    ontapi      domain         admin            -
DOM\l33tadmin    ssh         domain         admin            -

And that's all, folks!

NetApp: Show failed disks

If you have a filer with more than a shelf or two, you will naturally have more failed disks.  Sometimes, you may have more than one disk fail.

Either way, if you have hundreds of disks - you don't want to do a disk show and look through a list of hundreds of disks.

This non-intuitive command (vol - really?  Why vol?  Shouldn't it be aggr?)  will show you a list of failed or broken disks:

vol status -f

Broken disks

RAID Disk       Device          HA  SHELF BAY CHAN Pool Type  RPM
---------       ------          ------------- ---- ---- ---- -----
failed          5a.14.3         5a    14  3   SA:A   0   SAS 15000
failed          5a.14.20        5a    14  20  SA:A   0   SAS 15000

NetApp disk errors: pre-fail a questionable disk

I recently starting receiving NHT health warnings on one of my filers, concerning a disk.

Disk 5a.14.20 received NHT health trigger (0x1 0xb 0x5d 0x10)

After further inspection, this was accompanied with several other block errors regarding this same disk.

scsi.cmd.notReadyCondition:notice]: Disk device 5a.14.20: Device returns not yet ready: CDB 0x00:00000000:0200: Sense Data SCSI:not ready - Drive spinning up (0x2 - 0x4 0x1 0x0)(7281)

raid.read.media.err:debug]: Read error on Disk /aggrXX/plex0/rg4/5a.14.20 Shelf 14 Bay 20 [NETAPP   X412_S15K7560A15 NA04]

raid.tetris.media.err:debug]: Read error on Disk /aggrXX/plex0/rg4/5a.14.20 Shelf 14 Bay 20 [NETAPP   X412_S15K7560A15 NA04] during stripe write

raid.rg.readerr.repair.data:debug]: Fixing bad data on Disk /aggrXX/plex0/rg4/5a.14.20 Shelf 14 Bay 20 [NETAPP   X412_S15K7560A15 NA04]

raid.tetris.media.recommend.reassign.err:info]: Block recommended for reassignment on Disk /aggrXX/plex0/rg4/5a.14.20 Shelf 14 Bay 20 [NETAPP   X412_S15K7560A15 NA04] during stripe write 

I decided to proactively fail the disk - pre-copy the disk to a spare, and then put it in the maintenance center.

disk maint start -d 5a.14.20

You can check on the status by running the aggr status command.  Look for prefail / copy in progress - and the disk it is being copied to.

aggr status -r aggrXX

RAID group /aggrXX/plex0/rg4 (normal, block checksums)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type

      --------- ------          ------------- ---- ---- ----

      data      5a.14.20        5a    14  20  SA:B   0   SAS (prefail, copy in progress)

      -> copy   5a.14.23        5a    14  23  SA:B   0   SAS (copy 7% completed)

Once the disk copy is complete, run:

disk maint status

You will see a status similar to this:

Testing 5a.14.20 test 2/5 cycle 1/1

To get more detail, you can add a -v to the command:

disk maint status -v
Testing 5a.14.20 test 2/5 cycle 1/1
Test name: Power Cycle Test       Iteration: 1/1    Cycle: 1   Status: Not supported   
Test name: Write Same Test        Iteration: 1/1    Cycle: 1   Status: Running            Time:   00:15:42
Test name: Verify Test            Iteration: 0/1    Cycle: 1   Status: Not started     
Test name: Data Integrity Test    Iteration: 0/1    Cycle: 1   Status: Not started     
Test name: Write Same Test        Iteration: 0/1    Cycle: 1   Status: Not started     

NetApp Syslog Configuration Example

The default syslog settings on a NetApp are just to log locally.  If you want to log remotely, you need to edit the /etc/syslog.conf file.

To see the current configuration, use this command:

rdfile /etc/syslog.conf

The default on most NetApp controllers appears to be:

*.info

Here is an sample configuration file I use on my controllers:

wrfile /etc/syslog.conf
# Log everything level info or higher to /etc/messages
*.info                        /etc/messages
# Send it to the console too, so I can see it
*.info                        /dev/console
# Send it to my syslog server too
*.info                        @remotesysloghostname
^C

By putting *.info - all messages of INFO level and above will be logged.  /etc/messages will write to the messages file, /dev/console will output to the console, and @ will send those messages to a remote syslog server.

The wrfile will overwrite the existing file, and you can hit CTRL-C to exit the write file operation.

To verify the file updated, re-read the file.

rdfile /etc/syslog.conf

Shortly after you have updated this file, the syslogd will restart automatically as it detected changes to the configuration file.  You should see something written to /etc/messsages like:

rdfile /etc/messages

Thu Apr 17 01:28:30 PDT [HOSTNAME:kern.syslogd.restarted:info]: syslogd: Restarted.

How to perform an online move of a NetApp volume

NetApp has the ability to move a volume online, between a set of disks or aggregates on the same controller.

Unfortunately, "online" to NetApp has two different meanings.

• For NAS volumes (NFS/CIFS), the volume can be "online" but cannot be exported or shared...therefore, it cannot be serving data.

• For block storage (FC/iSCSI), the volume can be online AND serving data - therefore, no downtime!

Requirements:

• 1. SSH/CLI access
• 2. Data ONTAP 7-Mode, version 8.x or higher

To perform the move, do the following:

vol move start <VOLNAME> <DESTINATION-AGGREGATE>

The move operation will be started immediately.  If you have NFS exports or CIFS shares on this volume, you will receive an error message and the move operation will not proceed.  You must first remove the NFS exports or CIFS shares and retry the move operation.

To monitor the progress, you have the choice of two commands.  

vol move status

or

snapmirror status

Snapmirror status will give you more information - showing you the progress/how much data has been transferred, but will also show other snapmirror sessions.

If the move has been taking a while, and you check the status using the two commands above - you may see that the operation is paused.  This can happen if you have a busy filer, or it was busy during it's attempted cutover.

"vol move status" will show "paused"

"snapmirror status" will show "idle"

If that is the case, you can resume/retry the move:

vol move resume

Once the move is complete (the output of "vol move status" will be blank) - and you can see the volume on the new aggregate, you'll want to go and clean up the temporary snapshot used to do the transfer.

snap list <VOLNAME>

look for the snapshot that contains "ndm_dstvol"

snap delete <VOLNAME> <ndm_dstvol_SNAPNAME>

The volume move is complete!

If you had to remove your NFS export or CIFS share, you can go and re-add it now.

NetApp - Volume / LUN Provisioning via CLI

The NetApp System Manager has some decent controls to set up a volume - once or twice.  However, if you provision a lot of storage and you want everything to be the same, here are the commands I use.

Typically, I will set this up in notepad and then just paste it into the CLI.  Just replace VOLNAME and AGGRNAME and the sizes.  This will do the following for each volume:

• Create the volume
• Turn on dedupe (SIS)
• Configure dedupe to run automatically when it is needed (20% new data written)
• Configure auto snapshot removal on
• Configure auto snapshot removal threshold to 90%
• Set the snapshot reserve to 20%
• Turn on and configure volume autosize maximum and internal
• Hide the snapshot directory
• Turn off unnecessary access time updates for LUNs.
• If you are using NFS or CIFS in a user/file share environment, you may want to leave this option off (i.e., leave access time updates on)
• Turn on reallocation based on the default schedule/interval

vol create VOLNAME -s none AGGRNAME 500g

sis on /vol/VOLNAME

sis config -s auto /vol/VOLNAME

snap autodelete VOLNAME on

snap autodelete VOLNAME target_free_space 10

snap reserve VOLNAME 20
vol autosize VOLNAME -m 750g -i 50g on
vol options VOLNAME nosnapdir true
vol options VOLNAME no_atime_update on
reallocate start -p /vol/VOLNAME

If you work in a NAS environment, export the path:

exportfs -io sec=sys,rw=NFS_SUBNET/24,root=NFS_SUBNET/24,nosuid /vol/VOLNAME

exportfs -w /etc/exports

If you want to add this volume to all VMware hosts, in PowerCLI:

connect-viserver VCENTER
Get-VMHost | New-Datastore -Nfs -Name VOLNAME -Path "/vol/VOLNAME" -NfsHost FILERIP

If you work in a block environment, create the lun.  This is for a Windows boot lun:

lun create -s 80g -t windows_2008 -o noreserve /vol/VOLNAME/INITIATOR_boot_lun

If you need to provision the block initiators, this will set it up with ALUA / MPIO and map it to the boot lun ID:

igroup create -f -t windows INITIATOR

igroup add INITIATOR 20:00:00:25:A0:00:00:0A

igroup add INITIATOR 20:00:00:25:B0:00:00:0B

igroup set INITIATOR alua yes

lun map /vol/VOLNAME/INITIATOR_boot_lun INITIATOR 0

NetApp Performance Monitoring: Disk Performance / How to find out which disks are busy

Disclaimer: I take no responsibility if you screw up, or otherwise render your system unusable.  That being said, if you aren't a moron, this works every time.

When running sysstat -x, you can see how busy the busiest disk is.  However, it doesn't tell you anything else.  The curious sysadmin would wonder:

Which disk is the busiest?
Is there more than one disk that's just as busy?  If so, which ones?

This is where the handy tool "statit" comes in.  But first you need to be in "advanced" mode:

NETAPP-CLI> priv set advanced
Warning: These advanced commands are potentially dangerous; use
         them only when directed to do so by NetApp

Let's start the statistics collection

NETAPP-CLI*> statit -b

Now, wait a few moments.  Depending on how long you want to collect data - wait that long.

NETAPP-CLI*> statit -e

You will see a lot of data output.  Scroll up, and look through your raid group(s).

You will want to focus on the 2nd column "ut%".  If you are running raid_dp (you should be), all of your disks (except for the two parity disks) should be within a few % of each other.

If some disks are substantially busier than others within the same raid group, you may have hot spots - and you should consider running a reallocation scan and subsequently reallocating if your reallocation score is poor.  Click here to learn more about reallocation.

Don't forget to exit out of advanced mode

NETAPP-CLI*> priv set
NETAPP-CLI>

NetApp Performance Monitoring: How to view the CPU status on a NetApp

Disclaimer: I take no responsibility if you screw up, or otherwise render your system unusable.  That being said, if you aren't a moron, this works every time.

If you're curious how busy your NetApp is, there are a variety of tools you can use, as well as  performance counters you can review on the command line.

Sysstat is one of the most common commands I use to troubleshoot performance issues, or just leave open to watch

usage: sysstat [-c count] [-s] [-u | -x | -m | -f | -i | -b] [-d] [interval]
-c count        - the number of iterations to execute
-s              - print out summary statistics when done
-u              - print out utilization format instead
-x              - print out all fields (overrides -u)
-m              - print out multiprocessor statistics
-f              - print out FCP target statistics
-i              - print out iSCSI target statistics
-b              - print out SAN statistics
-d              - print HDD and SSD stats seperately
                      for default, -u and -x formats
interval        - the interval between iterations in seconds, default is 15 seconds

Well, you can just run it without any parameters and still get some decent data.

NETAPP-CLI> sysstat

 CPU     NFS    CIFS    HTTP     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache
                                  in    out    read  write    read  write    age
  9%    1072       0       0    9814   1842    5080  11828       0      0     6 
 34%    4558       0       0   57324  44511   64167  58019       0      0     1s
 45%    6227       0       0   74712  67513   94815  88040       0      0     0s
 34%    2521       0       0   62385  51200   65610  73314       0      0     7 

By default, this will output a basic set of stats - CPU, NFS Ops, CIFS Ops, network and disk throughput at 15 second intervals.

If you want more data more often, add a -x 5.  The -x provides many more columns of data for your to analyze.

NETAPP-CLI> sysstat -x 5

This will give you CPU, NFS, CIFS, HTTP, FCP, and iSCSI Ops.  this will also give you a cache hit ratio, and CP times and types.  

if you are more curious about the CPU, you can go deeper.  sysstat -m 5 will show each CPU core and its level of activity

NETAPP-CLI> sysstat -m 5
 ANY  AVG  CPU0 CPU1 CPU2 CPU3
 24%  10%    9%  11%  11%  10%
 11%   4%    4%   5%   4%   5%

 11%   5%    5%   5%   5%   6%
 17%   7%    6%   8%   8%   8%
  9%   4%    3%   3%   4%   5%
 18%   8%    7%   9%   8%   7%

 12%   4%    4%   4%   4%   5%

Do you wonder what the CPU is doing?  

But first, you must change to diag mode.  Use at your own risk!

NETAPP-CLI*> priv set diag
NETAPP-CLI*> sysstat -M 1

This will show what percentage of the CPU is being dedicated to what process.  You will see RAID, network, storage, etc.  You will also notice something called Kahuna - this is the special NetApp "black box" process that houses everything from dedupe, WAFL block reclamation, and all sorts of other goodies.  If you're suffering from high CPU/controller bottleneck, there's a good chance the Kahuna domain is eating up a good portion of your CPU.

Don't forget to exit out of diag mode

NETAPP-CLI*> priv set
NETAPP-CLI>