World Wide Numbers on Solaris (WWNs)

or SAN port correlation.

Disks in an array will typically be connected to a Solaris system via a “Storage Area Network” or “SAN” over “Fibre Channel” or “FC” (http://en.wikipedia.org/wiki/Fibre_Channel) which is really just a networking topology that typically uses fiber optic cable to connect systems together. It is analogous to Ethernet in that it specifies protocols (a language) to use for speaking over the cable and the connectors and cable to be used.

The card used to connect this to a Solaris system will be a “Host Bus Adapter” or just “HBA” (http://en.wikipedia.org/wiki/Host_adapter). They are also just called “Host Adapter” sometimes.

Here is a qlogic brand “HBA” fiber channel card:
qlogic HBA

The fiber optic cable plugs into the back of that and the gold pins on the bottom plug into the computer system itself.
If you are familiar with basic desktop computer setup, this will look familiar to you. The principle is the same. You can imagine this card like a Network Interface Card (NIC) that plugs into your desktop computer and has a port on the back for a connection to the network. In this case, the network is Fibre Channel.

Now imagine a disk array box, connected via fiber optic cable to a fibre channel switch which is connected via more fiber optic cable to a port in the back of your system. That is the “Storage Area Network” or “SAN”.

With the array connected to the switch, and the switch connected to many systems, how does your system only “see” the “disks” that are assigned to it, and not every disk in the array?
The answer is that the array operator tells the array to only show certain disks to certain HBA ports. This is sometimes called “zoning” and the array operator will say that he needs to “zone” the “disks”. He does this using the HBA port World Wide Number, or WWN.

So what is the World Wide Number (WWN)? (Note it is alternately called a World Window “NAME” or a World Wide “NUMBER”, I’m not sure which is correct, but “number” is what I hear where I work.)
Each HBA has a unique World Wide Name (WWN), which is similar to an Ethernet MAC address in that it uses an Organizationally Unique Identifier (OUI) assigned by the IEEE. However, WWNs are longer (8 bytes). There are two types of WWNs on a HBA; a node WWN (WWNN), which can be shared by some or all ports of a device, and a port WWN (WWPN), which is necessarily unique to each port. (This line is cut and pasted directly from http://en.wikipedia.org/wiki/Fibre_Channel, but I couldn’t say it better myself).
Note that the “NODE” WWN may be shared among ports, like maybe all ports on the same multi-port card, and so the people operating the array do NOT want that, as it could be ambiguous.
The “PORT” WWN is guaranteed to be unique to an individual port, just like an Ethernet MAC address. That number is what the array operator will see when they look at their end. That number guarantees to them that they are looking at YOUR system, and the correct port on your system.

Here is another FC (Fibre Channel) HBA:
Four Port HBA

Note that it has FOUR ports, so while the NODE WWNs might be the same for each port, the PORT WWNs would be different.
So what does this look like from inside of the Solaris operating system?
Here are the commands to try out:

vxdmpadm listenclosure all
vxdmpadm listctlr all
fcinfo hba-port

luxadm –e  port
luxadm –e dump_map <device>
ls –l /dev/dsk

Here is output from the first command:

root@hosta: vxdmpadm listenclosure all
ENCLR_NAME        ENCLR_TYPE     ENCLR_SNO      STATUS       ARRAY_TYPE     LUN_COUNT
===================================================================================
disk              Disk           DISKS                CONNECTED    Disk        4
df8000            DF800          87040186             CONNECTED    A/A         3
df8001            DF800          87040727             CONNECTED    A/A        81
df8002            DF800          87041006             CONNECTED    A/A        135
df8003            DF800          87012602             CONNECTED    A/A        32
tagmastore-usp0   TagmaStore-USP 0FE27                CONNECTED    A/A        119

In this situation I was asked to give the Port WWNs for the ports connected to array “87040186”. You can see that “enclosure” as the second entry. It is called “df8000”. (Note there is a disk count also on this also!)

Here is the output from the next command:

root@hosta: vxdmpadm listctlr all
CTLR-NAME       ENCLR-TYPE      STATE      ENCLR-NAME
=====================================================
c0              Disk            ENABLED      disk
c4              DF800           DISABLED     df8001
c8              DF800           DISABLED     df8001
c4              DF800           ENABLED      df8002
c8              DF800           ENABLED      df8002
c4              DF800           DISABLED     df8000 <- here
c8              DF800           DISABLED     df8000 <- and here
c3              DF800           ENABLED      df8003
c7              DF800           ENABLED      df8003
c7              TagmaStore-USP  ENABLED      tagmastore-usp0
c3              TagmaStore-USP  ENABLED      tagmastore-usp0

On lines five and six we see the 87040186 array that is on enclosure “df8000” is using controllers c4 and c8. Never mind the “enclr-type” focus on the name, that should correlate between the listenclosure and the listctlr commands.

Before we go on, there is one issue. If the controller is already failed, or disabled via a “cfgadm –f –c unconfigure” command, then it will NOT show up in the above!
So another way to find the controllers is with the command:

vxdmpadm getsubpaths enclosure=df8001

Take the output of this command and on each disk, take the “t” number, minus the last digit and search for it in the output of “cfgadm –al”.

So if the output of vxdmpadm getsubpaths enclosure=df8001 is:

root@hostc: vxdmpadm getsubpaths enclosure=df8001
NAME         STATE[A]   PATH-TYPE[M] DMPNODENAME  ENCLR-NAME   CTLR    ATTRS
================================================================================
c6t50060E80100AACFBd0s2 DISABLED     -          df8001_1598  df8001       c6        -
c8t50060E80100AACF3d0s2 DISABLED     -          df8001_1598  df8001       c8        -
c6t50060E80100AACFBd1s2 DISABLED     -          df8001_1599  df8001       c6        -
…TRUNCATED…
c4t50060E80100AACFFd53s2 DISABLED     -          df8001_2137  df8001       c4        -
c4t50060E80100AACFFd54s2 DISABLED     -          df8001_2138  df8001       c4        -
c4t50060E80100AACFFd55s2 DISABLED     -          df8001_2139  df8001       c4        -
c4t50060E80100AACFFd56s2 DISABLED     -          df8001_2140  df8001       c4        -
c4t50060E80100AACFFd57s2 DISABLED     -          df8001_2141  df8001       c4        -
c4t50060E80100AACFFd58s2 DISABLED     -          df8001_2142  df8001       c4        -
c4t50060E80100AACFFd59s2 DISABLED     -          df8001_2143  df8001       c4        -
c4t50060E80100AACFFd60s2 DISABLED     -          df8001_2144  df8001       c4        -
c4t50060E80100AACFFd61s2 DISABLED     -          df8001_2145  df8001       c4        -
c4t50060E80100AACFFd62s2 DISABLED     -          df8001_2146  df8001       c4        -
c4t50060E80100AACFFd63s2 DISABLED     -          df8001_2147  df8001       c4        -
c4t50060E80100AACFFd64s2 DISABLED     -          df8001_2148  df8001       c4        -
c4t50060E80100AACFFd65s2 DISABLED     -          df8001_2149  df8001       c4        -
c4t50060E80100AACFFd66s2 DISABLED     -          df8001_2150  df8001       c4        -
c4t50060E80100AACFFd67s2 DISABLED     -          df8001_2151  df8001       c4        -

Then search for 50060E80100AACF <-Note we took the last character, a second "F" off, as it changes for each controller.

root@hostc: cfgadm -al|grep -i 50060E80100AACF
c2::50060e80100aacf3           unavailable  connected    unconfigured failed
c2::50060e80100aacf7           disk         connected    unconfigured unknown
c4::50060e80100aacfb           unavailable  connected    unconfigured failed
c4::50060e80100aacff           disk         connected    configured   unknown
c6::50060e80100aacfb           disk         connected    configured   unknown
c8::50060e80100aacf3           disk         connected    configured   unknown

So in that case, it is C2, C4, C6 and C8. (Sorry that this interlude is from another system.)

Now use fcinfo hba-port to find the Port WWNs:

root@hosta: fcinfo hba-port
HBA Port WWN: 2100001b329e9a24
        OS Device Name: /dev/cfg/c2
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764999
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2000001b329e9a24
HBA Port WWN: 2101001b32be9a24
        OS Device Name: /dev/cfg/c3
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764999
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2001001b32be9a24
HBA Port WWN: 2100001b329ec032
        OS Device Name: /dev/cfg/c4 <- HERE Is controller 4, and just ABOVE it is the Port WWN.
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764933
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2000001b329ec032 <- In this case the Node WWN is the same, but that is NOT always the case.
HBA Port WWN: 2101001b32bec032
        OS Device Name: /dev/cfg/c5
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764933
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2001001b32bec032
HBA Port WWN: 2100001b329d7e4c
        OS Device Name: /dev/cfg/c6
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764948
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2000001b329d7e4c
HBA Port WWN: 2101001b32bd7e4c
        OS Device Name: /dev/cfg/c7
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764948
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2001001b32bd7e4c
HBA Port WWN: 2100001b329e5731
        OS Device Name: /dev/cfg/c8 <- Here is controller c8 and, again, the Port WWN is just above it!
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006765424
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2000001b329e5731
HBA Port WWN: 2101001b32be5731
        OS Device Name: /dev/cfg/c9
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006765424
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2001001b32be5731

That is it. Give those Port WWNs to the array team, and they should be able to “zone” the disks so that they appear to your system, and only your system!

Now the next commands are helpful for correlation. Also, fcinfo does not exist on Solairs 9 and below, so you can use the below to find port WWNs on Solaris 8 and 9. You can also use this information for trouble shooting and correlating disks to controllers.

root@hosta: luxadm -e port
/devices/pci@400/pci@0/pci@c/SUNW,qlc@0/fp@0,0:devctl              NOT CONNECTED
/devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0:devctl            CONNECTED
/devices/pci@500/pci@0/pci@d/SUNW,qlc@0/fp@0,0:devctl              CONNECTED
/devices/pci@500/pci@0/pci@d/SUNW,qlc@0,1/fp@0,0:devctl            NOT CONNECTED
/devices/pci@600/pci@0/pci@9/SUNW,qlc@0/fp@0,0:devctl              NOT CONNECTED
/devices/pci@600/pci@0/pci@9/SUNW,qlc@0,1/fp@0,0:devctl            CONNECTED
/devices/pci@700/pci@0/pci@c/SUNW,qlc@0/fp@0,0:devctl              CONNECTED
/devices/pci@700/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0:devctl            NOT CONNECTED

Now, the way to correlate the above devices to controllers is with ls like this:

root@hosta: ls -l /dev/dsk | grep pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s0 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:a
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s1 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:b
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s2 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:c
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s3 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:d
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s4 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:e
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s5 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:f
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s6 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:g
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s7 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:h
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s0 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:a
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s1 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:b
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s2 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:c
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s3 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:d
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s4 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:e
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s5 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:f

This will go on for a long time if you have a lot of disks.
Note how I cut off “devices/” at the front and “:devctl” at the end of the luxadm device name and then grep for that name in the output of ls –l /dev/dsk

What should happen is that EVERY disk listed should have the SAME C number. In this case that device is c3.

So what WWN is c3 on?

root@hosta: luxadm -e dump_map /devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0:devctl
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    32004e  0         50060e8006fe2732 50060e8006fe2732 0x0  (Disk device)
1    320062  0         50060e8006fe2734 50060e8006fe2734 0x0  (Disk device)
2    32009a  0         50060e80100584a9 50060e80100584a9 0x0  (Disk device)
3    340600  0         2101001b32be9a24 2001001b32be9a24 0x1f (Unknown Type,Host Bus Adapter)

Trouble shooting NOTE: The end of each line should say “Disk device”, except for the last, if they say something else here, something is wrong on the Array side!
The last line is the WWN for c3, and the column heading tells you that the FIRST one is the Port WWN.

ALSO note that the (Disk device) WWNs, while not helpful to the Array team, DO correlate with what is in “vxdisk –e list” output, so you can use this to find out what disks are connected to this controller!
(Although, the ls –l method kind of already told you that, this makes a full circle and could help in many trouble shooting and correlation situations.)

NOTE: I used the terms “disk” and “LUN” interchangeably. I hope you understand that while there are very real disks in the array, what you see on your end are just “pretend disks” that are probably made up of striped volumes in the array. “LUN” is an old SCSI term, but it has come to be used for “disk” too. Either way, what we see as “disks” or “luns” are just whatever the array operator decides to show us, and that is what is important here.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *