World Wide Numbers on Solaris (WWNs)

or SAN port correlation.

Disks in an array will typically be connected to a Solaris system via a “Storage Area Network” or “SAN” over “Fibre Channel” or “FC” (http://en.wikipedia.org/wiki/Fibre_Channel) which is really just a networking topology that typically uses fiber optic cable to connect systems together. It is analogous to Ethernet in that it specifies protocols (a language) to use for speaking over the cable and the connectors and cable to be used.

The card used to connect this to a Solaris system will be a “Host Bus Adapter” or just “HBA” (http://en.wikipedia.org/wiki/Host_adapter). They are also just called “Host Adapter” sometimes.

Here is a qlogic brand “HBA” fiber channel card:
qlogic HBA

The fiber optic cable plugs into the back of that and the gold pins on the bottom plug into the computer system itself.
If you are familiar with basic desktop computer setup, this will look familiar to you. The principle is the same. You can imagine this card like a Network Interface Card (NIC) that plugs into your desktop computer and has a port on the back for a connection to the network. In this case, the network is Fibre Channel.

Now imagine a disk array box, connected via fiber optic cable to a fibre channel switch which is connected via more fiber optic cable to a port in the back of your system. That is the “Storage Area Network” or “SAN”.

With the array connected to the switch, and the switch connected to many systems, how does your system only “see” the “disks” that are assigned to it, and not every disk in the array?
The answer is that the array operator tells the array to only show certain disks to certain HBA ports. This is sometimes called “zoning” and the array operator will say that he needs to “zone” the “disks”. He does this using the HBA port World Wide Number, or WWN.

So what is the World Wide Number (WWN)? (Note it is alternately called a World Window “NAME” or a World Wide “NUMBER”, I’m not sure which is correct, but “number” is what I hear where I work.)
Each HBA has a unique World Wide Name (WWN), which is similar to an Ethernet MAC address in that it uses an Organizationally Unique Identifier (OUI) assigned by the IEEE. However, WWNs are longer (8 bytes). There are two types of WWNs on a HBA; a node WWN (WWNN), which can be shared by some or all ports of a device, and a port WWN (WWPN), which is necessarily unique to each port. (This line is cut and pasted directly from http://en.wikipedia.org/wiki/Fibre_Channel, but I couldn’t say it better myself).
Note that the “NODE” WWN may be shared among ports, like maybe all ports on the same multi-port card, and so the people operating the array do NOT want that, as it could be ambiguous.
The “PORT” WWN is guaranteed to be unique to an individual port, just like an Ethernet MAC address. That number is what the array operator will see when they look at their end. That number guarantees to them that they are looking at YOUR system, and the correct port on your system.

Here is another FC (Fibre Channel) HBA:
Four Port HBA

Note that it has FOUR ports, so while the NODE WWNs might be the same for each port, the PORT WWNs would be different.
So what does this look like from inside of the Solaris operating system?
Here are the commands to try out:

vxdmpadm listenclosure all
vxdmpadm listctlr all
fcinfo hba-port

luxadm –e  port
luxadm –e dump_map <device>
ls –l /dev/dsk

Here is output from the first command:

root@hosta: vxdmpadm listenclosure all
ENCLR_NAME        ENCLR_TYPE     ENCLR_SNO      STATUS       ARRAY_TYPE     LUN_COUNT
===================================================================================
disk              Disk           DISKS                CONNECTED    Disk        4
df8000            DF800          87040186             CONNECTED    A/A         3
df8001            DF800          87040727             CONNECTED    A/A        81
df8002            DF800          87041006             CONNECTED    A/A        135
df8003            DF800          87012602             CONNECTED    A/A        32
tagmastore-usp0   TagmaStore-USP 0FE27                CONNECTED    A/A        119

In this situation I was asked to give the Port WWNs for the ports connected to array “87040186”. You can see that “enclosure” as the second entry. It is called “df8000”. (Note there is a disk count also on this also!)

Here is the output from the next command:

root@hosta: vxdmpadm listctlr all
CTLR-NAME       ENCLR-TYPE      STATE      ENCLR-NAME
=====================================================
c0              Disk            ENABLED      disk
c4              DF800           DISABLED     df8001
c8              DF800           DISABLED     df8001
c4              DF800           ENABLED      df8002
c8              DF800           ENABLED      df8002
c4              DF800           DISABLED     df8000 <- here
c8              DF800           DISABLED     df8000 <- and here
c3              DF800           ENABLED      df8003
c7              DF800           ENABLED      df8003
c7              TagmaStore-USP  ENABLED      tagmastore-usp0
c3              TagmaStore-USP  ENABLED      tagmastore-usp0

On lines five and six we see the 87040186 array that is on enclosure “df8000” is using controllers c4 and c8. Never mind the “enclr-type” focus on the name, that should correlate between the listenclosure and the listctlr commands.

Before we go on, there is one issue. If the controller is already failed, or disabled via a “cfgadm –f –c unconfigure” command, then it will NOT show up in the above!
So another way to find the controllers is with the command:

vxdmpadm getsubpaths enclosure=df8001

Take the output of this command and on each disk, take the “t” number, minus the last digit and search for it in the output of “cfgadm –al”.

So if the output of vxdmpadm getsubpaths enclosure=df8001 is:

root@hostc: vxdmpadm getsubpaths enclosure=df8001
NAME         STATE[A]   PATH-TYPE[M] DMPNODENAME  ENCLR-NAME   CTLR    ATTRS
================================================================================
c6t50060E80100AACFBd0s2 DISABLED     -          df8001_1598  df8001       c6        -
c8t50060E80100AACF3d0s2 DISABLED     -          df8001_1598  df8001       c8        -
c6t50060E80100AACFBd1s2 DISABLED     -          df8001_1599  df8001       c6        -
…TRUNCATED…
c4t50060E80100AACFFd53s2 DISABLED     -          df8001_2137  df8001       c4        -
c4t50060E80100AACFFd54s2 DISABLED     -          df8001_2138  df8001       c4        -
c4t50060E80100AACFFd55s2 DISABLED     -          df8001_2139  df8001       c4        -
c4t50060E80100AACFFd56s2 DISABLED     -          df8001_2140  df8001       c4        -
c4t50060E80100AACFFd57s2 DISABLED     -          df8001_2141  df8001       c4        -
c4t50060E80100AACFFd58s2 DISABLED     -          df8001_2142  df8001       c4        -
c4t50060E80100AACFFd59s2 DISABLED     -          df8001_2143  df8001       c4        -
c4t50060E80100AACFFd60s2 DISABLED     -          df8001_2144  df8001       c4        -
c4t50060E80100AACFFd61s2 DISABLED     -          df8001_2145  df8001       c4        -
c4t50060E80100AACFFd62s2 DISABLED     -          df8001_2146  df8001       c4        -
c4t50060E80100AACFFd63s2 DISABLED     -          df8001_2147  df8001       c4        -
c4t50060E80100AACFFd64s2 DISABLED     -          df8001_2148  df8001       c4        -
c4t50060E80100AACFFd65s2 DISABLED     -          df8001_2149  df8001       c4        -
c4t50060E80100AACFFd66s2 DISABLED     -          df8001_2150  df8001       c4        -
c4t50060E80100AACFFd67s2 DISABLED     -          df8001_2151  df8001       c4        -

Then search for 50060E80100AACF <-Note we took the last character, a second "F" off, as it changes for each controller.

root@hostc: cfgadm -al|grep -i 50060E80100AACF
c2::50060e80100aacf3           unavailable  connected    unconfigured failed
c2::50060e80100aacf7           disk         connected    unconfigured unknown
c4::50060e80100aacfb           unavailable  connected    unconfigured failed
c4::50060e80100aacff           disk         connected    configured   unknown
c6::50060e80100aacfb           disk         connected    configured   unknown
c8::50060e80100aacf3           disk         connected    configured   unknown

So in that case, it is C2, C4, C6 and C8. (Sorry that this interlude is from another system.)

Now use fcinfo hba-port to find the Port WWNs:

root@hosta: fcinfo hba-port
HBA Port WWN: 2100001b329e9a24
        OS Device Name: /dev/cfg/c2
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764999
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2000001b329e9a24
HBA Port WWN: 2101001b32be9a24
        OS Device Name: /dev/cfg/c3
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764999
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2001001b32be9a24
HBA Port WWN: 2100001b329ec032
        OS Device Name: /dev/cfg/c4 <- HERE Is controller 4, and just ABOVE it is the Port WWN.
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764933
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2000001b329ec032 <- In this case the Node WWN is the same, but that is NOT always the case.
HBA Port WWN: 2101001b32bec032
        OS Device Name: /dev/cfg/c5
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764933
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2001001b32bec032
HBA Port WWN: 2100001b329d7e4c
        OS Device Name: /dev/cfg/c6
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764948
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2000001b329d7e4c
HBA Port WWN: 2101001b32bd7e4c
        OS Device Name: /dev/cfg/c7
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006764948
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2001001b32bd7e4c
HBA Port WWN: 2100001b329e5731
        OS Device Name: /dev/cfg/c8 <- Here is controller c8 and, again, the Port WWN is just above it!
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006765424
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2000001b329e5731
HBA Port WWN: 2101001b32be5731
        OS Device Name: /dev/cfg/c9
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402H00-1006765424
        Driver Name: qlc
        Driver Version: 3.00p
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2001001b32be5731

That is it. Give those Port WWNs to the array team, and they should be able to “zone” the disks so that they appear to your system, and only your system!

Now the next commands are helpful for correlation. Also, fcinfo does not exist on Solairs 9 and below, so you can use the below to find port WWNs on Solaris 8 and 9. You can also use this information for trouble shooting and correlating disks to controllers.

root@hosta: luxadm -e port
/devices/pci@400/pci@0/pci@c/SUNW,qlc@0/fp@0,0:devctl              NOT CONNECTED
/devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0:devctl            CONNECTED
/devices/pci@500/pci@0/pci@d/SUNW,qlc@0/fp@0,0:devctl              CONNECTED
/devices/pci@500/pci@0/pci@d/SUNW,qlc@0,1/fp@0,0:devctl            NOT CONNECTED
/devices/pci@600/pci@0/pci@9/SUNW,qlc@0/fp@0,0:devctl              NOT CONNECTED
/devices/pci@600/pci@0/pci@9/SUNW,qlc@0,1/fp@0,0:devctl            CONNECTED
/devices/pci@700/pci@0/pci@c/SUNW,qlc@0/fp@0,0:devctl              CONNECTED
/devices/pci@700/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0:devctl            NOT CONNECTED

Now, the way to correlate the above devices to controllers is with ls like this:

root@hosta: ls -l /dev/dsk | grep pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s0 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:a
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s1 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:b
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s2 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:c
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s3 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:d
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s4 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:e
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s5 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:f
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s6 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:g
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d0s7 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,0:h
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s0 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:a
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s1 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:b
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s2 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:c
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s3 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:d
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s4 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:e
lrwxrwxrwx   1 root     root          79 Apr 10 22:26 c3t50060E8006FE2732d10s5 -> ../../devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0/ssd@w50060e8006fe2732,a:f

This will go on for a long time if you have a lot of disks.
Note how I cut off “devices/” at the front and “:devctl” at the end of the luxadm device name and then grep for that name in the output of ls –l /dev/dsk

What should happen is that EVERY disk listed should have the SAME C number. In this case that device is c3.

So what WWN is c3 on?

root@hosta: luxadm -e dump_map /devices/pci@400/pci@0/pci@c/SUNW,qlc@0,1/fp@0,0:devctl
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    32004e  0         50060e8006fe2732 50060e8006fe2732 0x0  (Disk device)
1    320062  0         50060e8006fe2734 50060e8006fe2734 0x0  (Disk device)
2    32009a  0         50060e80100584a9 50060e80100584a9 0x0  (Disk device)
3    340600  0         2101001b32be9a24 2001001b32be9a24 0x1f (Unknown Type,Host Bus Adapter)

Trouble shooting NOTE: The end of each line should say “Disk device”, except for the last, if they say something else here, something is wrong on the Array side!
The last line is the WWN for c3, and the column heading tells you that the FIRST one is the Port WWN.

ALSO note that the (Disk device) WWNs, while not helpful to the Array team, DO correlate with what is in “vxdisk –e list” output, so you can use this to find out what disks are connected to this controller!
(Although, the ls –l method kind of already told you that, this makes a full circle and could help in many trouble shooting and correlation situations.)

NOTE: I used the terms “disk” and “LUN” interchangeably. I hope you understand that while there are very real disks in the array, what you see on your end are just “pretend disks” that are probably made up of striped volumes in the array. “LUN” is an old SCSI term, but it has come to be used for “disk” too. Either way, what we see as “disks” or “luns” are just whatever the array operator decides to show us, and that is what is important here.

Posted in Uncategorized | Leave a comment

Finding Global Zone from Local Zone

It is difficult if not impossible, by design, to find your “global zone” host name if you are in a Solaris 10 Local Zone.

One method that works sometimes is to run “arp –a|grep SP” and that should list the global zone and all local zones if your network is set up just right.

(FYI: I did not come up with this, just posting it here for reference.)

Posted in Uncategorized | 1 Comment

Migrate IPMP system from active/active to active/standby aka active/passive

ifconfig bge2 standby <- will push the IP on bge2 over to the other NIC move addif from hostname.bge2 to hostname.bge0 root@hostname: cat /etc/hostname.bge0 group prod1 hostname-prod1-if1 deprecated -failover up addif hostname up root@hostname: cat /etc/hostname.bge2 group prod1 hostname-prod1-if2 deprecated -failover up addif hostname-bkup up Add "standby" to bge2 line root@hostname: cat /etc/hostname.bge0 group prod1 hostname-prod1-if1 deprecated -failover up addif hostname up addif hostname-bkup up root@hostname: cat /etc/hostname.bge2 group prod1 hostname-prod1-if2 deprecated -failover standby up

Posted in Uncategorized | Leave a comment

Veritas Disk group has no valid configuration copies

After applying some Solaris 10 patches and rebooting I got this error during boot from one of my disk groups:

vxvm:vxconfigd: V-5-1-569 Disk group appldg, Disk c3t50060482D5303536d13s2: Cannot auto-import group:
vxvm:vxconfigd:         Disk group has no valid configuration copies

And if I try to import it by hand after logging in:

# vxdg import appldg
VxVM vxdg ERROR V-5-1-10978 Disk group appldg: import failed:
Disk group has no valid configuration copies

There was no disk corruption, and booting back to an unpatched copy of the OS worked fine, so the issue wasn’t with the actual disk group.

What solved the problem was adding more copies of the configuration to the disks.

Here is the vxdg list output before:

# vxdg list appldg
Group:     appldg
dgid:      1210705160.16.ssprd131
import-id: 1024.18
flags:     cds
version:   140
alignment: 8192 (bytes)
ssb:            on
autotagging:    off
detach-policy: global
dg-fail-policy: dgdisable
copies:    nconfig=default nlog=default
config:    seqno=0.1170 permlen=48144 free=48126 templen=9 loglen=7296
config disk c3t50060482D5303536d6s2 copy 1 len=48144 disabled
config disk c3t50060482D5303536d7s2 copy 1 len=48144 disabled
config disk c3t50060482D5303536d8s2 copy 1 len=48144 disabled
config disk c3t50060482D5303536d11s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d12s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d13s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d14s2 copy 1 len=48144 disabled
config disk c3t50060482D5303536d15s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d16s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d43s2 copy 1 len=48144 disabled
log disk c3t50060482D5303536d6s2 copy 1 len=7296 disabled
log disk c3t50060482D5303536d7s2 copy 1 len=7296 disabled
log disk c3t50060482D5303536d8s2 copy 1 len=7296 disabled
log disk c3t50060482D5303536d11s2 copy 1 len=7296
log disk c3t50060482D5303536d12s2 copy 1 len=7296
log disk c3t50060482D5303536d13s2 copy 1 len=7296
log disk c3t50060482D5303536d14s2 copy 1 len=7296
log disk c3t50060482D5303536d15s2 copy 1 len=7296
log disk c3t50060482D5303536d16s2 copy 1 len=7296 disabled
log disk c3t50060482D5303536d43s2 copy 1 len=7296 disabled

And then, using an unpatched copy of the OS, in single user mode, I ran this:

# vxedit -g appldg set nconfig=all appldg
# vxedit -g appldg set nlog=all appldg

and this is how it changed the output in vxdg list:

# vxdg list appldg
Group:     appldg
dgid:      1210705160.16.ssprd131
import-id: 1024.22
flags:     cds
version:   140
alignment: 8192 (bytes)
ssb:            on
autotagging:    off
detach-policy: global
dg-fail-policy: dgdisable
copies:    nconfig=all nlog=all
config:    seqno=0.1176 permlen=48144 free=48126 templen=9 loglen=7296
config disk c3t50060482D5303536d6s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d7s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d8s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d11s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d12s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d13s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d14s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d15s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d16s2 copy 1 len=48144 state=clean online
config disk c3t50060482D5303536d43s2 copy 1 len=48144 state=clean online
log disk c3t50060482D5303536d6s2 copy 1 len=7296
log disk c3t50060482D5303536d7s2 copy 1 len=7296
log disk c3t50060482D5303536d8s2 copy 1 len=7296
log disk c3t50060482D5303536d11s2 copy 1 len=7296
log disk c3t50060482D5303536d12s2 copy 1 len=7296
log disk c3t50060482D5303536d13s2 copy 1 len=7296
log disk c3t50060482D5303536d14s2 copy 1 len=7296
log disk c3t50060482D5303536d15s2 copy 1 len=7296
log disk c3t50060482D5303536d16s2 copy 1 len=7296
log disk c3t50060482D5303536d43s2 copy 1 len=7296

After that the system boots fine with the new OS patches.

I don’t know if one of those config copies is bad, or if under some newer Solaris patch more than five copies of the config are required. I really have no idea why it fixed the problem, but there it is.

Posted in Veritas File System | Leave a comment

clri and the BAD STATE 0x52 TO BLKERR error in fsck

Tonight I had a server reboot that could not pass fsck on the root disk:

root> fsck -F ufs -o f -y /dev/md/rdsk/d30
** /dev/md/rdsk/d30
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
FRAGMENT 1957912 DUP I=228437 LFN 48
FRAGMENT 1957913 DUP I=228437 LFN 49
FRAGMENT 1957914 DUP I=228437 LFN 50
FRAGMENT 1957915 DUP I=228437 LFN 51
FRAGMENT 1957916 DUP I=228437 LFN 52
FRAGMENT 1957917 DUP I=228437 LFN 53
FRAGMENT 1957918 DUP I=228437 LFN 54
FRAGMENT 1957919 DUP I=228437 LFN 55
FRAGMENT 1958752 DUP I=228451 LFN 0
FRAGMENT 1958753 DUP I=228451 LFN 1
FRAGMENT 1958754 DUP I=228451 LFN 2
FRAGMENT 1958755 DUP I=228451 LFN 3
FRAGMENT 1958756 DUP I=228451 LFN 4
FRAGMENT 1958757 DUP I=228451 LFN 5
FRAGMENT 1958758 DUP I=228451 LFN 6
FRAGMENT 1958759 DUP I=228451 LFN 7
FRAGMENT 1958760 DUP I=228451 LFN 8
FRAGMENT 1958761 DUP I=228451 LFN 9
EXCESSIVE DUPLICATE FRAGMENTS I=228451
CONTINUE?  yes

** Phase 1b - Rescan For More DUPS
FRAGMENT 1957912 DUP I=228425 LFN 0
BAD STATE 0x52 TO BLKERR

The results were the same on every fsck run. Through digging in documents online I found referenc to the “clri” command.

man clri

gives the usage.

It worked!

root> clri /dev/md/rdsk/d30 228451
clearing 228451
root> fsck -F ufs -o f -y /dev/md/rdsk/d30
** /dev/md/rdsk/d30
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
FRAGMENT 1957912 DUP I=228437 LFN 48
FRAGMENT 1957913 DUP I=228437 LFN 49
FRAGMENT 1957914 DUP I=228437 LFN 50
FRAGMENT 1957915 DUP I=228437 LFN 51
FRAGMENT 1957916 DUP I=228437 LFN 52
FRAGMENT 1957917 DUP I=228437 LFN 53
FRAGMENT 1957918 DUP I=228437 LFN 54
FRAGMENT 1957919 DUP I=228437 LFN 55
** Phase 1b - Rescan For More DUPS
FRAGMENT 1957912 DUP I=228425 LFN 0
BAD STATE 0x52 TO BLKERR

root> clri /dev/md/rdsk/d30 228437
clearing 228437

root> fsck -F ufs -o f -y /dev/md/rdsk/d30
** /dev/md/rdsk/d30
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
UNALLOCATED  I=228451  OWNER=root MODE=0
SIZE=0 MTIME=Jan  1 00:00 1970
NAME=/etc/svc/repository-boot-20110906_195222

REMOVE DIRECTORY ENTRY FROM I=228663?  yes

UNALLOCATED  I=228437  OWNER=root MODE=0
SIZE=0 MTIME=Jan  1 00:00 1970
NAME=/dev/.devlink_db

REMOVE DIRECTORY ENTRY FROM I=227903?  yes

** Phase 3a - Check Connectivity
** Phase 3b - Verify Shadows/ACLs
** Phase 4 - Check Reference Counts
UNREF FILE  I=228425  OWNER=root MODE=100644
SIZE=36581 MTIME=Jul 21 15:19 2011
RECONNECT?  yes

LINK COUNT FILE I=228425  OWNER=root MODE=100644
SIZE=36581 MTIME=Jul 21 15:19 2011  COUNT 0 SHOULD BE 1
ADJUST?  yes

** Phase 5 - Check Cylinder Groups

CORRECT BAD CG SUMMARIES?  yes

CORRECTED SUMMARY FOR CG 39
FILE BITMAP WRONG
FIX?  yes

FRAG BITMAP WRONG (CORRECTED)
CORRECTED SUMMARY FOR CG 52
FRAG BITMAP WRONG (CORRECTED)
CORRECTED SUMMARY FOR CG 66
FRAG BITMAP WRONG (CORRECTED)
CORRECT GLOBAL SUMMARY
SALVAGE?  yes

133998 files, 3967637 used, 1198465 free (208129 frags, 123792 blocks, 4.0% fragmentation)

***** FILE SYSTEM WAS MODIFIED *****


root> fsck -F ufs -o f -y /dev/md/rdsk/d30
** /dev/md/rdsk/d30
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3a - Check Connectivity
** Phase 3b - Verify Shadows/ACLs
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cylinder Groups

FILE SYSTEM STATE IN SUPERBLOCK IS WRONG; FIX?  yes

133998 files, 3967637 used, 1198465 free (208129 frags, 123792 blocks, 4.0% fragmentation)

root> fsck -F ufs -y /dev/md/rdsk/d30
** /dev/md/rdsk/d30
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes

** Phase 2 - Check Pathnames
** Phase 3a - Check Connectivity
** Phase 3b - Verify Shadows/ACLs
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cylinder Groups
133998 files, 3967637 used, 1198465 free (208129 frags, 123792 blocks, 4.0% fragmentation)
Posted in Uncategorized | Leave a comment

Large file transfer failure with IPMP

I had an issue today where large file transfers via SCP from within the data center were failing intermittently to an IPMP NIC pair. We quickly found that the transfer always failed to one IP but not the other.

Half of the IPMP pair is running on a CE interface at gigabit.
The other half on an ERI interface that only support 100megabit.

Both interfaces ping perfectly well and SSH works to both just fine.
From outside of the data center, large files transfer to either IP just fine.
However, from inside of the data center large file transfers stall at 152K.
Also the ERI NICs report errors.

Failed test:

$ scp /tmp/USERtestDELETEME 192.168.1.100:/tmp/USERtestDELETEME2
USERtestDELETEME   100% |******************************************************************************************************************************|   263 MB    00:10

$ scp /tmp/USERtestDELETEME 192.168.1.101:/tmp/USERtestDELETEME2
USERtestDELETEME     0% |                                                                                                                              |   152 KB  - stalled -^CKilled by signal 2.

Previous settings:

Network speed settings for HOSTNAME

NDD settings for ce Instance 0

adv_autoneg_cap 1
adv_1000fdx_cap 1
adv_1000hdx_cap 1
adv_100T4_cap 0
adv_100fdx_cap 1
adv_100hdx_cap 1
adv_10fdx_cap 1
adv_10hdx_cap 1

NDD settings for eri Instance 0

adv_autoneg_cap 0
adv_100T4_cap 0
adv_100fdx_cap 1
adv_100hdx_cap 0
adv_10fdx_cap 0
adv_10hdx_cap 0

Interface Link Speed Duplex Link Partner Autoneg LP Setting
_________ ____ _____ ______ ____________________ __________
ce0 UP 1000 FULL ENABLED 1000_FULL
ce1 UP 1000 FULL ENABLED 1000_FULL
eri0 UP 100 FULL DISABLED 100_FULL

Root Cause:
My theory is that the switch ports are running at gigabit when the NIC is locked to 100MB and auto negotiation is turned off.
If the traffic comes in at under 100MB then the NIC handles it fine, but if the traffic exceeds that level, data is corrupted, packets drop and the tranfer stalls.
By turning on auto negotiation the NIC is able to tell the switch port to slow down to 100MB.

Solution:
Turn on auto negotiation for the ERI0 NIC.

Changes made:

ndd -set /dev/eri instance 0
ndd -set /dev/eri adv_autoneg_cap 1

New Settings:

Network speed settings for HOSTNAME

NDD settings for ce Instance 0

adv_autoneg_cap 1
adv_1000fdx_cap 1
adv_1000hdx_cap 1
adv_100T4_cap 0
adv_100fdx_cap 1
adv_100hdx_cap 1
adv_10fdx_cap 1
adv_10hdx_cap 1

NDD settings for eri Instance 0

adv_autoneg_cap 1
adv_100T4_cap 0
adv_100fdx_cap 1
adv_100hdx_cap 0
adv_10fdx_cap 0
adv_10hdx_cap 0

Interface Link Speed Duplex Link Partner Autoneg LP Setting
_________ ____ _____ ______ ____________________ __________
ce0 UP 1000 FULL ENABLED 1000_FULL
ce1 UP 1000 FULL ENABLED 1000_FULL
eri0 UP 100 FULL ENABLED 100_FULL

Test Passes:

$ scp /tmp/USERtestDELETEME 192.168.1.100:/tmp/USERtestDELETEME2
USERtestDELETEME   100% |******************************************************************************************************************************|   263 MB    00:10

$ scp /tmp/USERtestDELETEME 192.168.1.101:/tmp/USERtestDELETEME2
USERtestDELETEME   100% |******************************************************************************************************************************|   263 MB    00:23
Posted in Network Settings | Leave a comment

ZFS hot spares

If you have more disks than you need to use in your ZFS pool, then by all means, give them to ZFS to use as “hot spares.”

Here is an experiment to show the process and results:

# zpool create testpool mirror c0t2d0 c0t3d0 mirror c0t4d0 c0t5d0
# zpool status
  pool: testpool
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        testpool    ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c0t3d0  ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0
            c0t5d0  ONLINE       0     0     0

errors: No known data errors

# zpool add testpool spare c0t6d0 c0t7d0
# zpool status
  pool: testpool
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        testpool    ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c0t3d0  ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0
            c0t5d0  ONLINE       0     0     0
        spares
          c0t6d0    AVAIL
          c0t7d0    AVAIL

errors: No known data errors

# zpool status
  pool: testpool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        testpool    DEGRADED     0     0     0
          mirror-0  DEGRADED     0     0     0
            c0t2d0  UNAVAIL      0     0     0  cannot open
            c0t3d0  ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0
            c0t5d0  ONLINE       0     0     0
        spares
          c0t6d0    AVAIL
          c0t7d0    AVAIL

errors: No known data errors

# zpool set autoreplace=on testpool

Oct 28 15:36:16 unknown EVENT-TIME: Fri Oct 28 15:36:16 CDT 2011
Oct 28 15:36:16 unknown PLATFORM: VirtualBox, CSN: 0, HOSTNAME: unknown
Oct 28 15:36:16 unknown SOURCE: zfs-diagnosis, REV: 1.0
Oct 28 15:36:16 unknown EVENT-ID: fb216b01-1dba-ebb6-8404-d7bcfd52bdbe
Oct 28 15:36:16 unknown DESC: A ZFS device failed.  Refer to http://sun.com/msg/ZFS-8000-D3 for more information.
Oct 28 15:36:16 unknown AUTO-RESPONSE: No automated response will occur.
Oct 28 15:36:16 unknown IMPACT: Fault tolerance of the pool may be compromised.
Oct 28 15:36:16 unknown REC-ACTION: Run 'zpool status -x' and replace the bad device.

# zpool status
  pool: testpool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scan: resilvered 52.5K in 0h0m with 0 errors on Fri Oct 28 15:36:17 2011
config:

        NAME          STATE     READ WRITE CKSUM
        testpool      DEGRADED     0     0     0
          mirror-0    DEGRADED     0     0     0
            spare-0   DEGRADED     0     0     0
              c0t2d0  UNAVAIL      0     0     0  cannot open
              c0t6d0  ONLINE       0     0     0
            c0t3d0    ONLINE       0     0     0
          mirror-1    ONLINE       0     0     0
            c0t4d0    ONLINE       0     0     0
            c0t5d0    ONLINE       0     0     0
        spares
          c0t6d0      INUSE     currently in use
          c0t7d0      AVAIL

errors: No known data errors

Oct 28 15:41:17 unknown EVENT-TIME: Fri Oct 28 15:36:16 CDT 2011
Oct 28 15:41:17 unknown PLATFORM: VirtualBox, CSN: 0, HOSTNAME: unknown
Oct 28 15:41:17 unknown SOURCE: zfs-diagnosis, REV: 1.0
Oct 28 15:41:17 unknown EVENT-ID: fb216b01-1dba-ebb6-8404-d7bcfd52bdbe
Oct 28 15:41:17 unknown DESC: All faults associated with an event id have been addressed.
Oct 28 15:41:17 unknown   Refer to http://sun.com/msg/FMD-8000-6U for more information.
Oct 28 15:41:17 unknown AUTO-RESPONSE: All system components offlined because of the original fault have been brought back online.
Oct 28 15:41:17 unknown IMPACT: Performance degradation of the system due to the original fault has been recovered.
Oct 28 15:41:17 unknown REC-ACTION: Use fmdump -v -u <EVENT-ID> to identify the repaired components.

# zpool status
  pool: testpool
 state: ONLINE
 scan: resilvered 52.5K in 0h0m with 0 errors on Fri Oct 28 15:36:17 2011
config:

        NAME          STATE     READ WRITE CKSUM
        testpool      ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            spare-0   ONLINE       0     0     0
              c0t2d0  ONLINE       0     0     0
              c0t6d0  ONLINE       0     0     0
            c0t3d0    ONLINE       0     0     0
          mirror-1    ONLINE       0     0     0
            c0t4d0    ONLINE       0     0     0
            c0t5d0    ONLINE       0     0     0
        spares
          c0t6d0      INUSE     currently in use
          c0t7d0      AVAIL

errors: No known data errors

# zpool status
  pool: testpool
 state: ONLINE
 scan: resilvered 9.50K in 0h0m with 0 errors on Fri Oct 28 15:42:08 2011
config:

        NAME        STATE     READ WRITE CKSUM
        testpool    ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c0t3d0  ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0
            c0t5d0  ONLINE       0     0     0
        spares
          c0t6d0    AVAIL
          c0t7d0    AVAIL

errors: No known data errors

As you can see if you add a spare AND you turn on autoreplace, then disks can come and go as they please and ZFS will deal with it by itself.

Obviously this simple disk pulling is a best case scenario. ZFS may or may not perform as admirably if the disk issue is more complex than a simple offline.

Here is the same example taken further:

Oct 28 15:46:29 unknown EVENT-TIME: Fri Oct 28 15:46:29 CDT 2011
Oct 28 15:46:29 unknown PLATFORM: VirtualBox, CSN: 0, HOSTNAME: unknown
Oct 28 15:46:29 unknown SOURCE: zfs-diagnosis, REV: 1.0
Oct 28 15:46:29 unknown EVENT-ID: e42609fe-da81-c68c-bea2-a232a2a5b6d0
Oct 28 15:46:29 unknown DESC: A ZFS device failed.  Refer to http://sun.com/msg/ZFS-8000-D3 for more information.
Oct 28 15:46:29 unknown AUTO-RESPONSE: No automated response will occur.
Oct 28 15:46:29 unknown IMPACT: Fault tolerance of the pool may be compromised.
Oct 28 15:46:29 unknown REC-ACTION: Run 'zpool status -x' and replace the bad device.
Oct 28 15:46:29 unknown fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major

Oct 28 15:46:29 unknown EVENT-TIME: Fri Oct 28 15:46:29 CDT 2011
Oct 28 15:46:29 unknown PLATFORM: VirtualBox, CSN: 0, HOSTNAME: unknown
Oct 28 15:46:29 unknown SOURCE: zfs-diagnosis, REV: 1.0
Oct 28 15:46:29 unknown EVENT-ID: aaaac038-7699-eff1-aede-800ef60c7408
Oct 28 15:46:29 unknown DESC: A ZFS device failed.  Refer to http://sun.com/msg/ZFS-8000-D3 for more information.
Oct 28 15:46:29 unknown AUTO-RESPONSE: No automated response will occur.
Oct 28 15:46:29 unknown IMPACT: Fault tolerance of the pool may be compromised.
Oct 28 15:46:29 unknown REC-ACTION: Run 'zpool status -x' and replace the bad device.

# zpool status
  pool: testpool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scan: resilvered 71.5K in 0h0m with 0 errors on Fri Oct 28 15:46:31 2011
config:

        NAME          STATE     READ WRITE CKSUM
        testpool      DEGRADED     0     0     0
          mirror-0    DEGRADED     0     0     0
            spare-0   DEGRADED     0     0     0
              c0t2d0  UNAVAIL      0     0     0  cannot open
              c0t6d0  ONLINE       0     0     0
            c0t3d0    ONLINE       0     0     0
          mirror-1    DEGRADED     0     0     0
            spare-0   DEGRADED     0     0     0
              c0t4d0  UNAVAIL      0     0     0  cannot open
              c0t7d0  ONLINE       0     0     0
            c0t5d0    ONLINE       0     0     0
        spares
          c0t6d0      INUSE     currently in use
          c0t7d0      INUSE     currently in use

errors: No known data errors

# digest -a md5 -v /testpool/testfile
md5 (/testpool/testfile) = bb449a441fd64200213bf0217b13414b

Oct 28 15:52:46 unknown EVENT-TIME: Fri Oct 28 15:52:46 CDT 2011
Oct 28 15:52:46 unknown PLATFORM: VirtualBox, CSN: 0, HOSTNAME: unknown
Oct 28 15:52:46 unknown SOURCE: zfs-diagnosis, REV: 1.0
Oct 28 15:52:46 unknown EVENT-ID: 12273a2a-c3a5-6ff2-99fc-b68b9c06dcdf
Oct 28 15:52:46 unknown DESC: A ZFS device failed.  Refer to http://sun.com/msg/ZFS-8000-D3 for more information.
Oct 28 15:52:46 unknown AUTO-RESPONSE: No automated response will occur.
Oct 28 15:52:46 unknown IMPACT: Fault tolerance of the pool may be compromised.
Oct 28 15:52:46 unknown REC-ACTION: Run 'zpool status -x' and replace the bad device.
Oct 28 15:52:46 unknown fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major

Oct 28 15:52:46 unknown EVENT-TIME: Fri Oct 28 15:52:46 CDT 2011
Oct 28 15:52:46 unknown PLATFORM: VirtualBox, CSN: 0, HOSTNAME: unknown
Oct 28 15:52:46 unknown SOURCE: zfs-diagnosis, REV: 1.0
Oct 28 15:52:46 unknown EVENT-ID: 9a80e839-c349-e246-e6e2-915bc0718bc1
Oct 28 15:52:46 unknown DESC: A ZFS device failed.  Refer to http://sun.com/msg/ZFS-8000-D3 for more information.
Oct 28 15:52:46 unknown AUTO-RESPONSE: No automated response will occur.
Oct 28 15:52:46 unknown IMPACT: Fault tolerance of the pool may be compromised.
Oct 28 15:52:46 unknown REC-ACTION: Run 'zpool status -x' and replace the bad device.

# zpool status
  pool: testpool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scan: resilvered 71.5K in 0h0m with 0 errors on Fri Oct 28 15:46:31 2011
config:

        NAME          STATE     READ WRITE CKSUM
        testpool      DEGRADED     0     0     0
          mirror-0    DEGRADED     0     0     0
            spare-0   DEGRADED     0     0     0
              c0t2d0  UNAVAIL      0     0     0  cannot open
              c0t6d0  ONLINE       0     0     0
            c0t3d0    UNAVAIL      0     0     0  cannot open
          mirror-1    DEGRADED     0     0     0
            spare-0   DEGRADED     0     0     0
              c0t4d0  UNAVAIL      0     0     0  cannot open
              c0t7d0  ONLINE       0     0     0
            c0t5d0    UNAVAIL      0     0     0  cannot open
        spares
          c0t6d0      INUSE     currently in use
          c0t7d0      INUSE     currently in use

errors: No known data errors

# digest -a md5 -v /testpool/testfile
md5 (/testpool/testfile) = bb449a441fd64200213bf0217b13414b

Now we have lost all four original drives, but the file system content is still in tact.

# zpool status
  pool: testpool
 state: ONLINE
 scan: resilvered 71.5K in 0h0m with 0 errors on Fri Oct 28 15:46:31 2011
config:

        NAME          STATE     READ WRITE CKSUM
        testpool      ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            spare-0   ONLINE       0     0     0
              c0t2d0  ONLINE       0     0     0
              c0t6d0  ONLINE       0     0     0
            c0t3d0    ONLINE       0     0     0
          mirror-1    ONLINE       0     0     0
            spare-0   ONLINE       0     0     0
              c0t4d0  ONLINE       0     0     0
              c0t7d0  ONLINE       0     0     0
            c0t5d0    ONLINE       0     0     0
        spares
          c0t6d0      INUSE     currently in use
          c0t7d0      INUSE     currently in use

errors: No known data errors

# zpool status
  pool: testpool
 state: ONLINE
 scan: resilvered 354K in 0h0m with 0 errors on Fri Oct 28 15:57:19 2011
config:

        NAME        STATE     READ WRITE CKSUM
        testpool    ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c0t3d0  ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0
            c0t5d0  ONLINE       0     0     0
        spares
          c0t6d0    AVAIL
          c0t7d0    AVAIL

errors: No known data errors

# digest -a md5 -v /testpool/testfile
md5 (/testpool/testfile) = bb449a441fd64200213bf0217b13414b

Put the disks back, and everything returns to normal all by itself, with no loss of file system integrity.

Obviously if we had not had time for the hot spare to silver before the next disk went in it would not have worked.

In this particular setup, it would probably actually make more sense to just mirror across three disks, so that you could lose two at once and still survive.

Also RAIDZ or RAIDZ2 could be used, allowing you to lose 1 or 2 disks respectively, and you can also use hot spares with RAIDZ.

# zpool destroy testpool
# zpool create testpool raidz2 c0t2d0 c0t3d0 c0t4d0 c0t5d0 c0t6d0
# zpool add testpool spare c0t7d0
# zpool status
  pool: testpool
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        testpool    ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c0t3d0  ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0
            c0t5d0  ONLINE       0     0     0
            c0t6d0  ONLINE       0     0     0
        spares
          c0t7d0    AVAIL

errors: No known data errors
# zpool set autoreplace=on testpool
# cp /var/adm/messages /testpool/testfile
# digest -a md5 -v /testpool/testfile
md5 (/testpool/testfile) = 336cd7cbdf0926fbf04392a3ed9b512b

# Remove TWO disks at once
# RAIDZ2 keeps two parity disks, so it can lose two at once

# zpool status
  pool: testpool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scan: resilvered 166K in 0h0m with 0 errors on Fri Oct 28 16:07:58 2011
config:

        NAME          STATE     READ WRITE CKSUM
        testpool      DEGRADED     0     0     0
          raidz2-0    DEGRADED     0     0     0
            spare-0   DEGRADED     0     0     0
              c0t2d0  UNAVAIL      0     0     0  cannot open
              c0t7d0  ONLINE       0     0     0
            c0t3d0    UNAVAIL      0     0     0  cannot open
            c0t4d0    ONLINE       0     0     0
            c0t5d0    ONLINE       0     0     0
            c0t6d0    ONLINE       0     0     0
        spares
          c0t7d0      INUSE     currently in use

errors: No known data errors

# digest -a md5 -v /testpool/testfile
md5 (/testpool/testfile) = 336cd7cbdf0926fbf04392a3ed9b512b

# Data is still good and the spare has stepped in for one parity disk

# zpool status
  pool: testpool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scan: resilvered 166K in 0h0m with 0 errors on Fri Oct 28 16:07:58 2011
config:

        NAME          STATE     READ WRITE CKSUM
        testpool      DEGRADED     0     0     0
          raidz2-0    DEGRADED     0     0     0
            spare-0   DEGRADED     0     0     0
              c0t2d0  UNAVAIL      0     0     0  cannot open
              c0t7d0  ONLINE       0     0     0
            c0t3d0    UNAVAIL      0     0     0  cannot open
            c0t4d0    UNAVAIL      0     0     0  cannot open
            c0t5d0    ONLINE       0     0     0
            c0t6d0    ONLINE       0     0     0
        spares
          c0t7d0      INUSE     currently in use

errors: No known data errors

# Third disk gone, but still ok, as long as ZFS had time to rebuild the parity disk before it went away

# digest -a md5 -v /testpool/testfile
md5 (/testpool/testfile) = 336cd7cbdf0926fbf04392a3ed9b512b

# zpool status
  pool: testpool
 state: UNAVAIL
status: One or more devices could not be opened.  There are insufficient
        replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-3C
 scan: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        testpool      UNAVAIL      0     0     0  insufficient replicas
          raidz2-0    UNAVAIL      0     0     0  insufficient replicas
            spare-0   DEGRADED     0     0     0
              c0t2d0  UNAVAIL      0     0     0  cannot open
              c0t7d0  ONLINE       0     0     0
            c0t3d0    UNAVAIL      0     0     0  cannot open
            c0t4d0    UNAVAIL      0     0     0  cannot open
            c0t5d0    UNAVAIL      0     0     0  cannot open
            c0t6d0    ONLINE       0     0     0

# Ok, now we went too far. The volume is now offline due to "insufficient replicas"
# There is no no data mounted at /testpool

# But, in case you wonder, yes, if you just put the drives all back, ZFS will figure it out!

# zpool status
  pool: testpool
 state: ONLINE
 scan: resilvered 52.5K in 0h0m with 0 errors on Fri Oct 28 16:19:38 2011
config:

        NAME        STATE     READ WRITE CKSUM
        testpool    ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c0t3d0  ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0
            c0t5d0  ONLINE       0     0     0
            c0t6d0  ONLINE       0     0     0
        spares
          c0t7d0    AVAIL

errors: No known data errors
# digest -a md5 -v /testpool/testfile
md5 (/testpool/testfile) = 336cd7cbdf0926fbf04392a3ed9b512b

# That won't help if two non spares plus all non spares truly go bad, but if you just make a mistake while swapping drives, you should still be able to recover.

And all of that was mostly just for my own entertainment, and also so I’ll have something to look at the next time I have a ZFS problem and forget how it works.

Posted in Uncategorized | Leave a comment

Solaris Console GUI

To disable the GUI auto startup:/usr/dt/bin/dtconfig -d
To ENable the GUI auto startup: /usr/dt/bin/dtconfig -e

Posted in Uncategorized | Leave a comment

Solaris Single User Mode on x86/i386

This may seem silly, but I work exclusively on SPARC boxes, so when I need to get an x86 box into single user mode, such as in VirtualBox, I am lost.

To get Solaris x86 into single use mode:

1. At the “GRUB” boot menu where you get the boot options press ‘e’ to edit the boot line.
2. Use the arrow keys to highlight the line that starts with “kernel”
3. Press ‘e’ again to edit that line.
4. Add ” -s” to the end of that line and press
5. Press ‘b’ to boot

That is it.

FYI, in case you are an x86 guy looking for SPARC instructions, the way to boot into single user mode is to get to the “OBP” prompt and run “boot -s”.
If you are not sure how to get to the “OBP” prompt, run “shutdown -i0” from the command line of a running Solaris box. WARNING! This will shut down Solaris and leave you at the “OBP” prompt, if you are at the console. You will only be able to access the box from the console now. No SSH or Telnet, so make sure you have console access before doing this!

Posted in Uncategorized | 2 Comments

Changing network settings on an M5000 (or M9000) XSCF

First, here is the documentation on the XSCF commands:
http://download.oracle.com/docs/cd/E19580-01/821-2797-10/21ch2p.html#50450504_62902

Second, you will have to reset (“reboot”) the XSCF to apply any network setting changes. This will NOT reboot the running operating system, but it does take a few minutes.

Here is an example of correcting the default router for an XSCF network card:

XSCF> ping host.domain.com
XSCF> ping 192.168.60.245
PING 192.168.60.245 (192.168.60.245): 56 data bytes

--- 192.168.60.245 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

XSCF> shownetwork -a
xscf#0-lan#0
Link encap:Ethernet HWaddr 00:21:28:A4:BF:06
inet addr:192.168.244.22 Bcast:192.168.244.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:9252517 errors:0 dropped:0 overruns:0 frame:0
TX packets:46509 errors:2 dropped:0 overruns:0 carrier:2
collisions:0 txqueuelen:1000
RX bytes:676265882 (644.9 MiB) TX bytes:3120612 (2.9 MiB)
Base address:0xe000

xscf#0-lan#1
Link encap:Ethernet HWaddr 00:21:28:A4:BF:07
inet addr:192.168.9.39 Bcast:192.168.9.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:23131268 errors:0 dropped:0 overruns:0 frame:0
TX packets:62990 errors:4 dropped:0 overruns:0 carrier:4
collisions:0 txqueuelen:1000
RX bytes:2389000588 (2.2 GiB) TX bytes:4226398 (4.0 MiB)
Base address:0xc000

XSCF> showroute -a
#The above does not work because DNS resolutin is not working
# without a working network connection
XSCF> showroute -an
Destination Gateway Netmask Flags Interface
192.168.244.0 0.0.0.0 255.255.255.0 U xscf#0-lan#0
192.168.9.0 0.0.0.0 255.255.255.0 U xscf#0-lan#1
0.0.0.0 192.168.9.4 0.0.0.0 UG xscf#0-lan#1
0.0.0.0 192.168.244.4 0.0.0.0 UG xscf#0-lan#0

XSCF> setroute -c add -n 0.0.0.0 -g 192.168.9.1 xscf#0-lan#1

XSCF> showroute -an
Destination Gateway Netmask Flags Interface
192.168.244.0 0.0.0.0 255.255.255.0 U xscf#0-lan#0
192.168.9.0 0.0.0.0 255.255.255.0 U xscf#0-lan#1
0.0.0.0 192.168.9.4 0.0.0.0 UG xscf#0-lan#1
0.0.0.0 192.168.244.4 0.0.0.0 UG xscf#0-lan#0

XSCF> applynetwork
The following network settings will be applied:
xscf#0 hostname :HOSTNAME-xscf
DNS domain name :domain.com
nameserver :192.168.210.5
nameserver :192.168.210.205

interface :xscf#0-lan#0
status :up
IP address :192.168.244.22
netmask :255.255.255.0
route :-n 0.0.0.0 -m 0.0.0.0 -g 192.168.244.4

interface :xscf#0-lan#1
status :up
IP address :192.168.9.39
netmask :255.255.255.0
route :-n 0.0.0.0 -m 0.0.0.0 -g 192.168.9.1

Continue? [y|n] :y
Please reset the XSCF by rebootxscf to apply the network settings.
Please confirm that the settings have been applied by executing
showhostname, shownetwork, showroute and shownameserver after rebooting
the XSCF.

XSCF> rebootxscf
The XSCF will be reset. Continue? [y|n] :y
XSCF> execute J00shutdown_start -- complete

. . .

XSCF> showroute -an
Destination Gateway Netmask Flags Interface
192.168.244.0 0.0.0.0 255.255.255.0 U xscf#0-lan#0
192.168.9.0 0.0.0.0 255.255.255.0 U xscf#0-lan#1
0.0.0.0 192.168.9.1 0.0.0.0 UG xscf#0-lan#1
0.0.0.0 192.168.244.4 0.0.0.0 UG xscf#0-lan#0

XSCF> showroute -a
Destination Gateway Netmask Flags Interface
192.168.244.0 * 255.255.255.0 U xscf#0-lan#0
192.168.9.0 * 255.255.255.0 U xscf#0-lan#1
default router1 0.0.0.0 UG xscf#0-lan#1
default router2 0.0.0.0 UG xscf#0-lan#0

XSCF> ping host.domain.com
PING host.domain.com (192.168.60.245): 56 data bytes
64 bytes from 192.168.60.245: icmp_seq=0 ttl=245 time=85.2 ms
64 bytes from 192.168.60.245: icmp_seq=1 ttl=245 time=84.7 ms
64 bytes from 192.168.60.245: icmp_seq=2 ttl=245 time=84.7 ms
64 bytes from 192.168.60.245: icmp_seq=3 ttl=245 time=84.6 ms

--- host.domain.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 84.6/84.8/85.2 ms
Posted in Uncategorized | Leave a comment