iSCSI on Solaris 10 x86

[출처] http://blog.scottlowe.org/2006/12/05/iscsi-on-solaris-10-x86/

Given that I’m neither a Solaris expert (yet) nor an iSCSI expert (yet), I knew that it would be a bit of a challenge to make this work.  Fortunately, a found avery useful blog posting by Frank Berger that gave me the framework I needed to get started.  From there, Sun’s documentation provided the rest of the necessary details.  Perhaps this documentation will prove moderately useful as well.

First, I added the following lines to the /etc/ietd.conf on the CentOS iSCSI target server:

Target iqn.2006-08.net.example:server.lun1
        IncomingUser username complicatedpassword
        Lun 1 Path=/dev/vg00/isanvol1,Type=fileio
        Alias iet-lun1
        MaxConnections          8
        InitialR2T              No
        ImmediateData           Yes

A quick restart of the iSCSI target service and I was all set on the target side.  If you were going to do this yourself in your own environment, you’d need to modify the “IncomingUser” and “Lun X Path”.  In this instance I’m using LVM on CentOS, so my path specifies a logical volume in a volume group.  Your configuration will differ, obviously.  Alternately, if you are using a different iSCSI target implementation, you’d need to configure it appropriately.  (I hope to be able to do some testing against a NetApp iSCSI target in the near future.)

On the initiator side, everything is done with the “iscsiadm” command.  This command is fairly self-explanatory and has built-in help (-?) throughout most of the options, but it did take me a little bit of time to get things working.

First, we have to make sure that the iSCSI initiator is online:

svcs -a | grep iscsi

If disabled, then we can enable it with this command:

svcadm enable svc:/network/iscsi_initiator

From there, we configure the iSCSI initiator:

iscsiadm add discovery-address 10.1.1.1:3260
iscsiadm modify initiator-node -a CHAP
iscsiadm modify initiator-node -H username
iscsiadm modify initiator-node -C
(specify CHAP password)
iscsiadm modify discovery --sendtargets enable

Because I’d also specified a static config as well (dynamic discovery didn’t seem to be working as I expected; more on that in a moment), using “iscsiadm list target” now returned two iSCSI targets.  They appeared to be different targets, and since I do have two targets defined on the iSCSI server (one for VMware and one for this), I didn’t want to take any chances on affecting the VMware LUN.  So, I removed and disabled the dynamic discovery, removed the static config, and then re-added the static config:

iscsiadm add static-config iqn.2006-08.net.example:server.lun1,
10.1.1.1:3260

(This should all be on one line; it was wrapped here for readability.)  After doing that, “iscsiadm list target” showed only a single target identified as “server.lun1”, which assured me that I wasn’t seeing the VMware LUN.

<aside>In a more complex environment, how does one ensure that iSCSI LUNs are properly isolated from unwanted hosts?  The “IncomingUser” parameter was different between my VMware LUN and the raw LUN being presented to Solaris, so in theory I would have been safe.  Better safe than sorry, in my opinion.</aside>

After I was sure that the iSCSI initiator was properly seeing the LUN, then I created a new device, created a new filesystem on that device, and then mounted it:

devfsadm -c iscsi
format (selected new disk identified as iSCSI)
newfs /dev/dsk/c2t1d0s2
mount /dev/dsk/c2t1d0s2 /export/iscsi

Of course, you’ll need to modify the above commands slightly depending upon your configuration, but the overall process should be pretty close to what I’ve outlined above.

Step 1: configuring the iSCSI initiator on both V215 nodes:


root@solcluster03:~> svcadm enable iscsi_initiator
root@solcluster03:~> iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:00144f7109f2.48aa5d81
Initiator node alias: -
Login Parameters (Default/Configured):
Header Digest: NONE/-
Data Digest: NONE/-
Authentication Type: NONE
RADIUS Server: NONE
RADIUS access: unknown
Configured Sessions: 1
root@solcluster03:~> iscsiadm add discovery-address 172.19.107.9
root@solcluster03:~> devfsadm -c iscsi
root@solcluster03:~> iscsiadm modify discovery -t enable
root@solcluster03:~> format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c1t0d0 
/pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/sd@0,0
1. c1t1d0 
/pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/sd@1,0
2. c2t010000144F20E16400002A0048AEB3ABd0 
/scsi_vhci/ssd@g010000144f20e16400002a0048aeb3ab
3. c2t010000144F20E16400002A0048609421d0 
/scsi_vhci/ssd@g010000144f20e16400002a0048609421
Specify disk (enter its number): ^D
root@solcluster03:~>

---------------------------------------------------------------

Step 2: ZFS creation (some uninteresting stuff in there, but I don't 
want to cut something out):


root@solcluster03:~> NOINUSE_CHECK=1 zpool create iscsi 
c2t010000144F20E16400002A0048609421d0
root@solcluster03:~> zfs set mountpoint=/export/iscsi iscsi

root@solcluster03:~> zfs list
NAME USED AVAIL REFER MOUNTPOINT
iscsi 92K 9,78G 24,5K /export/iscsi
root@solcluster03:~>

root@solcluster03:~> zfs set compression=on iscsi
root@solcluster03:~> zfs set backup:zfs:avoidscrub=true iscsi
root@solcluster03:~> zfs set storage:cluster:hasp=true iscsi
root@solcluster03:~> zfs set storage:cluster:avs=false iscsi
root@solcluster03:~> zfs get all iscsi
NAME EIGENSCHAFT WERT QUELLE
iscsi type filesystem -
iscsi creation Mi Sep 3 9:14 2008 -
iscsi used 92K -
iscsi available 9,78G -
iscsi referenced 24,5K -
iscsi compressratio 1.00x -
iscsi mounted yes -
iscsi quota none default
iscsi reservation none default
iscsi recordsize 128K default
iscsi mountpoint /export/iscsi local
iscsi sharenfs off default
iscsi checksum on default
iscsi compression off default
iscsi atime on default
iscsi devices on default
iscsi exec on default
iscsi setuid on default
iscsi readonly off default
iscsi zoned off default
iscsi snapdir hidden default
iscsi aclmode groupmask default
iscsi aclinherit secure default
iscsi canmount on default
iscsi shareiscsi off default
iscsi xattr on default
iscsi storage:cluster:hasp true local
iscsi backup:zfs:avoidscrub true local
iscsi storage:cluster:avs false local
root@solcluster03:~>
root@solcluster03:~> zfs create iscsi/data
root@solcluster03:~> zfs list | grep iscsi
iscsi 158K 9,78G 26,5K /export/iscsi
iscsi/data 24,5K 9,78G 24,5K /export/iscsi/data
root@solcluster03:~>

---------------------------------------------------------------

Step 3: Creating the HA-NFS resource group (called "hanfs-rg")

root@solcluster03:~> clresourcegroup create -p Pathprefix=/export/iscsi 
hanfs-rg
root@solcluster03:~> clresourcegroup status

Cluster Resource Groups ===

Group Name Node Name Suspended Status
---------- --------- --------- ------
hanfs-rg solcluster04 No Unmanaged
solcluster03 No Unmanaged

root@solcluster03:~>

---------------------------------------------------------------

Step 4: Creating a logical hostname resource (would be needed for HA-NFS 
later)

root@solcluster03:~> host solcluster
solcluster.dlan.cinetic.de has address 172.20.5.125
root@solcluster03:~>

root@solcluster03:~> grep solcluster.dlan.cinetic.de /etc/hosts
172.20.5.125 solcluster solcluster.dlan.cinetic.de
root@solcluster03:~>

root@solcluster03:~> clreslogicalhostname create -g hanfs-rg -h 
solcluster hanfs-ip
root@solcluster03:~> clreslogicalhostname status

Cluster Resources ===

Resource Name Node Name State Status Message
------------- --------- ----- --------------
hanfs-ip solcluster04 Offline Offline
solcluster03 Offline Offline

root@solcluster03:~>

---------------------------------------------------------------

Step 5: Creating the HAStoragePlus resource


root@solcluster03:~> clresourcetype register SUNW.HAStoragePlus
root@solcluster03:~>

root@solcluster03:~> clresource create -t SUNW.HAStoragePlus -g hanfs-rg 
-p Zpools=iscsi hanfs-hasp
root@solcluster03:~> clresource list
hanfs-ip
hanfs-hasp
root@solcluster03:~>
---------------------------------------------------------------

Note: The zpool is being exported by the clresource command. Takes some 
time.


Step 6: Taking the resource group online

root@solcluster03:~> clresourcegroup list
hanfs-rg
root@solcluster03:~>


root@solcluster03:~> clresourcegroup online -M hanfs-rg

---------------------------------------------------------------


And that's it. The command never returns. the zpool is never imported, 
the resource themselves stay "Offline". After a reboot, the hasp 
resource is "Pending" and further modification of the configuration is 
impossible. No log entries btw, seems like

Solaris 10 Update 5, Solaris Cluster 3.2 Update 1, actual recommended 
patches cluster applied, using 2x V215 as iSCSI initiatiors and 1x X4500 
as the iSCSI target.