Home > Hardware, NAS, Storage > zfs shared lun stoage set up for oracle RAC – using iscsiadm

zfs shared lun stoage set up for oracle RAC – using iscsiadm

January 4th, 2013
  • create iSCSI Target Group

Open zfs BUI, navigate through "Configuration" -> "SAN" -> "iSCSI Targets". Then create new iSCSI Target by clicking plus sign. Give it an alias, and then select the Network interface(may be bond or LACP) you want to use(check it from "Configuration" -> "Network" and "Configuration" -> "Cluster"). After creating this iSCSI target, drag the newly created target to the right side "iSCSI Target Groups" to create one iSCSI Target Group. You can give that iSCSI target group an name too. Note down the iSCSI Target Group's iqn, this is important for later operations.(Network interfaces:use NAS interface. You can select multiple interfaces)

  • create iSCSI Initiator Group

Before going on the next step, we need first get the iSCSI initiator IQN for each hosts we want LUN allocated. On each host, execute the following command to get the iqn for iscsi on linux platform(You can edit this file before read it, for example, make iqn name ended with` hostname` so it's easier for later operations on LUN<do a /etc/init.d/iscsi restart after your modification to initiatorname.iscsi>. You should first ensure package iscsi-initiator-utils is installed):

[root@test-host ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=<your host's iqn name>

Now go back to zfs BUI, navigate through "Configuration" -> "SAN" -> "Initiators". On the left side, click "iSCSI Initiators", then click plus sign on it. Enter IQN you get from previos step and give it an name.(do this for each host you want iSCSI LUN allocated). After this, drag the newly created iSCSI initiator(s) from left side to form new iSCSI Initiator Groups on the right side(drag two items from the left to the same item on the right to form an group).

  • create shared LUNs for iSCSI Initiator Group

After this, we need now create LUNs for iSCSI Initiator Group(so that shared lun can be allocated, for example, oracle RAC need shared storage). Click on diskette sign on the just created iSCSI Initiator Group,select the project you want the LUN allocated from, give it a name, and assign the volume size. Select the right target group you created before(you can also create a new one e.g. RAC in shares).

PS:You can also now go to "shares" -> "Luns" and create Lun(s) using the target group you created and use default Initiator group. Note that one LUN need one iSCSI target. So you should create more iSCSI targets and add them to iSCSI target group if you want more LUNs.

  • scan shared LUNs from hosts

Now we're going to operate on linux hosts. On each host you want iSCSI LUN allocated, do the following steps:

iscsiadm -m discovery -t st -p <ip address of your zfs storage>(use cluster's ip if there's zfs cluster) #Discover available targets from a discovery portal
iscsiadm -m node -T <variable, iSCSI Target Group iqn> -p <ip address of your zfs storage> -l #Log into a specific target. Or use output from above command($1 is --portal, $3 is --targetname, -l is --login). Use -u to log out for a specified record.

If there's error:

iscsiadm: default: 1 session requested, but 1 already present.
iscsiadm: Could not log into all portals


#iscsiadm -m session --rescan

#iscsiadm -m node --loginall=all

#iscsiadm -m discovery -t sendtargets -p <ZFS IP>

#iscsiadm -m node --loginall=all

iscsiadm -m session --rescan

#iscsiadm -m node --loginall=all

After these steps, you host(s) should now see the newly allocated iSCSI LUN(s), you can run fdisk -l to confirm.


  • To remove discovered targets on one specific portal(or ZFS appliance):

iscsiadm -m discovery -t st -p

ls -l /var/lib/iscsi/nodes/

iscsiadm -m node -T <iscsi target group iqn> -p -u

iscsiadm -m discovery -o show

iscsiadm -m node -o delete -p

ls -l /var/lib/iscsi/nodes/

  • Here's more about iscsiadm:

iscsiadm -m session -P 1 #Display list of all current sessions logged in, -P for level(1 to 3). From here, we can know which target are the local disks mapped to(e.g. /dev/sde is mapped to target iqn.1986-03.com.sun:02:f82d223a-ca68-e9ac-87d5-95b079b2ba05, then we can know the NAS ZFS appliance name. On ZFS appliance, after got the target group name, we can check which LUNs use that target group, thus we know the mapping between local iscsi disk and ZFS LUN)

iscsiadm -m session --rescan #rescan all sessions

iscsiadm -m session -r SID --rescan #rescan a specific session #SID can be got from iscsiadm -m session -P 1

iscsiadm -m node -T targetname -p ipaddress -u #Log out of a specific target

If you want to remove specific iSCSI LUN from system, then do the following:

cd /sys/class/iscsi_session/session<SID>/device/target<scsi host number>:0:0/<scsi host number>:0:0:<Lun number>
echo 1 > delete

iscsiadm -m node -T targetname -p ipaddress #Display information about a target
iscsiadm -m node -s -T targetname -p ipaddress #Display statistics about a target

iscsiadm -m discovery -o show #View iSCSI database regarding discovery
iscsiadm -m node -o show #View iSCSI database regarding targets to log into
iscsiadm -m session -o show #View iSCSI database regarding sessions logged into
multipath -ll #View if the targets are multipathed (MPIO)

You can find more information about iscsi disk in /sys/class/{scsi_device, scsi_disk, scsi_generic,scsi_host} and /sys/block/ after get the info from iscsiadm -m session -P 3.

Here's the CMD equivalent of this article(aksh of oracle ZFS appliance):

shares project rac186187
set mountpoint=/export/rac186187
set quota=1T
set readonly=true
set default_permissions=777

set default_user=root

set default_group=root
set sharenfs="sec=sys,ro=test.example.com,rw=@,root=@"
#get reservation
#get pool
#get snapdir #snapdir = visible
#get default_group #default_group = root
#get default_user #default_user = root
#get exported #exported = true

configuration net interfaces list
configuration net datalinks list

configuration san iscsi targets create rac186187
set alias=rac186187
set interfaces=aggr93001

configuration san iscsi targets list
configuration san iscsi targets groups create rac186187
set name=rac186187
set targets=iqn.1986-03.com.sun:02:0a3dec0f-b830-c17a-c957-dd2dc1755a16

[cat /etc/iscsi/initiatorname.iscsi] -> iqn.1988-12.com.oracle.testhost186, iqn.1988-12.com.oracle.testhost187
configuration san iscsi initiators create testhost186
set alias=testhost186
set initiator=iqn.1988-12.com.oracle.testhost186

configuration san iscsi initiators create testhost187
set alias=testhost187
set initiator=iqn.1988-12.com.oracle.testhost187

configuration san iscsi initiators list
configuration san iscsi initiators groups create rac186187
set name=rac186187
set initiators=iqn.1988-12.com.oracle.testhost186,iqn.1988-12.com.oracle.testhost187

shares select rac186187 #project must set readonly=false
lun rac186187
set volsize=500G
set targetgroup=rac186187
set initiatorgroup=rac186187

And if you want to create one share, here's the way:

shares select <project name> #to show current properties, run "show"
filesystem <new share name>
set quota=100G
set quota_snap=false
set reservation=50G
set reservation_snap=false
set root_permissions=777
set root_user=root
set root_group=root
cd /

To search for logs, you can try below:

ssh root@testzfs|tee log.txt

testzfs:> maintenance logs select alert list -a
testzfs:> maintenance logs select audit list -a
testzfs:> maintenance logs select fltlog list -a
testzfs:> maintenance logs select scrk list -a
testzfs:> maintenance logs select system list -a

Then grep <entry> log.txt


Suppose we know that there is one session logged in to target iqn.1986-03.com.sun:02:a7a92623-dfc2-eab0-cf84-be0b549dc5c9, then how can we add LUNs or change the size of existed LUN on ZFS?

First, to change the size of existing LUN:

[root@testhost1 ~]# iscsiadm -m session
tcp: [2],2 iqn.1986-03.com.sun:02:a7a92623-dfc2-eab0-cf84-be0b549dc5c9

Log on ZFS UI, go to Configuration, SAN, iSCSI, Targets, search for "iqn.1986-03.com.sun:02:a7a92623-dfc2-eab0-cf84-be0b549dc5c9" which is the target name, then you'll find the target and target group it belongs to. Note down the target group name, e.g. RAC01, then go to Shares, LUNs, click on the LUN and change the size as needed.

To add new LUN to the host:

First, find the iscsi initiator name on this host testhost1-p:

[root@testhost1 ~]# cat /etc/iscsi/initiatorname.iscsi

Log on ZFS UI, go to Configuration, SAN, iSCSI, Initiators, search "iqn.1988-12.com.oracle.testhost1", you'll find the initiator and initiator group. From here you can click the diskette icon and add new LUN. Make sure to select the right target group you got from previous step.

Good Luck!

Categories: Hardware, NAS, Storage Tags:
Comments are closed.