ldap auto_home error – Could not chdir to home directory /home/xxx: No such file or directory

October 10th, 2012

If you can log on the host but the home directory failed mouting with the following error message:

Could not chdir to home directory /home/xxx: No such file or directory

Then one method you can try is that:

  1. Ensure the home directory for your username exists on the exported NFS server
  2. Append /etc/auto_home on the host with text like the following:<username> <NFS server>:/export/home/&  #this assume the exported home directory is on /export/home, your environment may varies
  3. At last, ensure automount is running on the host and then try log on again. You should now able to mount your home directory.
Categories: IT Architecture, Linux, Systems Tags:

how to turn on hba flags connected to EMC arrays

October 3rd, 2012

As per EMC recommendation following flags should be enabled for Vmware ESX hosts, if not there will be performance issues:

Common_Serial_Number(C)
SCSI_3(SC3)
SPC2_Protocol_Version(SPC2)

Here's the commands that'll do the trick:

sudo symmask -sid <sid> set hba_flags on C,SPC2,SC3 -enable -wwn <port wwn> -dir <dir number> -p <port number>

Categories: Hardware, NAS, SAN, Storage Tags:

Resolved – Errors found during scanning of LUN allocated from IBM XIV array

October 2nd, 2012

So here's the story:
After the LUN(IBM XIV array) allocated, we run a 'xiv_fc_admin -R' to make the LUN visible to OS(testhost-db-clstr-vol_37 is the new LUN's Volume Name):
root@testhost01 # xiv_devlist -o device,vol_name,vol_id
XIV Devices
-------------------------------------------------------------------
Device Vol Name Vol Id
-------------------------------------------------------------------
/dev/dsk/c2t500173804EE40140d19s2 testhost-db-clstr-vol_37 1974
-------------------------------------------------------------------
/dev/dsk/c2t500173804EE40150d19s2 testhost-db-clstr-vol_37 1974
-------------------------------------------------------------------
/dev/dsk/c4t500173804EE40142d19s2 testhost-db-clstr-vol_37 1974
-------------------------------------------------------------------
/dev/dsk/c4t500173804EE40152d19s2 testhost-db-clstr-vol_37 1974
-------------------------------------------------------------------
...
...
...
/dev/vx/dmp/xiv0_16 testhost-db-clstr-vol_17 1922
...
...
...
Non-XIV Devices
--------------------
Device
--------------------
/dev/vx/dmp/disk_0
--------------------
/dev/vx/dmp/disk_1
--------------------
/dev/vx/dmp/disk_2
--------------------
/dev/vx/dmp/disk_3
--------------------

Then, I ran 'vxdctl enable' in order to make the DMP device visible to OS, but error message prompted:
root@testhost01 # vxdctl enable
VxVM vxdctl ERROR V-5-1-0 Data Corruption Protection Activated - User Corrective Action Needed
VxVM vxdctl INFO V-5-1-0 To recover, first ensure that the OS device tree is up to date (requires OS specific commands).
VxVM vxdctl INFO V-5-1-0 Then, execute 'vxdisk rm' on the following devices before reinitiating device discovery:
xiv0_18, xiv0_18, xiv0_18, xiv0_18

After this, the new LUN disappered from output of 'xiv_devlist -o device,vol_name,vol_id'(testhost-db-clstr-vol_37 disappered), and xiv0_18(the DMP device of new LUN) turned to 'Unreachable device', see below:

root@testhost01 # xiv_devlist -o device,vol_name,vol_id
XIV Devices
-----------------------------------------------------
Device Vol Name Vol Id
-----------------------------------------------------
...
...
...
Non-XIV Devices
--------------------
Device
--------------------
/dev/vx/dmp/disk_0
--------------------
/dev/vx/dmp/disk_1
--------------------
/dev/vx/dmp/disk_2
--------------------
/dev/vx/dmp/disk_3
--------------------
Unreachable devices: /dev/vx/dmp/xiv0_18
Also, 'vxdisk list' showed:
root@testhost01 # vxdisk list xiv0_18
Device: xiv0_18
devicetag: xiv0_18
type: auto
flags: error private autoconfig
pubpaths: block=/dev/vx/dmp/xiv0_18s2 char=/dev/vx/rdmp/xiv0_18s2
guid: -
udid: IBM%5F2810XIV%5F4EE4%5F07B6
site: -
Multipathing information:
numpaths: 4
c4t500173804EE40142d19s2 state=disabled
c4t500173804EE40152d19s2 state=disabled
c2t500173804EE40150d19s2 state=disabled
c2t500173804EE40140d19s2 state=disabled

I tried to format the new DMP device(xiv0_18), but failed with info below:
root@testhost01 # format -d /dev/vx/dmp/xiv0_18
Searching for disks...done

c2t500173804EE40140d19: configured with capacity of 48.06GB
c2t500173804EE40150d19: configured with capacity of 48.06GB
c4t500173804EE40142d19: configured with capacity of 48.06GB
c4t500173804EE40152d19: configured with capacity of 48.06GB
Unable to find specified disk '/dev/vx/dmp/xiv0_18'.

Also, 'vxdisksetup -i' failed with info below:
root@testhost01 # vxdisksetup -i /dev/vx/dmp/xiv0_18
prtvtoc: /dev/vx/rdmp/xiv0_18: No such device or address

And, 'xiv_fc_admin -R' failed with info below:
root@testhost01 # xiv_fc_admin -R
ERROR: Error during command execution: vxdctl enabled
====================================================
OK, that's all of the symptoms and the headache, here's the solution:
====================================================

1. Run 'xiv_fc_admin -R'(ERROR: Error during command execution: vxdctl enabled will prompt, ignore it. this step scanned for new LUN). You can also run a devfsadm -c disk(not needed actually)
2. Now exclude problematic paths of the DMP device(you can check the paths from vxdisk list xiv0_18)
root@testhost01 # vxdmpadm exclude vxvm path=c4t500173804EE40142d19s2
root@testhost01 # vxdmpadm exclude vxvm path=c4t500173804EE40152d19s2
root@testhost01 # vxdmpadm exclude vxvm path=c2t500173804EE40150d19s2
root@testhost01 # vxdmpadm exclude vxvm path=c2t500173804EE40140d19s2
3. Now run 'vxdctl enable', the following error message will NOT showed:
VxVM vxdctl ERROR V-5-1-0 Data Corruption Protection Activated - User Corrective Action Needed
VxVM vxdctl INFO V-5-1-0 To recover, first ensure that the OS device tree is up to date (requires OS specific commands).
VxVM vxdctl INFO V-5-1-0 Then, execute 'vxdisk rm' on the following devices before reinitiating device discovery:
xiv0_18, xiv0_18, xiv0_18, xiv0_18
4. Now include the problematic paths:
root@testhost01 # vxdmpadm include vxvm path=c4t500173804EE40142d19s2
root@testhost01 # vxdmpadm include vxvm path=c4t500173804EE40152d19s2
root@testhost01 # vxdmpadm include vxvm path=c2t500173804EE40150d19s2
root@testhost01 # vxdmpadm include vxvm path=c2t500173804EE40140d19s2

5. Run 'vxdctl enable'. After this, you should now see the DMP device in output of 'xiv_devlist -o device,vol_name,vol_id'
root@testhost01 # xiv_devlist -o device,vol_name,vol_id
XIV Devices
-----------------------------------------------------
Device Vol Name Vol Id
-----------------------------------------------------
...
...
...
-----------------------------------------------------
/dev/vx/dmp/xiv0_18 testhost-db-clstr-vol_37 1974
-----------------------------------------------------
...
...
...
Non-XIV Devices
--------------------
Device
--------------------
/dev/vx/dmp/disk_0
--------------------
/dev/vx/dmp/disk_1
--------------------
/dev/vx/dmp/disk_2
--------------------
/dev/vx/dmp/disk_3
--------------------

6. 'vxdisk list' will now show the DMP device(xiv0_18) as 'auto - - nolabel', obviously we should now label the DMP device:
root@testhost01 # format -d xiv0_18
Searching for disks...done

c2t500173804EE40140d19: configured with capacity of 48.06GB
c2t500173804EE40150d19: configured with capacity of 48.06GB
c4t500173804EE40142d19: configured with capacity of 48.06GB
c4t500173804EE40152d19: configured with capacity of 48.06GB
Unable to find specified disk 'xiv0_18'.

root@testhost01 # vxdisk list xiv0_18
Device: xiv0_18
devicetag: xiv0_18
type: auto
flags: nolabel private autoconfig
pubpaths: block=/dev/vx/dmp/xiv0_18 char=/dev/vx/rdmp/xiv0_18
guid: -
udid: IBM%5F2810XIV%5F4EE4%5F07B6
site: -
errno: Disk is not usable
Multipathing information:
numpaths: 4
c4t500173804EE40142d19s2 state=enabled
c4t500173804EE40152d19s2 state=enabled
c2t500173804EE40150d19s2 state=enabled
c2t500173804EE40140d19s2 state=enabled

root@testhost01 # vxdisksetup -i /dev/vx/dmp/xiv0_18
prtvtoc: /dev/vx/rdmp/xiv0_18: Unable to read Disk geometry errno = 0x16

Not again! But don't panic this time. Now run format for each subpath of the DMP device(can be found in output of vxdisk list xiv0_18), for example:
root@testhost01 # format c4t500173804EE40142d19s2

c4t500173804EE40142d19s2: configured with capacity of 48.06GB
selecting c4t500173804EE40142d19s2
[disk formatted]
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
!<cmd> - execute <cmd>, then return
quit
format> label
Ready to label disk, continue? yes

format> save
Saving new disk and partition definitions
Enter file name["./format.dat"]:
format> quit

7. After the subpaths were labelled, now run a 'vxdctl enable' again. After this, you'll find the DMP device turned it's state from 'auto - - nolabel' to 'auto:none - - online invalid', and vxdisk list no longer showed the DMP device as 'Disk is not usable':
root@testhost01 # vxdisk list xiv0_18
Device: xiv0_18
devicetag: xiv0_18
type: auto
info: format=none
flags: online ready private autoconfig invalid
pubpaths: block=/dev/vx/dmp/xiv0_18s2 char=/dev/vx/rdmp/xiv0_18s2
guid: -
udid: IBM%5F2810XIV%5F4EE4%5F07B6
site: -
Multipathing information:
numpaths: 4
c4t500173804EE40142d19s2 state=enabled
c4t500173804EE40152d19s2 state=enabled
c2t500173804EE40150d19s2 state=enabled
c2t500173804EE40140d19s2 state=enabled

8. To add the new DMP device to Disk Group, the following steps should be followed:
/usr/lib/vxvm/bin/vxdisksetup -i xiv0_18
vxdg -g <dg_name> adddisk <disk_name>=<device name>
/usr/sbin/vxassist -g <dg_name> maxgrow <vol name> alloc=<newly-add-luns>
/etc/vx/bin/vxresize -g <dg_name> -bx <vol name> <new size>

 

Categories: Hardware, SAN, Storage Tags:

change mysql proxy agent max-open-files

September 24th, 2012

Change Step details
==============
1. Stop both the mysql service and mysql proxy agent on the SLAVE mysql host  using the startup script command:
sudo /etc/init.d/mysql stop
2. Change the user context to mysql user with the command:
sudo su - mysql
3. Update the mysql proxy agent configuration file "/apps/mysql/agent/etc/mysql-monitor-agent.ini", add a new line like below at the end of the file:
max-open-files = 8192
4. Under ROOT user context, update the mysql stratup script file "/etc/init.d/mysql", replace all the existing content there with the content below:
#!/bin/ksh
# chkconfig: 345 90 02
# description: MySQL startup
/apps/mysql/install/support-files/mysql.server $*
/apps/mysql/agent/etc/init.d/mysql-monitor-agent $*

5. Under ROOT user context, execute the command:
chkconfig --add mysql
6. Start both the mysql service and mysql proxy agent on the SLAVE mysql host using the startup script command:
sudo /etc/init.d/mysql start
7. Repeat the step#1 to #7 on the other MASTER mysql host

Categories: Databases, IT Architecture, MySQL DB Tags:

lvm snapshot backup

September 20th, 2012

Here are the updated steps for LVM snapshot backup.

# determine available space.

sudo vgs | awk '/VolGroup/ {print $NF}'

# create snapshots, 1Gb should be enough but if you really struggle you can try even smaller cause it will only have to hold the delta between now and when you destroy them.

sudo lvcreate -L 1G -s -n rootLVsnap VolGroup00/rootLV

sudo lvcreate -L 1G -s -n varLVsnap VolGroup00/varLV

# create the FS for backups. allocate as much space as you have. last-resort: use NFS.

sudo lvcreate -n OSbackup -L 5G VolGroup00

sudo mkfs -t ext3 /dev/VolGroup00/OSbackup

sudo mkdir /OSbackup

sudo mount /dev/VolGroup00/OSbackup /OSbackup

# create a backup of root

# gzip is important. if you are really tight on space, try gzip -9 or even bzip2

sudo dd if=/dev/VolGroup00/rootLVsnap bs=1M | sudo sh -c 'gzip -c > /OSbackup/root.dd.gz'

# now, remove root snapshot and extend backup fs

sudo lvremove VolGroup00/rootLVsnap

sudo lvextend -L +1G VolGroup00/OSbackup

sudo resize2fs /dev/VolGroup00/OSbackup

# backup var

sudo dd if=/dev/VolGroup00/varLVsnap bs=1M | sudo sh -c 'gzip -c > /OSbackup/var.dd.gz'

sudo lvremove VolGroup00/varLVsnap

# backup boot

cd /boot; sudo tar -pczf /OSbackup/boot.tar.gz .

# unmount the fs and destroy mountpoint

sudo umount /OSbackup

sudo rmdir /OSbackup

PS: Here's more info http://www.howtoforge.com/linux_lvm_snapshots

Categories: Hardware, Storage Tags:

resolved – bnx2i dev eth0 does not support iscsi

September 19th, 2012

There's a weird incident occurred on a linux box. The linux box turned not responsible to ping or ssh, although from ifconfig and /proc/net/bonding/bond0 file, the system said it's running ok. After some google work, I found that the issue may related to the NIC driver. I tried bring down/bring up NICs one by one, but got error:

Bringing up loopback interface bond0: bnx2i: dev eth0 does not support iscsi

bnx2i: iSCSI not supported, dev=eth0

bonding: no command found in slaves file for bond bond0. Use +ifname or -ifname

At last, I tried restart the whole network i.e. /etc/init.d/network restart. And that did the trick, the networking was then running ok and can ping/ssh to it without problem.