Archive

Archive for September, 2010

ip ping in bulk batch shell script

September 30th, 2010 2 comments

Imagine,you've got a task now to verify a bulk of domains the speed of response and  whether they are active or not.Here a bulk of,means,for example,1,000.On getting this task,what will you do next?Type in every ip in the command line in Microsoft's cmd and waits for the result?1,000,you must consider it.You may imagine,how tired after you've pinged the full 1,000 of the urls.

Here is my implement of the task.Firstly,put the urls in domain.txt,one url per line.Then run the script:sh if.sh.Lastly,waits for the result in ip.txt.During the execution time,you may go out and breath fresh air or have a cup of tea,not type in the Microsoft cmd any more!

Now,have a try by yourself!

if.sh:
#!/bin/bash
#copyright:doxer.org
rm -rf ./ip.txt
for i in `cat ./domain.txt`
do
echo -e 'BEGIN\n\n' >>ip.txt
ping $i -c 3 >>ip.txt
echo -e '\n\nEND\n' >>ip.txt
done

Put your list of domains in the file:
xx1.com
xx2.com
xx3.com
xx4.com

Now,please use cat ip.txt to see the joyful result!

PS:

I have known that the script in this article ping host one by one for 3 years(now is 2013). So I've written an upgraded rpm package called pping_doxer, which pings hosts in parallel and proves to be very efficient. You can read more about it on http://www.doxer.org/projects-doxer/.

reverse engineering:get your compiled parameters of apache,php,mysql,nginx

September 18th, 2010 No comments

Sometimes,we want to see the compiled parameters of nginx,apache,mysql,php etc. after the compilation.Consider the following method.
nginx compilation parameters:
#/usr/local/nginx/sbin/nginx -V
nginx version: nginx/0.6.32 built by gcc 4.1.2 20071124 (Red Hat 4.1.2-42) configure arguments: –user=www –group=www –prefix=/usr/local/nginx/ –with-http_stub_status_module –with-openssl=/usr/local/openssl

apache compilation parameters:
# cat /usr/local/apache2/build/config.nice
#! /bin/sh # # Created by configure “./configure” \ “–prefix=/usr/local/apache2″ \ “–with-included-apr” \ “–enable-so” \ “–enable-deflate=shared” \ “–enable-expires=shared” \ “–enable-rewrite=shared” \ “–enable-static-support” \ “–disable-userdir” \ “$@”

php compilation parameters:
# /usr/local/php/bin/php -i |grep configure
Configure Command => ‘./configure’ ‘–prefix=/usr/local/php’ ‘–with-apxs2=/usr/local/apache2/bin/apxs’ ‘–with-config-file-path=/usr/local/php/etc’ ‘–with-mysql=/usr/local/mysql’ ‘–with-libxml-dir=/usr/local/libxml2/bin’ ‘–with-gd=/usr/local/gd2′ ‘–with-jpeg-dir’ ‘–with-png-dir’ ‘–with-bz2′ ‘–with-xmlrpc’ ‘–with-freetype-dir’ ‘–with-zlib-dir’

mysql compilation parameters:
# cat “/usr/local/mysql/bin/mysqlbug”|grep configure
# This is set by configure CONFIGURE_LINE=”./configure ‘–prefix=/usr/local/mysql’ ‘–localstatedir=/var/lib/mysql’ ‘–with-comment=Source’ ‘–with-server-suffix=-H863′ ‘–with-mysqld-user=mysql’ ‘–without-debug’ ‘–with-big-tables’ ‘–with-charset=gbk’ ‘–with-collation=gbk_chinese_ci’ ‘–with-extra-charsets=all’ ‘–with-pthread’ ‘–enable-static’ ‘–enable-thread-safe-client’ ‘–with-client-ldflags=-all-static’ ‘–with-mysqld-ldflags=-all-static’ ‘–enable-assembler’ ‘–without-isam’ ‘–without-innodb’ ‘–without-ndb-debug’”

Install vhcs2 panel under debian(shell scripts)

September 15th, 2010 No comments

vhcs2_debian.sh,chmod +x ./vhcs2_debian.sh,./vhcs2_debian.sh.In the process,you must type in the main ip of your server.And,set the parameters during the installation(for example,mysql password etc.)

Here it goes:
#!/bin/bash
#by doxer.org
read -p 'enter the main ip of the server:' mainip #main ip of the server
read -p 'input your hostname:' your_hostname
echo -e "127.0.0.1 $your_hostname $your_hostname\n$mainip $your_hostname $your_hostname" >/etc/hosts
echo $your_hostname >/etc/hostname
/etc/init.d/hostname.sh start
echo -e "deb http://ftp.debian.org/debian/ lenny main contrib non-free\ndeb http://security.debian.org/ lenny/updates main contrib non-free" >>/etc/apt/sources.list
apt-get update
apt-get -y install gcc
apt-get install ssh openssh-server
apt-get install tar bzip2 wget lsb-release
wget http://server5.moll-newmedia.de/vhcs-2.4.8.tar.bz2
tar -xjvf vhcs-2.4.8.tar.bz2
echo -e "gcc
apache2
apache2.2-common
apache2-mpm-prefork
bind9
bzip2
courier-authdaemon
courier-base
courier-imap
courier-maildrop
courier-pop
diff
dnsutils
gzip
iptables
libapache2-mod-php5
libberkeleydb-perl
libc6-dev
libcrypt-blowfish-perl
libcrypt-cbc-perl
libcrypt-passwdmd5-perl
libdate-calc-perl
libdate-manip-perl
libdbd-mysql-perl
libdbi-perl
libio-stringy-perl
libmail-sendmail-perl
libmailtools-perl
libmcrypt4
libmd5-perl
libmime-perl
libnet-dns-perl
libnet-netmask-perl
libnet-smtp-server-perl
libperl5.10
libsasl2-2
libsasl2-modules
libsnmp-session-perl
libterm-readkey-perl
libterm-readpassword-perl
libtimedate-perl
make
mysql-client
mysql-common
mysql-server
original-awk
patch
perl
perl-base
perl-modules
php5
php5-gd
php5-mcrypt
php5-mysql
php-pear
postfix
postfix-tls
procmail
proftpd
sasl2-bin
ssh
tar
wget" >/root/debian5-packages.txt

apt-get -y install `cat /root/debian5-packages.txt` #some options:Internet Site;standalone;
sed -i 's/START=no/START=yes/' /etc/default/saslauthd
cd /root/vhcs-2.4.8
make -f Makefile install
cp -R /tmp/vhcs-2.4.8/* /
/var/www/vhcs2/engine/setup/vhcs2-setup
sed -i 's/#Include \/etc\/proftpd\/modules.conf/Include \/etc\/proftpd\/modules.conf/' /etc/proftpd/proftpd.conf
sed -i 's/DisplayFirstChdir/DisplayChdir/' /etc/proftpd/proftpd.conf
sed -i 's/#LoadModule mod_sql.c/LoadModule mod_sql.c/' /etc/proftpd/modules.conf
sed -i 's/#LoadModule mod_sql_mysql.c/LoadModule mod_sql_mysql.c/' /etc/proftpd/modules.conf
apt-get install php5-curl
sed -i 's/display_errors = On/display_errors = Off/' /etc/php5/apache2/php.ini
cp /etc/apache2/mods-available/rewrite.load /etc/apache2/mods-enabled/
cp /etc/apache2/mods-available/ssl.* /etc/apache2/mods-enabled/
sed -i 's/^AllowOverride.*$/AllowOverride All/' /etc/vhcs2/apache/parts/dmn_entry.tpl
sed -i 's/^MaxClients.*$/MaxClients 100/' /etc/apache2/apache2.conf
/etc/init.d/proftpd start
/etc/init.d/apache2 restart
#set password for root:set password for 'root'@'localhost' = password('yourpassword');
echo "Please visit http://$mainip/vhcs2 to see whether it's working now.Bye."

How to get the real user ip address when using nginx as reverse proxy(from backend apache server)

September 14th, 2010 2 comments

To put an elephant into a fridge-three steps(which I think you already knew it):

Firstly(open the fridge):In nginx,add a line in /usr/local/nginx/conf/proxy.conf:
proxy_set_header        X-Real-IP       $remote_addr;
#Note:Add this just in proxy.conf,repetitive addition in nginx.conf location and server place will result your backend server log repetitive times.

Secondly(put the elephant in):In backend apache server,add the following lines in place VirtualHost or Server:
LogFormat "%{User-agent}i %{X-Real-IP}i"
TransferLog /var/log/test.log
Thirdly(close the fridge):reboot nginx and backend to take your changes effect.
killall nginx && /usr/local/nginx/sbin/nginx

Y/etc/init.d/apache2 restart #or /etc/init.d/httpd restart
All right.Now,use tail -f /var/log/apache2/yoursite-access.log to see the real user ip address not filtered by your reverse nginx server.

Install nginx reverse proxy server under centos(bash shell)

September 13th, 2010 2 comments

This script by me accomplish the auto installation of nginx reverse server under centos.Please modify upstream and server_name scope in nginx.conf after the installation.And you do not need to modify proxy.conf(Of course,you can.For example,to get the real user ip in backend apache server).

Here the script goes:
#!/bin/bash
#nginx_reverse_proxy_centos.sh
#by doxer.org
os="";
grep "centos" /etc/issue -i -q
if [ $? = '0' ];then
os='centos'
fi

grep "debian" /etc/issue -i -q
if [ $? = '0' ];then
os='debian'
fi

if [ $os = "" ];then
echo "not a valid system os"
exit 1
fi
#set variables begin
downloads="/root/downloads"

#set variables end

#centos begin
rpm -qa|grep httpd #under centos it's httpd,debian apache
if [ $? = '0' ];then
aa="`rpm -qa|grep httpd`"
rpm -e $aa
fi

yum -y install gcc gcc-c++ zlib-devel openssl-devel #expect Is this ok =>y
if [ ! -d $downloads ];then
mkdir $downloads
fi
cd $downloads #pwd /root/downloads
#install pcre begin
wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-7.9.tar.gz
tar zxvf pcre-7.9.tar.gz
cd pcre-7.9/ #pwd /root/downloads/pcre-7.9
./configure
make && make install
#install pcre end
cd .. #pwd /root/downloads
#install nginx begin
wget http://nginx.org/download/nginx-0.7.63.tar.gz
tar zxvf nginx-0.7.63.tar.gz
mkdir -p /usr/local/nginx
cp -R ./nginx-0.7.63/* /usr/local/nginx
cd /usr/local/nginx #pwd /usr/local/nginx
./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module
make && make install
mkdir -p /usr/local/nginx/logs
chmod +w /usr/local/nginx/logs
groupadd www
useradd -g www www
chown -R www:www /usr/local/nginx/logs
touch  /usr/local/nginx/conf/proxy.conf
echo "proxy_redirect off;
proxy_set_header Host \$host;
proxy_set_header Port \$proxy_port;
proxy_set_header XHost \$host:\$proxy_port;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header REMOTE_ADDR \$remote_addr;
proxy_set_header HTTP_CLIENT_IP \$remote_addr;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
#client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 30;
proxy_send_timeout 30;
proxy_read_timeout 30;
proxy_buffers 32 4k;
proxy_buffer_size 16k;
expires 60;
client_body_timeout 60;
client_header_timeout 60;">/usr/local/nginx/conf/proxy.conf
rm -f /usr/local/nginx/conf/nginx.conf
touch /usr/local/nginx/conf/nginx.conf
echo -e 'user www www;
worker_processes 4;
error_log /usr/local/nginx/logs/error.log crit;
pid /usr/local/nginx/nginx.pid;

events {
use epoll;
worker_connections 1024;
}

http {
include mime.types;
include proxy.conf;
default_type application/octet-stream;
charset utf-8;
sendfile on;
tcp_nopush on;
keepalive_timeout 30;
gzip on;
gzip_min_length 1k;
#gzip_buffers 4 1
gzip_http_version 1.0;
gzip_comp_level 2;
gzip_types text/plain application/x-javascript text/css application/xml;
gzip_vary on;

upstream localhost{
server 127.0.0.1:81;
}

server{
listen 80;
server_name _;
location / {
proxy_pass http://localhost;
}
}

server {
listen 80;
server_name _;
location / {
if ($host !~* www\.(.*)) {
set $host_without_www $1;
set $xhost www.$host;
rewrite ^(.*)$ http://$xhost$host_without_www$1 permanent;
}
proxy_pass http://localhost;
}

location /ns {
stub_status on;
}
location ~ /\.ht {
deny all;
}
}
}'>/usr/local/nginx/conf/nginx.conf
#install nginx end

/usr/local/nginx/sbin/nginx #start nginx
ps aux|grep nginx|grep grep #check if nginx running now
if [ $? = '0' ];then
echo "nginx is running now"
else
echo "not running"
fi
#centos end
Ok.That's all.Now start up the nginx server:/usr/local/nginx/sbin/nginx.Change the listening port on apache to 81(NameVirtualHost *:81 and <VirtualHost *:81>).Visit your site and now take a look at the response header sent by your nginx reverse server.You will find Server:nginx/0.7.30 in your firebug's Net label.Enjoy the high concurrent processing ability of nginx now!

Related:Install nginx reverse proxy server under debian(bash shell)

Install nginx reverse proxy server under debian(bash shell)

September 13th, 2010 Comments off

This script by me accomplish the auto installation of nginx reverse server under debian.Please modify upstream and server_name scope in nginx.conf after the installation.And you do not need to modify proxy.conf(Of course,you can.For example,to <a href="http://www.doxer.org/nginx-real-userip/" target="_blank"></a>

Here the script goes:
#!/bin/bash
#nginx_debian.sh
#by doxer.org
#note:you should killall nginx and change the port of apache2 to 81 and restart apache2 after running the script
echo -e "deb http://ftp.debian.org/debian/ lenny main contrib non-free\ndeb http://security.debian.org/ lenny/updates main contrib non-free">>/etc/apt/sources.list
apt-get update
apt-get install gcc g++ make patch libpcre3 libpcre3-dev libpcrecpp0 libssl-dev zlib1g-dev
cd ~
wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-7.9.tar.gz
tar zxvf pcre-7.9.tar.gz
cd pcre-7.9/ #pwd /root/downloads/pcre-7.9
./configure
make && make install
cd ..
wget http://sysoev.ru/nginx/nginx-0.7.30.tar.gz
tar -zxvf nginx-0.7.67.tar.gz
cd nginx-0.7.30
./configure --sbin-path=/usr/local/nginx/sbin --with-http_ssl_module  --with-http_stub_status_module
make && make install
#rm -rf /etc/nginx/sites-enabled/default
groupadd www
useradd -g www www
echo -e 'user www www;
worker_processes 4;
error_log /usr/local/nginx/logs/error.log crit;
pid /usr/local/nginx/nginx.pid;

events {
use epoll;
worker_connections 1024;
}

http {
include mime.types;
include proxy.conf;
default_type application/octet-stream;
charset utf-8;
sendfile on;
tcp_nopush on;
keepalive_timeout 30;
gzip on;
gzip_min_length 1k;
#gzip_buffers 4 1
gzip_http_version 1.0;
gzip_comp_level 2;
gzip_types text/plain application/x-javascript text/css application/xml;
gzip_vary on;

upstream localhost{
server 127.0.0.1:81;
}

server{
listen 80;
server_name _;
location / {
proxy_pass http://localhost;
}
}

}'>/usr/local/nginx/conf/nginx.conf

echo -E 'proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header    Port        $proxy_port;
proxy_set_header    XHost        $host:$proxy_port;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        REMOTE_ADDR     $remote_addr;
proxy_set_header        HTTP_CLIENT_IP  $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
#client_max_body_size    10m;
client_body_buffer_size 128k;
proxy_connect_timeout   30;
proxy_send_timeout      30;
proxy_read_timeout      30;
proxy_buffers           32 4k;
proxy_buffer_size       16k;'>/usr/local/nginx/conf/proxy.conf
Ok.That’s all.Now bootstrap from start the nginx server:/usr/local/nginx/sbin/nginx.Change the listening port on apache to 81(NameVirtualHost *:81 and <VirtualHost *:81>).Visit your site and now take a look at the response header sent by your nginx reverse server.You will find Server:nginx/0.7.30 in your firebug’s Net label.Enjoy the high concurrent processing ability of nginx now!

Related:Install nginx reverse proxy server under centos(bash shell)

ban specific country’s ip address from visiting sites

September 11th, 2010 No comments

To drop ip addresses that has 20 and more concurrent requests:
#!/bin/bash
banips=`netstat -an| grep :80 | grep -v 127.0.0.1 |grep -v 0.0.0.0 |awk '{ print $5 }' | sort|awk -F: '{print $1}' | uniq -c | awk '$1 >20 {print $2}'`
for ip in $banips
do
iptables -A INPUT -s $ip -j DROP
done
Ban specific country's ip address:
#!/bin/bash
# Block traffic from a specific country
# written by vpsee.com
#modified by doxer.org

#COUNTRY = "cn.zone"
bandir="/root/banips" #put .zone files under /root/banips/,later comes the .zone files link
[ ! -d $bandir] && mkdir -p $bandir && echo "put .zone file in $bandir" && exit 0
IPTABLES = /sbin/iptables

if [ "$(id -u)" != "0" ]; then
echo "you must be root" 1>&2
exit 1
fi

resetrules() {
$IPTABLES -F
$IPTABLES -t nat -F
$IPTABLES -t mangle -F
$IPTABLES -X
}

resetrules

zonelists=`ls /root/banips/`
for COUNTRY in $zonelists
do
IPS=`cat $COUNTRY`
for ip in $IPS
do
echo "blocking $ip"
$IPTABLES -A INPUT -s $ip -j DROP
done
done
exit 0
Country-region ip ranges:http://www.ipdeny.com/ipblocks/

using rsync and crontab to automatically backup sites

September 3rd, 2010 1 comment

What does rsync mean?Here the man page goes:rsync - faster, flexible replacement for rcp.rcp is a program that executes remote copying,and rsync is the upgrade version of it.Besides,rsync has its advantages on security and high efficiency matters.

Now,we want to backup sites(database files included) at a fixed period.You may think of ftp method,but considering the large scaling of sites source files(Bandwidth is so expensive!) and the insecurity of ftp protocal,the idea is deprecated.(About this method,link here to see it).

Note:

1.If no rsync installed on the source server & destination server,please install it first.

2.If no expect installed on the source server,install it first too.
#!/bin/bash
#rsync_sites.sh
#rsync sites,please crontab -e to set the circle time
sites_dirs=" /var/www/virtual/yourdomain.tld/htdocs /var/www/virtual/yourdomain2.tld/htdocs" #directorys to be backuped
sshhost='xxx.xxx.xxx.xxx' #destination server ip address
password='xxxxxx' #destination password
#using expect to auto-fill the interactive process(programmed dialogue with interactive program)
/usr/bin/expect <<EOF

spawn ssh -lroot -p22 $sshhost { [ ! -d /root/backup ] && mkdir /root/backup }
#mkdir of the destination dir to save backup files
expect "password:" #auto response
send "$password\r"
expect eof
exit
EOF
#sites_dirs="/root/"
if [ -x /usr/bin/expect ]
then
for site in $sites_dirs #loop the $site_dirs
do
site_name=`awk -F '/' '{print $5}' <<<$site` #notice:in '/var/www/virtual/yourdomain.tld/htdocs',$site_name will be yourdomain.tld
/usr/bin/expect <<EOF
spawn rsync --delete -azvv -e ssh $site root@$sshhost:/root/backup/$site_name
expect -re "password"
send "$password\r"
expect -re "Are you sure"
send "yes\r"
expect -re "please type"
send "yes\r"
expect eof
exit
EOF
done
else
echo 'please first install expect'
fi

Categories: IT Architecture, Linux, Systems Tags: