Archive

Archive for the ‘Programming’ Category

perl tips

April 2nd, 2014 No comments
#!/usr/bin/perl
use strict;
use warnings;##arrays
my @animals = (“dog”, “pig”, “cat”);
print “The last element of array \$animals is : “.$animals[$#animals].”\n”;
if(@animals>2){
print “more than 2 animals found\n”;
}
else{
print “less than 2 animals found\n”
}
foreach(@animals){
print $_.”\n”;
}
##hashes
my %fruit_color=(“apple”, “red”, “banana”, “yellow”);
print “Color of banana is : “.$fruit_color{“banana”}.”\n”;

for $char (keys %fruit_color)
{
print(“$char => $fruit_color{$char}\n”);
}

##references
my $variables = {
scalar  =>  {
description => “single item”,
sigil => ‘$’,
},
array   =>  {
description => “ordered list of items”,
sigil => ‘@’,
},
hash    =>  {
description => “key/value pairs”,
sigil => ‘%’,
},
};
print “Scalars begin with a $variables->{‘scalar’}->{‘sigil’}\n”;

##Files and I/O
##regular expressions
open (my $passwd, “<”, “/etc/passwd2″) or die (“can  a not open”);
while (<$passwd>) {
print $_ if $_ =~ “test”;
}
close $passwd or die “$passwd: $!”;
my $next = “doing a first”;
$next =~ s/first/second/;
print $next.”\n”;

my $email = “testaccount\@doxer.org”;
if ($email =~ /([^@]+)@(.+)/) {
print “Username is : $1\n”;
print “Hostname is : $2\n”;
}

##subroutines
sub multiply{
my ($num1, $num2) = @_;
my $result = $num1 * $num2;
return $result;
}

my $result2 = multiply(3, 5);
print “3 * 5 = $result2\n”;

! system(‘date’) or die(“failed it”); #if a subroutine returns ok, it’ll return 0
PS:
Categories: Perl, Programming, tips Tags:

resolved – sudo: sorry, you must have a tty to run sudo

April 1st, 2014 2 comments

The error message below sometimes will occur when you run a sudo <command>:

sudo: sorry, you must have a tty to run sudo

To resolve this, you may comment out “Defaults requiretty” in /etc/sudoers(revoked by running visudo). Here is more info about this method: http://www.cyberciti.biz/faq/linux-unix-bsd-sudo-sorry-you-must-haveattytorun/

However, sometimes it’s not convenient or even not possible to modify /etc/sudoers, then you can consider the following:

echo -e “<password>\n”|sudo -S <sudo command>

For -S parameter of sudo, you may refer to sudo man page:

-S‘ The -S (stdin) option causes sudo to read the password from the standard input instead of the terminal device. The password must be followed by a newline character.

So here -S bypass tty(terminal device) to read the password from the standard input. And by this, we can now pipe password to sudo.

Categories: Linux, Programming, SHELL, Systems Tags: ,

Resolved – print() on closed filehandle $fh at ./perl.pl line 6.

March 19th, 2014 No comments

You may find that print sometimes won’t work as expected in perl, for example:

[root@centos-doxer test]# cat perl.pl
#!/usr/bin/perl
use warnings;
open($fh,”test.txt”);
select $fh;
close $fh;
print “test”;

You may expect “test” to be printed, but actually you got error message:

print() on closed filehandle $fh at ./perl.pl line 6.

So how’s this happened? Please see my explanation:

[root@centos-doxer test]# cat perl.pl
#!/usr/bin/perl
use warnings;
open($fh,”test.txt”);
select $fh;
close $fh; #here you closed $fh filehandle, but you should now reset filehandle to STDOUT
print “test”;

Now here’s the updated script:

#!/usr/bin/perl
use warnings;
open($fh,”test.txt”);
select $fh;
close $fh;
select STDOUT;
print “test”;

This way, you’ll get “test” as expected!

 

Categories: Perl, Programming Tags:

Oracle BI Publisher reports – send mail when filesystems getting full

March 17th, 2014 No comments

Let’s assume you have one Oracle BI Publisher report for filesystem checking. And now you want to write script for checking that report page and send mail to system admins when filesystems are getting full. As the default output of Oracle BI Publisher report needs javascript to work, and as you may know javascript is evil that wget/curl can not get them, so after log on, the next step you need to do is to find the html version’s url of that report for you to use in your script(and the html page has all records when javascript one has only part of them):

BI_report_login

BI_export_html

 

Let’s assume that the html’s url is “http://www.example.com:9703/report.html”, and the display of it was like the following:

bi report

Then here goes the script that will check this page for hosts that has less than 10% available space and send mail to system admins:

#!/usr/bin/perl
use HTML::Strip;
system(“rm -f spacereport.html”);
system(“wget -q –no-proxy –no-check-certificate –post-data ‘id=admin&passwd=password’ ‘http://www.example.com:9703/report.html’ -O spacereport.html”);
open($fh,”spacereport.html”);

#or just @spacereport=<$fh>;
foreach(<$fh>){
push(@spacereport,$_);
}

#change array to hash
$index=0;
map {$pos{$index++}=$_} @spacereport;

#get location of <table> and </table>
#sort numerically ascending
for $char (sort {$a<=>$b} (keys %pos))
{
if($pos{$char} =~ /<table class=”c27″>/)
{
$table_start=$char;
}

if($pos{$char} =~ /<\/table>/)
{
$table_end=$char;
}

}

#get contents between <table> and </table>
for($i=$table_start;$i<=$table_end;$i++){
push(@table_array,$spacereport[$i]);
}
$table_htmlstr=join(“”,@table_array);

#get clear text between <table> and </table>
my $hs=HTML::Strip->new();
my $clean_text = $hs->parse($table_htmlstr);
$hs->eof;

@array_filtered=split(“\n”,$clean_text);

#remove empty array element
@array_filtered=grep { !/^\s+$/ } @array_filtered;
system(“rm -f space_mail_warning.txt”);
open($fh_mail_warning,”>”,”space_mail_warning.txt”);
select $fh_mail_warning;
for($j=4;$j<=$#array_filtered;$j=$j+4){
#put lines that has free space lower than 10% to space_mail_warning.txt
if($array_filtered[$j+2] <= 10){
print “Host: “.$array_filtered[$j].”\n”;
print “Part: “.$array_filtered[$j+1].”\n”;
print “Free(%): “.$array_filtered[$j+2].”\n”;
print “Free(GB): “.$array_filtered[$j+3].”\n”;
print “============\n\n”;
}
}
close $fh_mail_warning;

system(“rm -f space_mail_info.txt”);
open($fh_mail_info,”>”,”space_mail_info.txt”);
select $fh_mail_info;
for($j=4;$j<=$#array_filtered;$j=$j+4){
#put lines that has free space lower than 15% to space_mail_info.txt
if($array_filtered[$j+2] <= 15){
print “Host: “.$array_filtered[$j].”\n”;
print “Part: “.$array_filtered[$j+1].”\n”;
print “Free(%): “.$array_filtered[$j+2].”\n”;
print “Free(GB): “.$array_filtered[$j+3].”\n”;
print “============\n\n”;
}
}
close $fh_mail_info;

#send mail
#select STDOUT;
if(-s “space_mail_warning.txt”){
system(‘cat space_mail_warning.txt | /bin/mailx -s “Space Warning – please work with component owners to free space” [email protected]’);
} elsif(-s “space_mail_info.txt”){
system(‘cat space_mail_info.txt | /bin/mailx -s “Space Info – Space checking mail” [email protected]’);
}

Categories: Perl, Programming Tags:

wget and curl tips

March 14th, 2014 No comments

Imagine you want to download all files under http://www.example.com/2013/downloads, and not files under http://www.example.com/2013 except for directory ‘downloads’, then you can do this:

wget -r –level 100 -nd –no-proxy –no-parent –reject “index.htm*” –reject “*gif” ‘http://www.example.com/2013/downloads/’ #–level 100 is large enough, as I’ve seen no site has more than 100 levels of sub-directories so far.

wget -p -k –no-proxy –no-check-certificate –post-data ‘id=username&passwd=password’ <url> -O output.html

wget –no-proxy –no-check-certificate –save-cookies cookies.txt <url>

wget –no-proxy –no-check-certificate –load-cookies cookies.txt <url>

curl -k -u ‘username:password’ <url>

curl -k -L -d id=username -d passwd=password <url>

curl –data “loginform:id=username&loginform:passwd=password” -k -L <url>

 

Categories: Linux, Programming, SHELL Tags:

Oracle VM operations – poweron, poweroff, status, stat -r

January 27th, 2014 No comments

Here’s the script:

#!/usr/bin/perl
#1.OVM must be running before operations
#2.run ovm_vm_operation.pl status before running ovm_vm_operation.pl poweroff or poweron
use Net::SSH::Perl;
$host = $ARGV[0];
$operation = $ARGV[1];
$user = ‘root’;
$password = ‘password’;

if($host eq “help”) {
print “$0 OVM-name status|poweron|poweroff|stat-r\n”;
exit;
}

$ssh = Net::SSH::Perl->new($host);
$ssh->login($user,$password);

if($operation eq “status”) {
($stdout,$stderr,$exit) = $ssh->cmd(“ovm -uadmin -pwelcome1 vm ls|grep -v VM_test”);
open($host_fd,’>’,”/var/tmp/${host}.status”);
select $host_fd;
print $stdout;
close $host_fd;
} elsif($operation eq “poweroff”) {
open($poweroff_fd,’<’,”/var/tmp/${host}.status”);
foreach(<$poweroff_fd>){
if($_ =~ “Server_Pool|OVM|Powered”) {
next;
}
if($_ =~ /(.*?)\s+([0-9]{1,})\s+([0-9]{1,})\s+([0-9]{1,})\s+([a-zA-Z]{1,})\s+(.*)/){
$ssh->cmd(“ovm -uadmin -pwelcome1 vm poweroff -n $1 -s $6″);
sleep 12;
}
}
} elsif($operation eq “poweron”) {
open($poweron_fd,’<’,”/var/tmp/${host}.status”);
foreach(<$poweron_fd>){
if($_ =~ “Server_Pool|OVM|Running”) {
next;
}
if($_ =~ /(.*?)\s+([0-9]{1,})\s+([0-9]{1,})\s+([0-9]{1,})\s+([a-zA-Z]{1,})\s+Off(.*)/){
$ssh->cmd(“ovm -uadmin -pwelcome1 vm poweron -n $1 -s $6″);
#print “ovm -uadmin -pwelcome1 vm poweron -n $1 -s $6″;
sleep 20;
}
}
} elsif($operation eq “stat-r”) {
open($poweroff_fd,’<’,”/var/tmp/${host}.status”);
foreach(<$poweroff_fd>){
if($_ =~ /(.*?)\s+([0-9]{1,})\s+([0-9]{1,})\s+([0-9]{1,})\s+(Shutting\sDown|Initializing)\s+(.*)/){
#print “ovm -uadmin -pwelcome1 vm stat -r -n $1 -s $6″;
$ssh->cmd(“ovm -uadmin -pwelcome1 vm stat -r -n $1 -s $6″);
sleep 1;
}
}
}

You can use the following to make the script run in parallel:

for i in <all OVMs>;do (./ovm_vm_operation.pl $i status &);done

Categories: Clouding, IT Architecture, Oracle Cloud, Perl Tags:

avoid putty ssh connection sever or disconnect

January 17th, 2014 2 comments

After sometime, ssh will disconnect itself. If you want to avoid this, you can try run the following command:

while [ 1 ];do echo hi;sleep 60;done &

This will print message “hi” every 60 seconds on the standard output.

PS:

You can also set some parameters in /etc/ssh/sshd_config, you can refer to http://www.doxer.org/learn-linux/make-ssh-on-linux-not-to-disconnect-after-some-certain-time/

Categories: Linux, SHELL, Unix Tags:

install java jdk on linux

January 7th, 2014 No comments

Here’s the steps if you want to install java on linux:

wget <path to jre-7u25-linux-x64.rpm> -P /tmp
rpm -ivh /tmp/jre-7u25-linux-x64.rpm
mkdir -p /root/.mozilla/plugins
rm -f /root/.mozilla/plugins/libnpjp2.so
ln -s /usr/java/jre1.7.0_25/lib/amd64/libnpjp2.so /root/.mozilla/plugins/libnpjp2.so
ll /root/.mozilla/plugins/libnpjp2.so

Categories: Java, Linux Tags: ,

self defined timeout for telnet on Linux

December 26th, 2013 No comments

telnet’s default timeout value is relative high, so you may want to change timeout value to lower value such as 5 seconds. Here’s the way that we can fulfill this:

#!/bin/bash

timeout()
{
waitfor=5
command=$*
$command &
commandpid=$!
( sleep $waitfor ; kill -9 $commandpid > /dev/null 2>&1 ) &
watchdog=$!
sleeppid=$PPID
wait $commandpid > /dev/null 2>&1
kill $sleeppid > /dev/null 2>&1
}

timeout telnet slcc29-scan1.us.oracle.com 1521 >> $output

Also, we can use expect and set timeout for expect. When telnet is integrated with expect, we can fulfill timeout for telnet through using expect’s timeout value:

#!/usr/bin/expect

set timeout 30

send “<put telnet command here>\r”

Categories: Programming, SHELL Tags: , ,

remove duplicate images using fdupes and expect in linux

December 13th, 2013 No comments

I’ve got several thousands of pictures, but most of them had several exact copies of themselves. So I had to remove duplicate ones by hand firstly.

Later, I thought of that in linux we had md5sum which will give the same string for files with exact same contents. Then I tried to write some program, and that toke me some while.

I searched google and found that in linux, we had fdupes which can do the job very well. fdupes will calculate duplicate files based on file size/md5 value, and will prompt you to reserve one copy or all copies of the duplicates and remove others if you gave -d parameter to it. You can read more about fdupes here http://linux.die.net/man/1/fdupes

As all the pictures were on a windows machine, so I installed cygwin and installed fdupes and expect. Later I wrote a small script to reserve only one copy of the duplicate pictures for me(you will have to enter your option either reserving one copy or all copies by hand if you do not use expect, as there’s no option for reserve one copy by the author of fdupes). Here’s my program:

$ cat fdupes.expect
#!/usr/bin/expect
set timeout 1000000
spawn /home/andy/fdupes.sh
expect “preserve files” {
send “1\r”;exp_continue
}

$ cat /home/andy/fdupes.sh
fdupes.exe -d /cygdrive/d/pictures #yup, my pictures are all on this directory on windows, i.e. d:\pictures

After this, you can just run fdupes.expect, and it will reserve only one copy and remove other duplicates for you.

Categories: Programming, SHELL Tags: ,

quick configuration of python httpd server

November 28th, 2013 No comments

Let’s assume that you want to copy files from server A to server B, and you have found that no scp available, but wget is there for use. Then you can try run one python command and use wget to download files from server A.

Here’s the steps:

On server A:

cd <directory of files you want to copy>

python -m SimpleHTTPServer #notice the output of this command, for example, “Serving HTTP on 0.0.0.0 port 8000 …”

Now you can open browser and visit http://<hostname of server A>:8000. You will notice files are there now.

On server B:

wget http://<hostname of server A>:8000/<files to copy>

After you’ve copied files, you can press control+c to terminate that python http server on Server A. (Or you can press ctrl+z, and then bg %<job id> to make that python httpd server run in background)

Categories: Programming Tags:

resolved – how to show all results in one page when searching your wordpress blog

November 13th, 2013 No comments

Assume that you have your own wordpress blog, and you note down everything you met in daily work.

Now you have some trouble again in work, and remembered that you’ve noted similar issue before. So you tried searching your wordpress blog with a keyword such as “trouble”. Later, wordpress returned a result of 30 pages, each page had 10 articles. Now you scrolled and click “next page” a lot and that really frustrated you. What if you have all the searching result in one page? Thus you just need scroll the page and no waiting for loading pages of next, next, next page. (You may worry that the page load time will disappoint other guys searching your blog, but this proves to be little to worry, as no body will search your blog except yourself. Believe me buddy!)

Here goes the way to fulfill this functionality:

  1. Go to wordpress admin page, then click “Appearance” -> “Editor”;
  2. Click archive.php in the right to edit this file(search.php refers to archive.php, so you should edit archive.php);
  3. Search for “have_posts()”, and add one line above that line. The line to be added is like this: <?php query_posts($query_string . ‘&showposts=30′); ?> You may change 30 here to any number you want. As you guessed, this is the number that will show after searching.
  4. Save the change and try searching again. You’ll notice the change.

PS:

  1. Note that every time you upgrade wordpress or your wordpress theme you may need to do above steps again;
  2. The idea is from http://wordpress.org/support/topic/show-all-content-on-search-page
Categories: Programming Tags: ,

make tee to copy stdin as well as stderr & prevent ESC output of script

October 30th, 2013 No comments
  • Make tee to copy stdin as well as stderr

As said by manpage of tee:

read from standard input and write to standard output and files

So if you have error messages in your script, then the error messages will not copied and write to file.

Here’s one workaround for this:

./aaa.sh 2>&1 | tee -a log

Or you can use the more complicated one:

command > >(tee stdout.log) 2> >(tee stderr.log >&2)

  • Prevent ESC output of script

script literally captures every type of output that was sent to the screen. If you have colored or bold output, this shows up as esc characters within the output file. These characters can significantly clutter the output and are not usually useful. If you set the TERM environmental variable to dumb (using setenv TERM dumb for csh-based shells and export TERM=dumb for sh-based shells), applications will not output the escape characters. This provides a more readable output.

In addition, the timing information provided by script clutters the output. Although it can be useful to have automatically generated timing information, it may be easier to not use script’s timing, and instead just time the important commands with the time command mentioned in the previous chapter.

PS:

  1. Here’s the full version http://stackoverflow.com/questions/692000/how-do-i-write-stderr-to-a-file-while-using-tee-with-a-pipe
  2. Some contents of this article is excerpted from <Optimizing Linux® Performance: A Hands-On Guide to Linux® Performance Tools>.
Categories: Linux, SHELL Tags:

use batch script to start up & shutdown Virtualbox VMs

October 28th, 2013 No comments

I woke up before 8 every day on weekdays, and want to poweron two VMs in virtualbox named “xp” and “win2008″. So I can write a script and put it in “startup” folder, then these two VMs will startup with system automatically:

@echo off
date /t | find “Mon” && goto 1
date /t | find “Tue” && goto 1
date /t | find “Wed” && goto 1
date /t | find “Thu” && goto 1
date /t | find “Fri” && goto 1
exit

:1
if %time:~0,2% leq 8 (
c:\VirtualBox\VBoxManage startvm win2008 –type gui
c:\VirtualBox\VBoxManage startvm xp –type gui
) else exit

And I also want to shutdown these two VMs in one run:

c:\VirtualBox\VBoxManage controlvm win2008 acpipowerbutton
c:\VirtualBox\VBoxManage controlvm xp acpipowerbutton

PS:

You may also consider group policy(gpedit.msc -> Computer Configuration -> Windows Settings -> Scripts -> Shutdown) in windows so that when you shutdown your pc, all VMs will turned off automatically if you have a GPO for shutdown. More in

 

Categories: Programming, Windows Tags: ,

bash & expect tips

August 19th, 2013 No comments


/u02/0-sync.sh

#!/usr/bin/expect
set timeout 10800
spawn /u02/sync.sh
expect “*assword*” {send “test\r”;exp_continue
expect “*#*” {close}
}

/u02/sync.sh

#! /bin/sh
logpathroot=/u02/logs
synclist=/u02/sync.list

num=1
allnum=`cat $synclist | wc -l`
while [ $num -le $allnum ];
do
line=`awk ‘{if(NR==’$num’) print}’ $synclist`
component=`echo $line | awk ‘{print $2}’`
logtype=`echo $line | awk ‘{print $3}’`
host=`echo $line | awk ‘{print $4}’`
source=`echo $line | awk ‘{print $5}’`
env=`echo $line | awk ‘{print $1}’`
if [[ "$env" =~ "#" ]]; then
env=`echo $env | sed “s/#//”`
lnenv=`ls /u01/logs | grep -i “$env”`
exist=`ls “/u01/logs/${lnenv}/${component}_${logtype}-$host”`
if [ "$exist" != "" ]; then
rm -f /u01/logs/${lnenv}/${component}_${logtype}-$host
fi
else
lnenv=`ls /u01/logs | grep -i “$env”`
/bin/mkdir -p “${logpathroot}/${env}/${component}/${logtype}-$host”
/usr/bin/rsync -ave ssh –exclude ‘.zfs’ –delete-excluded logs@${host}:${source}/ ${logpathroot}/${env}/${component}/${logtype}-$host
#chmod 755 -R “${logpathroot}/${env}/${component}/”
exist=`ls “/u01/logs/${lnenv}/${component}_${logtype}-$host”`
if [ "$exist" = "" ]; then
# rm -f /u01/logs/${lnenv}/${component}_${logtype}-$host
component1=`echo $component | tr “[:upper:]” “[:lower:]“`
logtype1=`echo $logtype | sed “s/-/_/”`
ln -sf ${logpathroot}/${env}/${component}/${logtype}-$host/ /u01/logs/${lnenv}/${component1}_${host}_${logtype1}_log
fi
fi
num=`expr $num + 1`
done

rm -f /u02/sync_at_*
echo > /u02/sync_at_”`date +%Y-%m-%d` `date +%H:%M` (UTC)”

/u02/sync.list

DC1 apex AdminServer_adr test27vmf4053 /u01/local/config/m_domains/base_domain/servers/AdminServer/adr
DC1 emgc EMGC_ADMINSERVER_adr test27cn05 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/adr
DC1 emgc EMGC_OMS1_adr test27cn05 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr
DC1 emgc EMGC_OMS2_adr test27cn06 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS2/adr
#DC1 emgc EMGC_OMS3_adr test27cn07 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS3/adr

Categories: SHELL, tips Tags:

expect feedback expect_out

July 30th, 2013 No comments

Note: this is from book <exploring expect>

expect_feedback_01

 

expect_feedback_02

Categories: Programming Tags:

parallel execution of script using perl fork()

July 3rd, 2013 No comments

Let’s assume that we want to test 100 hosts to see whether they’re pingable or not. You may think of bash style for … do … done loop, but as that is executed one by one, so it’s not efficient. And in this article, I’m going to create parallel execution of this task using perl’s fork().

Firstly, here’s one script to get whether one host is pingable or not:

[root@centos-doxer pping_doxer]# cat script.sh
#!/usr/bin/expect
log_user 0
match_max -d 2000000
set host [lindex [lrange $argv 0 0] 0]
spawn -noecho ping $host
set timeout 2
expect “bytes from” {exit 0} default {puts “not pingable from $host”;exit 1}

Let’s run it and see the result:

[root@centos-doxer pping_doxer]# ./script.sh test-host
not pingable from test-host

So now, let’s write one perl script to wrapper this expect script, and also add parallel run to it:

#!/usr/bin/perl
our @hosts=(“testhost0001″,”testhost00022″,”testhost00033″,”testhost0007″);
#you can read a file to perl array too
#open($fh,”/doxer/projects/pping_doxer/servers.txt”);
#our @hosts=<$fh>;
our $zombies = 0;
our $kid_proc_num = 0;

$SIG{CHLD} = sub { $zombies++ };
for(my $i=0; $i<@hosts; $i++) {
my $pid = fork();
if( !defined($pid) ) { exit 1; }
unless($pid) {
system(“/doxer/projects/pping_doxer/script.sh $hosts[$i]“); #change the path according to your env
exit 0;
}
$kid_proc_num++;
}

while (1) {
if($zombies > 0) {
$zombies = 0;
my $collect;
while(($collect = waitpid(-1, WNOHANG)) > 0) {
$kid_proc_num–;
}
}
if($kid_proc_num==0) { last; }
else { next; }
}

And now let’s have a test:

[root@centos-doxer pping_doxer]# ./pping_doxer
not pingable from testhost0001
not pingable from testhost00033
not pingable from testhost0007
not pingable from testhost00022

You can modify script.sh to fulfill your own aim except for ping of course.

That’s it, enjoy!

PS:

1.I’ve wrote an updated version of pping_doxer, which allows you to specify a maximum number of parallel running processes and also puts hosts in a file. I’ve put the package in http://www.doxer.org/projects-doxer/, the package name is pping_doxer.

2.Thanks very much to http://www.ibm.com/developerworks/cn/linux/l-cn-perlmp/.

Categories: Perl, Programming Tags: ,

understanding perl fork

July 2nd, 2013 No comments

Let’s go through perl fork() by example:

#!/usr/bin/perl -w
use strict;

defined(my $pid=fork()) or die “Fork process failured:$!\n”;  #two processes are running now, one parent($pid is the pid of child process which is not zero) and child process($pid is zero)
unless($pid)
{
# This is the child process.
system “date”;
sleep(3);
print (“Exit child after 3 seconds wait!\n”);
exit();
}
# This is the parent process.
waitpid($pid,0);
system “date”;
print (“exit parent!\n”);

Here’s the result:

Tue Jul 2 18:29:33 CST 2013
Exit child after 3 seconds wait!
Tue Jul 2 18:29:36 CST 2013
exit parent!

And now let’s test when waitpid() is commented out:

#!/usr/bin/perl -w
use strict;

defined(my $pid=fork()) or die “Fork process failured:$!\n”;
unless($pid)
{
# This is the child process.
system “date”;
sleep(3);
print (“Exit child after 3 seconds wait!\n”);
exit();
}
# This is the parent process.
#waitpid($pid,0);  #parent no longer wait until child process exit
system “date”;
print (“exit parent!\n”);

And here’s the result:

[root@centos-doxer test]# ./perl.test
Tue Jul 2 18:30:05 CST 2013
Tue Jul 2 18:30:05 CST 2013
exit parent!

Categories: Perl Tags:

regular expression tips

July 1st, 2013 No comments
if($reply =˜ m/ˆ[0-9]+$/ig) # i is modifier
$var =˜ s/\bJeff\b/Jeff/i; #replace
$text =˜s/ˆ$/<p>/mg; #m treat string as multiple lines
$/ = “.\n”; #paragraph, string
#greedy
Source:aa<div>test1</div>bb<div>test2</div>cc
Greedy:<div>.*</div>
result:<div>test1</div>bb<div>test2</div>
Non-Greedy:<div>.*?</div>
result<div>test1</div>
 /<(.*?)>.*?<\/\1>/ #inside the

(?:xxx) #?: do not take into variables
file-slurp mode special
Categories: Programming, tips Tags:

expect eof timeout default close wait

June 30th, 2013 No comments
  • timeout, eof

I’ll take the following example:

#!/usr/local/bin/expect –
set timeout [lindex $argv 0]
eval spawn [lrange $argv 1 end]
expect {
timeout {puts “took too much time”}
eof {puts “finished in time”}
}

And let’s test this:

% maxtime 2 sleep 5
spawn sleep 5
took too much time
% maxtime 5 sleep 2
spawn sleep 2
finished in time

  • default

expect “alive” {exit 0} default {exit 1} #expect default instead of using expect “alive” {exit 0} timeout {exit 1} eof {exit 1}

  • close

By closing the connection, Expect is telling the spawned process that it has nothing
more to say to the process. Usually the process takes this as an indication to exit. This
is similar to what occurs when you press ^D while manually interacting with a process.
The process does not see the ^D. Rather, the system turns this into an eof. The process
reads the eof and then responds by closing the connection and exiting.

Here is an example of why you might want to call close explicitly. Imagine you are
interacting with ftp. If you have an “ftp>” prompt, you can send the command quit
\r and ftp will immediately exit, closing the connection from its end. But suppose
ftp is in the middle of transferring a file and you need to close the connection imme-
diately. You could interrupt ftp, wait for it to prompt, and then send the quit\r com-
mand. But it is simpler to just close the connection. ftp will abort the transfer and quit.

  • wait

After closing the connection, a spawned process can finish up and exit. Processes exit
similarly to the way Expect scripts do, with a number (for example, “exit 0″). The
operating system conveniently saves this number and some other information about
how the process died. This information is very useful for non-interactive commands
but useless for interactive commands. Consequently, it is of little value to Expect.
Nonetheless, Expect must deal with it.
Expect must retrieve this information—even if only to discard it. The act of retrieving
the information frees various valuable resources (process slots) within the computer.
Until the information is retrieved, the operating system maintains the information in-
definitely. This can be seen from the output of ps. Assuming a spawned process has
died and the connection has been closed, ps shows something like this:
PID TT STAT TIME COMMAND
4425 ? Z 0:00 <defunct>
The Z stands for zombie—someone’s attempt to humorously describe a process that is
dead but still haunts the system in an almost useless way. Even the process name and
arguments have been discarded—no matter what they were originally, they show up
here as <defunct>.
To get rid of this zombie, use the wait command.

The wait command returns a list of elements including the spawn id and process id.
These elements are further described in Chapter 14 (p. 309). For now, ignore the return
value of wait.
Because a process will not disappear from the system until you give the wait command,
it is common to speak of waiting for or waiting on a process. Some people also like to
use the term reap as in “reaping a process”.
Because wait follows close, it is very common to see people write “close;wait” on a
single line. But if the connection is closed implicitly, the wait must appear by itself.
Like close, the wait command can also occur implicitly. Unlike close, however, wait
implicitly happens in only one case—when an Expect process (i.e., script) exits. On
exit, all the spawned processes are waited for.
This means that Expect scripts that only spawn a single process and then exit, need not
call wait since it will be done automatically. The example scripts so far have all taken
advantage of this. Later on, I will show a script in which it is important to explicitly wait.
One last thing about wait: If you call it before a process has died, your Expect script
will wait for the process to die—hence the name. It is possible to avoid the delay by
using the -nowait flag.
wait -nowait

PS:

Idea from book <Exploring Expect>

 

Categories: Programming Tags:

* At The End Of A Pattern Can Be Tricky

June 30th, 2013 No comments

Here’s the output:

% ftp ftp.uu.net
Connected to ftp.uu.net.
220 ftp.UU.NET FTP server (Version 6.34 Thu Oct 22 14:32:01 EDT 1992) ready.
Name (ftp.uu.net:don):

What should I do if I want the ftp version to be matched using expect?

Below is from book <Exploring Expect>

Even though the program may have printed “Connected to ftp.uu.net.\n” all at once
—perhaps by a single printf in a C program—the UNIX kernel can break this into
small chunks, spitting out a few characters eachtime to the terminal. For example, it
might print out “Conn” and then”ecte” and then “d to” and so on. Fortunately, com-
puters are so fast that humans do not notice the brief pauses in the middle of output.
The reason the system breaks up output like this is that programs usually produce
characters faster than the terminal driver can display them. The operating system will
obligingly wait for the terminal driver to effectively say, “Okay, I’ve displayed that last
bunch of characters. Send me a couple more.” In reality, the system does not just sit
there and wait. Since it is running many other programs at the same time, the system
switches its attention frequently to other programs. Expect itself is one such “other
program” in this sense.

When Expect runs, it will immediately ask for all the characters that a program pro-
duced only to find something like “Conn”. If told to wait for a string that
matches “Name*:”, Expect will keep asking the computer if there is any more output,
and it will eventually find the output it is looking for.
As I said, humans are slow and do not notice this chunking effect. In contrast, Expect
is so fast that it is almost always waiting. Thus, it sees most output come as chunks
rather than whole lines. With this in mind, suppose you wanted to find out the version
of ftp that a host is using. By looking back at the output, you can see that it is contained
in the greeting line that begins “220″ and ends with “ready.”. Naively, you could wait
for that line as:
expect “220*” ;# dangerous
If you are lucky, you might get the entire line stored in $expect_out(0,string). You
might even get the next line in there as well. But more likely, you will only get a fragment,
such as “220 f” or “220 ftp.UU.NE”. Since the pattern 220* matches either of these,
expect has no reason to wait further and will return. As I stated earlier, expect returns
with whatever is the longest string that matches the pattern. The problem here is that
the remainder of the line may not have shown up yet!
If you want to get the entire line, you must be more specific. The following pattern
works:
“220*ready.”
By specifying the text that ends the line, you force expect to wait for the entire line to
arrive. The “.” is not actually needed just to find the version identifier. You could just
make the pattern:
“220*re”
Leaving off the e would be too short. This would allow the pattern to match the r in
server rather than ready. It is possible to make the overall pattern even shorter by
looking for more unusual patterns. But quite often you trade off readability. There is
an art to choosing patterns that are correct, yet not too long but still readable. A good
guideline is to give more priority to readability. The pattern matching performed by
Expect is very inexpensive.

Categories: Programming, SHELL Tags:

python module install

June 24th, 2013 No comments

Here’s one example of installing module from source:

wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz
tar zxvf setuptools-0.6c11.tar.gz
cd setuptools-0.6c11
python setup.py build
python setup.py install

And also you can use a packaging management tool to install(pip, distribute, setuptool, easy_install). And here’s one image about relationship between them:

python_module

Categories: Programming Tags:

Double-word example in Perl

June 18th, 2013 No comments

double-word

 

1. Because the doubled-word problem must work even when the doubled
words aresplit across lines, I can’t use the normal line-by-line processing I
used with the mail utility example. Setting the special variable$/(yes, that’s
avariable) as shown puts the subsequent<> into a magic mode such that it
retur nsnot single lines, but more-or-less paragraph-sized chunks. The value
retur ned is just one string, but a string that could potentially contain many of
what we would consider to be logical lines.

2.Did you notice that I don’t assign the value from<>to anything? When used

as the conditional of awhile like this,<>magically assigns the string to a
special default variable.†That same variable holds the default string that
s/˙˙˙/˙˙˙/works on, and thatprintdisplays. Using these defaults makes the
pr ogram less cluttered, but also less understandable to someone new to the
language, so I recommend using explicit operands until you’recomfortable.

3.Thenext unlessbefor ethe substitute command has Perl abort processing on

the current string (to continue with the next) if the substitution doesn’t actu-
ally do anything. There’s no need to continue working on a string in which
no doubled words arefound.

4.The replacement string is really just”$1 $2 $3″with interveningANSIescape

sequences that provide highlighting to the two doubled words, but not to
whatever separates them. These escape sequences are\e[7m to begin high-
lighting, and\e[m to end it. (\e is Perl’s regex and string shorthand for the
ASCIIescape character,which begins theseANSIescape sequences.)
Looking at how the parentheses in the regex are laid out, you’ll realize that
"$1$2$3" repr esentsexactly what was matched in the first place. So, other
than adding in the escape sequences, this whole substitute command is
essentially a (slow) no-op.
We know that$1and$3 repr esentmatches of the same word (the whole
point of the program!), so I could probably get by with using just one or the
other in the replacement. However,since they might differ in capitalization, I
use both variables explicitly.

5.The string may contain many logical lines, but once the substitution has

marked all the doubled words, we want to keep only logical lines that have
an escape character.Removing those that don’t leaves only the lines of inter-
est in the string. Since we used the enhanced line anchor match mode (the
/mmodifier) with this substitution, the regex !ˆ([ˆ\e]+\n)+ “can find logical
lines of non-escapes. Use of this regex in the substitute causes those
sequences to be removed. The result is that only logical lines that have an
escape remain, which means that only logical lines that have doubled words
in them remain.‡

6.The variable$ARGVmagically provides the name of the input file. Combined

with/mand/g, this substitution tacks the input filename to the beginning of
each logical line remaining in the string.Cool!

Finally, theprintspits out what’s left of the string, escapes and all. Thewhile
loop repeats the same processing for all the strings (paragraph-sized chunks of
text) that areread from the input.

  • Note: this is from book <Mastering Regular Expressions>
Categories: Perl, Programming Tags:

Non-Capturing Parentheses of regular expression

June 18th, 2013 No comments

Consider the following expression:

if ($input =˜ m/ˆ([-+]?[0-9]+(?:\.[0-9]+)?)([CF])$/)

Now, even though the parentheses surrounding ![CF] “ar eostensibly the third
set, the text they match goes to$2since, for counting purposes, the !(?:˙˙˙) ”
set doesn’t, well, count.

NOTE: this is from book <Mastering Regular Expressions>.

Categories: Perl, Programming Tags:

word boundaries in egrep

June 17th, 2013 No comments

Here’s one example:

[root@doxer_#1]# cat a
mcat me
here cat is
ok concaten

[root@doxer_#1]# egrep ‘\<cat\>’ a
here cat is

[root@doxer_#1]# egrep ‘\<cat’ a
here cat is

And here goes the details:

word-boundaries

Categories: Programming, SHELL Tags:

perl tips – most useful perl command line options switches

June 9th, 2013 No comments

Here’s all perl command line options, red ones are the most useful ones from me:

[root@centos-doxer test]# perl –help
Usage: perl [switches] [--] [programfile] [arguments]
-F/pattern/ split() pattern for -a switch (//’s are optional)
-l[octal] enable line ending processing, specifies line terminator #man ascii to see all ascii – octal mapping
-c check syntax only (runs BEGIN and CHECK blocks) #two simple, won’t talk more about this
-e program one line of program (several -e’s allowed, omit programfile) #e.g. perl -e ‘print “hi\n”;print “hi again\n”;’
-p assume loop like -n but print line also, like sed #perl -p -e “tr/[a-z]/[A-Z]/” a.txt
-n assume “while (<>) { … }” loop around program #like -p, but do not print
-a autosplit mode with -n or -p (splits $_ into @F) #perl -n -l072 -a -F/:/ -e ‘print $F[0]‘ a.txt
-i[extension] edit <> files in place (makes backup if extension supplied) #perl -pi.bak -e “tr/[a-z]/[A-Z]/” a.txt
-[mM][-]module execute “use/no module…” before executing program #perl -MCPAN -e ‘install xxx:xxx’
-s enable rudimentary parsing for switches after programfile 

[root@centos-doxer test]# cat perl.test
#!/usr/bin/perl -s
print “yes\n” if defined($r);
print “no\n” unless defined($r);

[root@centos-doxer test]# ./perl.test -a
no
[root@centos-doxer test]# ./perl.test test
no
[root@centos-doxer test]# ./perl.test -r
yes

echo ‘    (name 47850_WB2S-rdkG_wlsaas_131121_0603_virt16170__wlsaas_as_as-0)’|perl -p -e “s/^.*name\s\d{5}_(.*)\)/\1/g” #this will return WB2S-rdkG_wlsaas_131121_0603_virt16170__wlsaas_as_as-0

-C[number/list] enables the listed Unicode features
-0[octal] specify record separator (\0, if no argument)
-d[:debugger] run program under debugger
-D[number/list] set debugging flags (argument is a bit mask or alphabets)
-f don’t do $sitelib/sitecustomize.pl at startup
-Idirectory specify @INC/#include directory (several -I’s allowed)
-P run program through C preprocessor before compilation

-S look for programfile using PATH environment variable
-t enable tainting warnings
-T enable tainting checks
-u dump core after parsing program
-U allow unsafe operations
-v print version, subversion (includes VERY IMPORTANT perl info)
-V[:variable] print configuration summary (or a single Config.pm variable)
-w enable many useful warnings (RECOMMENDED)
-W enable all warnings
-x[directory] strip off text before #!perl line and perhaps cd to directory
-X disable all warnings

Categories: Perl, tips Tags:

perl – match fixed number of spaces or other characters

June 6th, 2013 No comments

Sometimes you want to match lines beginning with exactly four spaces(“<space><space><space><space>”) :

<space><space><space><space><space>Line1

<space><space><space><space>Line2  –only this line!

<space><space>Line3

So here’s the perl script to fulfill this:

#!/usr/bin/perl
open($fdme, ‘/doxer/test/a.txt’);
foreach(<$fdme>){
if($_ =~ /^\s{4}\w+.*/){
print $_.”\n”;
}
}

Categories: Perl, Programming Tags:

install perl module using tarball or cpan

June 2nd, 2013 No comments
##Install perl modules 
cpan> o conf prerequisites_policy follow
cpan> o conf commit
cpan > o conf init /proxy/
#PERL_MM_USE_DEFAULT=1 perl -MCPAN -e ‘install Net::SSH::Perl’
export PERL_MM_USE_DEFAULT=1
cpan> force install Net::SSH::Perl
To test whether the module has been installed or not, run this:

perl -e ‘use Module::Metadata’, if nothing returned, then it’s ok

To manually install a Perl module:
1. Download the Perl module from CPAN or other site.
2. Extract the tarball.
3. Run perl Makefile.PL
4. Run make
5. Run make test
6. Run make install
cpan> help #o conf init to reconfigure cpan
Display Information
 command  argument          description
 a,b,d,m  WORD or /REGEXP/  about authors, bundles, distributions, modules
 i        WORD or /REGEXP/  about anything of above
 r        NONE              reinstall recommendations
 ls       AUTHOR            about files in the author’s directory
 
Download, Test, Make, Install…
 get                        download
 make                       make (implies get)
 test      MODULES,         make test (implies make)
 install   DISTS, BUNDLES   make install (implies test)
 clean                      make clean
 look                       open subshell in these dists’ directories
 readme                     display these dists’ README files
 
Other
 h,?           display this menu       ! perl-code   eval a perl command
 o conf [opt]  set and query options   q             quit the cpan shell
 reload cpan   load CPAN.pm again      reload index  load newer indices
 autobundle    Snapshot                force cmd     unconditionally do cmd
Categories: Perl, Programming Tags:

resolved – perl get only package/file names and not including package version number using regular expression

May 9th, 2013 No comments

I got a list of packages installed on my centos linux box via:

rpm -qa|sort|uniq|sort

The result is like:

acl-2.2.39-8.el5
acpid-1.0.4-12.el5
alacarte-0.10.0-1.fc6
alsa-lib-1.0.17-1.el5
alsa-lib-devel-1.0.17-1.el5
amtu-1.0.6-2.el5
anacron-2.3-45.el5.centos
apr-1.2.7-11.el5_6.5
apr-util-1.2.7-11.el5_5.2

But now I only want the package names without version number, that is like:

acl
acpid
alacarte
alsa-lib
alsa-lib-devel
amtu
anacron
apr
apr-util

Here’s the perl script to fulfill this:

#!/usr/bin/perl
open($temp1,”<”,”/doxer/test/package-names.txt”) or die(“error opening”);
while(<$temp1>){
if($_ =~ /([a-zA-Z].*?)\-[0-9]{1,}\.[0-9]{1,}.*/){
print $1.”\n”;
}
}

Categories: Perl, Programming Tags:

perl script for getting sun zfs head project and share usage info

April 26th, 2013 No comments

Sometimes you would like to know on sun zfs head, which project occupies most of the space, and which shares of that occupies most space of that project.

Here’s a perl script to fulfill this(it’s a little cumbersome, but it works anyway)

#!/usr/bin/perl
use strict;
use warnings;
use Net::SSH::Perl;
use List::Util qw/sum/;
my $host = ‘test-zfs-host’;
my $user = ‘root’;
my $password = ‘password’;

my $ssh = Net::SSH::Perl->new($host);
$ssh->login($user,$password);
my ($stdout,$stderr,$exit) = $ssh->cmd(“shares show”);
my @std_arr=split(/:/,$stdout);
my @projects_arr = split(/\n/, $std_arr[2]);

foreach(@projects_arr){
$_ =~ s/^\s+|\s+$//g;
}
shift @projects_arr;
pop @projects_arr;
pop @projects_arr;

my @space_projects;
foreach(@projects_arr){
my ($stdout2,$stderr2,$exit2) = $ssh->cmd(“shares select $_ get”);
my @stdout_arr=split(/\n/,$stdout2);
my $space_temp=join(“\n”,grep(/space_total/,@stdout_arr));
my @space_total_project = “project “.$_.$space_temp;
push(@space_projects,@space_total_project);
}
my @space_projects3;
foreach(@projects_arr){
my ($stdout3,$stderr3,$exit3) = $ssh->cmd(“shares select $_ ls”);
my @stdout_arr3=split(/\n/,$stdout3);
my @space_temp3=grep(/\/export\//,@stdout_arr3);
my @space_temp4=grep(!/mountpoint/,@space_temp3);
push(@space_projects3,@space_temp4);
}
open(my $temp1,’>’,’/var/tmp/temp1′) or die(“cannot open file temp1″);
open(my $temp2,’>’,’/var/tmp/temp2′) or die(“cannot open file temp2″);
open(my $temp3,’>’,’/var/tmp/temp3′) or die(“cannot open file temp3″);
open(my $temp4,’>’,’/var/tmp/temp4′) or die(“cannot open file temp4″);
my @var_T;
my @var_G;
my @var_M;
my @var_K;

foreach(@space_projects){
if($_ =~ /.+space_total\s+(.*)K/){
push(@var_K,$_);
}
elsif($_ =~ /.+space_total\s+(.*)M/){
push(@var_M,$_);
}
elsif($_ =~ /.+space_total\s+(.*)G/){
push(@var_G,$_);
}
elsif($_ =~ /.+space_total\s+(.*)T/){
push(@var_T,$_);
}
}

select $temp1;
foreach(@var_T){
print $_.”\n”;
}
close $temp1;

select $temp2;
foreach(@var_G){
print $_.”\n”;
}
close $temp2;
select $temp3;
foreach(@var_M){
print $_.”\n”;
}
close $temp3;
select $temp4;
foreach(@var_K){
print $_.”\n”;
}
close $temp4;

system(“echo \”======zfs project usage info(Descending)======\””);
system(“sort -r -n -k 5 /var/tmp/temp1;sort -r -n -k 5 /var/tmp/temp2;sort -r -n -k 5 /var/tmp/temp3;sort -r -n -k 5 /var/tmp/temp4″);
open(my $temp5,’>’,’/var/tmp/temp5′) or die(“cannot open file temp5″);
open(my $temp6,’>’,’/var/tmp/temp6′) or die(“cannot open file temp6″);
open(my $temp7,’>’,’/var/tmp/temp7′) or die(“cannot open file temp7″);
open(my $temp8,’>’,’/var/tmp/temp8′) or die(“cannot open file temp8″);
my @var_T_2;
my @var_G_2;
my @var_M_2;
my @var_K_2;

foreach(@space_projects3){
if($_ =~ /\s+.*K\s+.*/){
push(@var_K_2,$_);
}
elsif($_ =~ /\s+.*M\s+.*/){
push(@var_M_2,$_);
}
elsif($_ =~ /\s+.*G\s+.*/){
push(@var_G_2,$_);
}
elsif($_ =~ /\s+.*T\s+.*/){
push(@var_T_2,$_);
}
}

select $temp5;
foreach(@var_T_2){
print $_.”\n”;
}
close $temp5;

select $temp6;
foreach(@var_G_2){
print $_.”\n”;
}
close $temp6;
select $temp7;
foreach(@var_M_2){
print $_.”\n”;
}
close $temp7;
select $temp8;
foreach(@var_K_2){
print $_.”\n”;
}
close $temp8;

system(“echo \”\n\n\n======zfs share usage info(Descending)======\””);
system(“sort -r -n -k 2 /var/tmp/temp5;sort -r -n -k 2 /var/tmp/temp6;sort -r -n -k 2 /var/tmp/temp7;sort -r -n -k 2 /var/tmp/temp8″);

The output would be like:

======zfs project usage info(Descending)======
project DC2_DMZ space_total = 7.68T
project dc2_c9testga space_total = 1.10T
project fa_trialadcf space_total = 277G
project fa_rehydration space_total = 266G
project common space_total = 10.0G
project NODE_8 space_total = 93K
project default space_total = 31K

 

======zfs share usage info(Descending)======
Service_Mid-2 1.44T /export/DC2_DMZ/Service_Mid-2
Service_Web 1.22T /export/DC2_DMZ/Service_Web
dc2_shared_idm 743G /export/DC2_DMZ/dc2_shared_idm
Infra_Web 400G /export/DC2_DMZ/Infra_Web
nuviaq_local02 988M /export/DC2_DMZ/nuviaq_local02
sftp_staging 127M /export/DC2_DMZ/sftp_staging
sftp_local03 14.8M /export/DC2_DMZ/sftp_local03
sftp_manager_local01 85K /export/DC2_DMZ/sftp_manager_local01

Categories: Perl Tags: , ,