Archive

Archive for the ‘Programming’ Category

resolved – Artifactory: Return code is: 502, ReasonPhrase:cannot connect

August 22nd, 2016 Comments off

Today when I tried to visit Maven Artifactory, below error prompted in browser:

Return code is: 502, ReasonPhrase:cannot connect.

And here's the log in

2016-08-19 22:48:19,942 [art-init] [ERROR] (o.a.w.s.ArtifactoryContextConfigListener:91) - Application could not be initialized: Connection refused
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_51]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_51
]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:
1.7.0_51]

......

Firstly, I thought it was caused by JDBC issue, here's the config in JDBC config file storage.properties:

# cat /u01/oracle/artifactory_home/etc/storage.properties
type=oracle
driver=oracle.jdbc.OracleDriver
url=jdbc:oracle:thin:@localhost:1521:ORCL
username=artifactory
password=password1

I tried connect to DB using sqlplus to have a test, it's OK:

[root@testvm ~]# su - oracle
[oracle@testvm ~]$ export ORACLE_HOME=/u01/oracle/db12c/product/12.1.0/dbhome_1
[oracle@testvm ~]$ ORACLE_SID=orcl
[oracle@testvm ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@testvm ~]$ sqlplus artifactory/password1

SQL*Plus: Release 12.1.0.1.0 Production on Mon Aug 22 03:28:20 2016

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Last Successful login time: Mon Aug 22 2016 03:21:26 +00:00

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL>

After some debugging, I found the issue was caused by proxy setting. We should disable it like below:

[oracle@testvm ~]# grep proxy /etc/profile
#export http_proxy=http://myproxy.example.com:80/

Then log out and log on the host again, and ran below commands to shutdown/startup Artifactory:

[oracle@testvm ~]$ export ORACLE_HOME=/u01/oracle/db12c/product/12.1.0/dbhome_1/
[oracle@testvm ~]$ export ORACLE_SID=orcl
[oracle@testvm ~]$ export PATH=$PATH:$ORACLE_HOME/bin
[oracle@testvm ~]$ /u01/oracle/apache-tomcat-7.0.52/bin/shutdown.sh
[oracle@testvm ~]$ /u01/oracle/apache-tomcat-7.0.52/bin/startup.sh

After this, Artifactory started working again.

Categories: IT Architecture, Programming Tags:

remove entries in perl array with specified value

December 30th, 2014 Comments off

Assume that in array @array_filtered:

my @array_filtered = ("your", "array", "here", 1, 3, 8, "here", 2, 5, 9, "sit", "here",3, 4, 7,"yes","now",8,1,7,6); #or my @array_filtered=qw(your array here 1 3 8 here 2 5 9 sit here 3 4 7 yes now 8 1 7 6) which uses Alternative Quotes(q, qq, qw, qx)

You want to remove entries that have value "here" or "now" and it's following 3 entries, you can use splice:

#!/usr/bin/perl
my @array_filtered = ("your", "array", "here", 1, 3, 8, "here", 2, 5, 9, "sit", "here",3, 4, 7,"yes","now",8,1,7,6);
my @search_for = ("here","now");
#return keys that have specified value, =~/!~ for regular expression, eq/ne for string, ==/!= for number. or use unless()/if(not()). use m{} instead of // if there's too much / in the expression and you're tired of using \/ to escape them.

$search_for_s=join('|',@search_for);
@index_all = grep { $array_filtered[$_] =~ /$search_for_s/ } 0..$#array_filtered;

for($i=0;$i<=$#index_all;$i++) {
@index_all_one = grep { $array_filtered[$_] =~ /$search_for_s/ } 0..$#array_filtered;
splice(@array_filtered,$index_all_one[0],4);
#print $indexone."\n"
}

print "@array_filtered"."\n";

The output is "your array sit yes 6".

PS:

  • For more info about perl regular expression(such as operators<m, s, tr> and their modifiers, complex regular expression cheat sheet<.\s\S\d\D\w\W[aeiou][^aeiou](foo|bar), \G, $, $&, $`, $'> and more), you can refer to this article.
  • The following is about perl alternative quotes:

q// is generally the same thing as using single quotes - meaning it doesn't interpolate values inside the delimiters.
qq// is the same as double quoting a string. It interpolates.
qw// return a list of white space delimited words. @q = qw/this is a test/ is functionally the same as @q = ('this', 'is', 'a', 'test')
qx// is the same thing as using the backtick operators.

Categories: IT Architecture, Perl, Programming Tags:

Resolved – AttributeError: ‘NoneType’ object has no attribute ‘_imgName’

November 6th, 2014 Comments off

Today when I tried to list Virtual Machines on one Oracle OVMM, error message prompted:

[root@ovmm_test ~]# ovm -uadmin -ppassword vm ls
Traceback (most recent call last):
  File "/usr/bin/ovm", line 43, in ?
    ovmcli.ovmmain.main(sys.argv[1:])
  File "/usr/lib/python2.4/site-packages/ovmcli/ovmmain.py", line 122, in main
    return ovm.ovmcli.runcmd(args)
  File "/usr/lib/python2.4/site-packages/ovmcli/ovmcli.py", line 147, in runcmd
    return method(options)
  File "/usr/lib/python2.4/site-packages/ovmcli/ovmcli.py", line 1578, in do_vm_ls
    result.append((serverpool._serverPoolName, vm._imgName))
AttributeError: 'NoneType' object has no attribute '_imgName'

Then I tried list VMs by server pool:

[root@ovmm_test ~]# ovm -uadmin -ppassword vm ls -s Pool1_test
Name                 Size(MB) Mem   VCPUs Status  Server_Pool
testvm1              17750    8196  4     Running Pool1_test
testvm2               50518    8196  4     Running Pool1_test
testvm3          19546    8192  2     Running Pool1_test
testvm4          50518    20929 4     Running Pool1_test
testvm5          19546    8192  2     Running Pool1_test
[root@ovmm_test ~]# ovm -uadmin -ppassword vm ls -s Pool1_test_A
Traceback (most recent call last):
  File "/usr/bin/ovm", line 43, in ?
    ovmcli.ovmmain.main(sys.argv[1:])
  File "/usr/lib/python2.4/site-packages/ovmcli/ovmmain.py", line 122, in main
    return ovm.ovmcli.runcmd(args)
  File "/usr/lib/python2.4/site-packages/ovmcli/ovmcli.py", line 147, in runcmd
    return method(options)
  File "/usr/lib/python2.4/site-packages/ovmcli/ovmcli.py", line 1578, in do_vm_ls
    result.append((serverpool._serverPoolName, vm._imgName))
AttributeError: 'NoneType' object has no attribute '_imgName'

One pool was working and the other was not, so the problematic VMs must reside in pool Pool1_test_A.

Another symptom was that, although ovmcli wouldn't work, the OVMM GUI worked as expected and returns all the VMs.

As ovmcli read entries from Oracle DB(SID XE) on OVMM, so the issue maybe caused by the inconsistency between DB & OVMM agent DB.

I got the list of all VMs on the problematic server pool from OVMM GUI, and then ran the following query to get all entries in DB:

select IMG_NAME from OVS_VM_IMG where SITE_ID=110 and STATUS !='Running' and length(IMG_NAME)>50; #Pool1_test_A was with ID 110, got from table OVS_SITE. I used length() here because in the problematic server pool, VMs all should have IMG_NAME with more than 50 characters; if less than 50, then they were VM templates which should have no issue

Comparing the output from OVMM GUI & OVM DB, I found some entries which only existed in DB. And for all of these entries, they all had "Status" in "Creating", and also the

select IMG_NAME from OVS_VM_IMG where STATUS='Creating';

Then I removed these weird entries:

delete from OVS_VM_IMG where SITE_ID=110 and STATUS !='Running' and length(IMG_NAME)>50; #you can try rename/drop table OVS_VM_IMG(alter table TBL1 rename to TBL2; drop table TBL1), remove entries in backup table(OVS_VM_IMG_bak20141106), and then rename backup table(OVS_VM_IMG_bak20141106) to OVS_VM_IMG if failed at this step caused by foreign key or other reasons (don't do this, it will cause problem about constraints etc)

After this, the issue got resolved.

PS:

  1. After you removed entries with STATUS being "Creating", and if you found some more entries of this kind occurred in OVM DB, then maybe it's caused by VM templates not working or DB table corrupted. In this case, you'll need recover OVMM by rollong back to previous version of your backup, and then import VM templates/VM images etc.
  2. Actually the issue was caused by breaking constraint(constraint name conflicts caused by table backups. So better not to backup table when doing operation against OVMM DB using sql directly). This issue can be resolved by alter constraint name, and later remove backup tables.

perl tips

April 2nd, 2014 Comments off
##system value
$count=`wc -l space_mail_info.txt | sed 's/ space_mail_info.txt//'`;
if ($count >= 1) {
system('cat space_mail_info.txt | /bin/mailx -s "SAs - Space Warning - please work with component owners to free space" test@example.com');
}
##arrays
#!/usr/bin/perl -w
my @animals = ("dog", "pig", "cat");
print "The last element of array \$animals is : ".$animals[$#animals]."\n";
print "@animals"."\n"; #will print values of array, delimitered by space
print $#animals."\n"; #the last key number of array, $#animals+1 is the number of array
if(@animals>2){
print "more than 2 animals found\n";
}
else{
print "less than 2 animals found\n"
}
foreach(@animals){
print $_."\n";
}
##hashes
my %fruit_color=("apple", "red", "banana", "yellow");
print "Color of banana is : ".$fruit_color{"banana"}."\n";

for $char (keys %fruit_color)
{
print("$char => $fruit_color{$char}\n");
}

##references
my $variables = {
scalar  =>  {
description => "single item",
sigil => '$',
},
array   =>  {
description => "ordered list of items",
sigil => '@',
},
hash    =>  {
description => "key/value pairs",
sigil => '%',
},
};
print "Scalars begin with a $variables->{'scalar'}->{'sigil'}\n";

##Files and I/O
open (my $passwd, "<", "/etc/passwd2") or die ("can  a not open");
while (<$passwd>) {
print $_ if $_ =~ "test";
}
close $passwd or die "$passwd: $!";
my $next = "doing a first";
$next =~ s/first/second/;
print $next."\n";

my $email = "testaccount\@doxer.org";
if ($email =~ /([^@]+)@(.+)/) {
print "Username is : $1\n";
print "Hostname is : $2\n";
}

##Subroutines
sub multiply{
my ($num1, $num2) = @_;
my $result = $num1 * $num2;
return $result;
}

my $result2 = multiply(3, 5);
print "3 * 5 = $result2\n";

##or
! system('date') or die("failed it"); #if a subroutine returns ok, it'll return 0
PS:
Categories: IT Architecture, Perl, Programming Tags:

resolved – sudo: sorry, you must have a tty to run sudo

April 1st, 2014 5 comments

The error message below sometimes will occur when you run a sudo <command>:

sudo: sorry, you must have a tty to run sudo

To resolve this, you may comment out "Defaults requiretty" in /etc/sudoers(revoked by running visudo). Here is more info about this method.

However, sometimes it's not convenient or even not possible to modify /etc/sudoers, then you can consider the following:

echo -e "<password>\n"|sudo -S <sudo command>

For -S parameter of sudo, you may refer to sudo man page:

-S' The -S (stdin) option causes sudo to read the password from the standard input instead of the terminal device. The password must be followed by a newline character.

So here -S bypass tty(terminal device) to read the password from the standard input. And by this, we can now pipe password to sudo.

PS:

From comments, you may also try below:

1. Comment out Defaults requiretty in /etc/sudoers

2. Defaults:[username] !requiretty #change [username]

3. You can use ssh -t to force pseudo-tty allocation. e.g. ssh -t user1@hostname1 "sudo df -h"

Resolved – print() on closed filehandle $fh at ./perl.pl line 6.

March 19th, 2014 Comments off

You may find that print sometimes won't work as expected in perl, for example:

[root@centos-doxer test]# cat perl.pl
#!/usr/bin/perl
use warnings;
open($fh,"test.txt");
select $fh;
close $fh;
print "test";

You may expect "test" to be printed, but actually you got error message:

print() on closed filehandle $fh at ./perl.pl line 6.

So how's this happened? Please see my explanation:

[root@centos-doxer test]# cat perl.pl
#!/usr/bin/perl
use warnings;
open($fh,"test.txt");
select $fh;
close $fh; #here you closed $fh filehandle, but you should now reset filehandle to STDOUT
print "test";

Now here's the updated script:

#!/usr/bin/perl
use warnings;
open($fh,"test.txt");
select $fh;
close $fh;
select STDOUT;
print "test";

This way, you'll get "test" as expected!

 

Categories: IT Architecture, Perl, Programming Tags:

Oracle BI Publisher reports – send mail when filesystems getting full

March 17th, 2014 Comments off

Let's assume you have one Oracle BI Publisher report for filesystem checking. And now you want to write script for checking that report page and send mail to system admins when filesystems are getting full. As the default output of Oracle BI Publisher report needs javascript to work, and as you may know javascript is evil that wget/curl can not get them, so after log on, the next step you need to do is to find the html version's url of that report for you to use in your script(and the html page has all records when javascript one has only part of them):

BI_report_login

BI_export_html

 

Let's assume that the html's url is "http://www.example.com:9703/report.html", and the display of it was like the following:

bi report

Then here goes the script that will check this page for hosts that has less than 10% available space and send mail to system admins:

#!/usr/bin/perl
use HTML::Strip;
#hosts that do not need reporting
my @remove_list = qw(host1.example.com host2.example.com);
system("rm -f spacereport.html");
system("wget -q --no-proxy --no-check-certificate --post-data 'id=admin&passwd=password' 'http://www.example.com:9703/report.html' -O spacereport.html");
open($fh,"spacereport.html");

#or just @spacereport=<$fh>;
foreach(<$fh>){
push(@spacereport,$_);
}

#change array to hash
$index=0;
map {$pos{$index++}=$_} @spacereport;

#get location of <table> and </table>
#sort numerically ascending
for $char (sort {$a<=>$b} (keys %pos))
{
if($pos{$char} =~ /<table class="c27">/)
{
$table_start=$char;
}

if($pos{$char} =~ /<\/table>/)
{
$table_end=$char;
}

}

#get contents between <table> and </table>
for($i=$table_start;$i<=$table_end;$i++){
push(@table_array,$spacereport[$i]);
}

$table_htmlstr=join("",@table_array);

#get clear text between <table> and </table>
my $hs=HTML::Strip->new();
my $clean_text = $hs->parse($table_htmlstr);
$hs->eof;

@array_filtered=split("\n",$clean_text);

#remove empty array element
@array_filtered=grep { !/^\s+$/ } @array_filtered;

#remove entries from showing
$remove_list_s=join('|',@remove_list);
@index_all = grep { $array_filtered[$_] =~ /$remove_list_s/ } 0..$#array_filtered;

for($i=0;$i<=$#index_all;$i++) {
@index_all_one = grep { $array_filtered[$_] =~ /$remove_list_s/ } 0..$#array_filtered;
splice(@array_filtered,$index_all_one[0],4);
}

system("rm -f space_mail_warning.txt");
open($fh_mail_warning,">","space_mail_warning.txt");
select $fh_mail_warning;
for($j=4;$j<=$#array_filtered;$j=$j+4){
#put lines that has free space lower than 10% to space_mail_warning.txt
if($array_filtered[$j+2] <= 10){
print "Host: ".$array_filtered[$j]."\n";
print "Part: ".$array_filtered[$j+1]."\n";
print "Free(%): ".$array_filtered[$j+2]."\n";
print "Free(GB): ".$array_filtered[$j+3]."\n";
print "============\n\n";
}
}
close $fh_mail_warning;

system("rm -f space_mail_info.txt");
open($fh_mail_info,">","space_mail_info.txt");
select $fh_mail_info;
for($j=4;$j<=$#array_filtered;$j=$j+4){
#put lines that has free space lower than 15% to space_mail_info.txt
if($array_filtered[$j+2] <= 15){
print "Host: ".$array_filtered[$j]."\n";
print "Part: ".$array_filtered[$j+1]."\n";
print "Free(%): ".$array_filtered[$j+2]."\n";
print "Free(GB): ".$array_filtered[$j+3]."\n";
print "============\n\n";
}
}
close $fh_mail_info;

#send mail
#select STDOUT;
if(-s "space_mail_warning.txt"){
system('cat space_mail_warning.txt | /bin/mailx -s "Space Warning - please work with component owners to free space" sysadmins@example.com');
} elsif(-s "space_mail_info.txt"){
system('cat space_mail_info.txt | /bin/mailx -s "Space Info - Space checking mail" sysadmins@example.com');
}

Categories: IT Architecture, Perl, Programming Tags:

wget and curl tips

March 14th, 2014 Comments off

Imagine you want to download all files under http://www.example.com/2013/downloads, and not files under http://www.example.com/2013 except for directory 'downloads', then you can do this:

wget -r --level 100 -nd --no-proxy --no-parent --reject "index.htm*" --reject "*gif" 'http://www.example.com/2013/downloads/' #--level 100 is large enough, as I've seen no site has more than 100 levels of sub-directories so far.

wget -p -k --no-proxy --no-check-certificate --post-data 'id=username&passwd=password' <url> -O output.html

wget --no-proxy --no-check-certificate --save-cookies cookies.txt <url>

wget --no-proxy --no-check-certificate --load-cookies cookies.txt <url>

curl -k -u 'username:password' <url>

curl -k -L -d id=username -d passwd=password <url>

curl --data "loginform:id=username&loginform:passwd=password" -k -L <url>

curl -i -u username:password -H X-Oracle-UserId:myname@example.com -H X-Oracle-IdentityDomain:domainname -X GET "https://login.example.com:443/api/v1/users?userLogin"

Here's one curl example to get SSL certs info on LTM:

#!/bin/bash
path="/var/tmp"
path_root="/var/tmp"

agent="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.2)"

curl -v -L -k -A "$agent" -c ${path}/cookie "https://ltm-url/tmui/login.jsp?msgcode=1&"

curl -v -L -k -A "$agent" -b ${path}/cookie -c ${path}/cookie -e "https://ltm-url/tmui/login.jsp?msgcode=1&" -d "username=myusername&passwd=mypassword" "https://ltm-url/tmui/logmein.html?msgcode=1&"

curl -v -L -k -A "$agent" -b ${path}/cookie -c ${path}/cookie -o ${path_root}/certs-env.html "https://ltm-url/tmui/Control/jspmap/tmui/locallb/ssl_certificate/list.jsp?&startListIndex=0&showAll=true"

Now you can have a check of /var/tmp/certs-env.html for SSL certs info of Big IP VIPs.

 

Oracle VM operations – poweron, poweroff, status, stat -r

January 27th, 2014 Comments off

Here's the script:
#!/usr/bin/perl
#1.OVM must be running before operations
#2.run ovm_vm_operation.pl status before running ovm_vm_operation.pl poweroff or poweron
use Net::SSH::Perl;
$host = $ARGV[0];
$operation = $ARGV[1];
$user = 'root';
$password = 'password';

$newname=$ARGV[2];
$newcpu=$ARGV[3];
$newmemory=$ARGV[4];
$newpool=$ARGV[5];
$newtmpl=$ARGV[6];
$newbridge=$ARGV[7];
$newbridge2=$ARGV[8];
$newvif='vif0';
$newvif2='VIF1';

if($host eq "help") {
print "$0 OVM-name status|poweron|poweroff|reboot|stat-r|stat-r-all|pool|new vmname 1 4096 poolname tmplname FE BE\n";
exit;
}

$ssh = Net::SSH::Perl->new($host);
$ssh->login($user,$password);

if($operation eq "status") {
($stdout,$stderr,$exit) = $ssh->cmd("ovm -uadmin -ppassword vm ls|grep -v VM_test");
open($host_fd,'>',"/var/tmp/${host}.status");
select $host_fd;
print $stdout;
close $host_fd;
} elsif($operation eq "poweroff") {
open($poweroff_fd,'<',"/var/tmp/${host}.status");
foreach(<$poweroff_fd>){
if($_ =~ "Server_Pool|OVM|Powered") {
next;
}
if($_ =~ /(.*?)\s+([0-9]{1,})\s+([0-9]{1,})\s+([0-9]{1,})\s+([a-zA-Z]{1,})\s+(.*)/){
$ssh->cmd("ovm -uadmin -ppassword vm poweroff -n $1 -s $6 -f");
sleep 12;
}
}
} elsif($operation eq "reboot") {
open($poweroff_fd,'<',"/var/tmp/${host}.status");
foreach(<$poweroff_fd>){
if($_ =~ "Server_Pool|OVM|Powered") {
next;
}
if($_ =~ /(.*?)\s+([0-9]{1,})\s+([0-9]{1,})\s+([0-9]{1,})\s+([a-zA-Z]{1,})\s+(.*)/){
$ssh->cmd("ovm -uadmin -ppassword vm reboot -n $1 -s $6");
sleep 12;
}
}
} elsif($operation eq "poweron") {
open($poweron_fd,'<',"/var/tmp/${host}.status");
foreach(<$poweron_fd>){
if($_ =~ "Server_Pool|OVM|Running|used|poweroff") {
next;
}
if($_ =~ /(.*?)\s+([0-9]{1,})\s+([0-9]{1,})\s+([0-9]{1,})\s+([a-zA-Z]{1,})\s+Off(.*)/){
$ssh->cmd("ovm -uadmin -ppassword vm poweron -n $1 -s $6");
#print "ovm -uadmin -ppassword vm poweron -n $1 -s $6";
sleep 15;
}
}
} elsif($operation eq "stat-r") {
open($poweroff_fd,'<',"/var/tmp/${host}.status");
foreach(<$poweroff_fd>){
if($_ =~ /(.*?)\s+([0-9]{1,})\s+([0-9]{1,})\s+([0-9]{1,})\s+(Shutting\sDown|Initializing|Error|Unknown|Rebooting|Deleting)\s+(.*)/){
#print "ovm -uadmin -ppassword vm stat -r -n $1 -s $6";
$ssh->cmd("ovm -uadmin -ppassword vm stat -r -n $1 -s $6");
sleep 1;
}
}
} elsif($operation eq "stat-r-all") {
open($poweroff_fd,'<',"/var/tmp/${host}.status");
foreach(<$poweroff_fd>){
$ssh->cmd("ovm -uadmin -ppassword vm stat -r -n $1 -s $6");
sleep 1;
}
} elsif($operation eq "pool") {
($stdoutp,$stderrp,$exitp) = $ssh->cmd("ovm -uadmin -ppassword svrp ls|grep Inactive");
open($host_fdp,'>',"/var/tmp/${host}-poolstatus");
select $host_fdp;
print $stdoutp;
close $host_fdp;
} elsif($operation eq "new") {
($stdoutp,$stderrp,$exitp) = $ssh->cmd("ovm -uadmin -ppassword tmpl ls -s $newpool | grep $newtmpl");
if($stdoutp =~ /$newtmpl/){
($stdoutp2,$stderrp2,$exitp2) = $ssh->cmd("ovm -uadmin -ppassword vm new -m template -s $newpool -t $newtmpl -n $newname -c password");
if($stdoutp2 =~ /is being created/){
print "Creating VM $newname in pool $newpool on OVMM $host now!"."\n";
while(1){
($stdoutp3,$stderrp3,$exitp3) = $ssh->cmd("ovm -uadmin -ppassword vm stat -n $newname -s $newpool");
if($stdoutp3 =~ /Powered Off/){
print "Done VM creation."."\n";
last;
}
sleep 300
}

print "Setting Cpu/Memory now."."\n";
($stdoutp32,$stderrp32,$exitp32) = $ssh->cmd("ovm -uadmin -ppassword vm conf -n $newname -s $newpool -x $newmemory -m $newmemory -c $newcpu -P");
sleep 2;

print "Creating NICs now."."\n";
($stdoutp4,$stderrp4,$exitp4) = $ssh->cmd("ovm -uadmin -ppassword vm nic conf -n $newname -s $newpool -N $newvif -i VIF0 -b $newbridge");
sleep 2;
($stdoutp5,$stderrp5,$exitp5) = $ssh->cmd("ovm -uadmin -ppassword vm nic add -n $newname -s $newpool -N $newvif2 -b $newbridge2");
sleep 2;

print "Powering on VM now."."\n";
($stdoutp6,$stderrp6,$exitp6) = $ssh->cmd("ovm -uadmin -ppassword vm poweron -n $newname -s $newpool");
sleep 30;

while(1){
($stdoutp7,$stderrp7,$exitp7) = $ssh->cmd("ovm -uadmin -ppassword vm info -n $newname -s $newpool");
if($stdoutp7 =~ /Running on: sl/){
print "VM is now Running, you can configure VM on hypervisor now:"."\n";
print $stdoutp7."\n";
last;
}
sleep 30;
}

#($stdoutp8,$stderrp8,$exitp8) = $ssh->cmd("ovm -uadmin -ppassword vm ls -l | grep $newname");
#print "You can configure VM on hypervisor now:"."\n";
#print $stdoutp8."\n";
} else {
print $stdoutp2."\n";
exit;
}
} else {
print "No template named $newtmpl in pool $newpool\n";
exit;
}
}

You can use the following to make the script run in parallel:

for i in <all OVMs>;do (./ovm_vm_operation.pl $i status &);done

avoid putty ssh connection sever or disconnect

January 17th, 2014 2 comments

After sometime, ssh will disconnect itself. If you want to avoid this, you can try run the following command:

while [ 1 ];do echo hi;sleep 60;done &

This will print message "hi" every 60 seconds on the standard output.

PS:

You can also set some parameters in /etc/ssh/sshd_config, you can refer to http://www.doxer.org/make-ssh-on-linux-not-to-disconnect-after-some-certain-time/

install java jdk on linux

January 7th, 2014 Comments off

Here's the steps if you want to install java on linux:

wget <path to jre-7u25-linux-x64.rpm> -P /tmp
rpm -ivh /tmp/jre-7u25-linux-x64.rpm
mkdir -p /root/.mozilla/plugins
rm -f /root/.mozilla/plugins/libnpjp2.so
ln -s /usr/java/jre1.7.0_25/lib/amd64/libnpjp2.so /root/.mozilla/plugins/libnpjp2.so
ll /root/.mozilla/plugins/libnpjp2.so

PS:

  • You'll need to install jre i386 version if your firefox browser is 32 bits. And you can install jre6 from here. You should download packages like "jre-6u33-linux-i586-rpm.bin" and run chmod +x jre-6u33-linux-i586-rpm.bin && ./jre-6u33-linux-i586-rpm.bin after that. You may locate /usr/java/jre1.6.0_33/bin/javaws for opening remote console when prompted.
  • If you want to install java plugin of firefox on linux, or even install firefox under linux, then you can refer to this article.

self defined timeout for telnet on Linux

December 26th, 2013 Comments off

telnet's default timeout value is relative high, so you may want to change timeout value to lower value such as 5 seconds. Here's the way that we can fulfill this:

#!/bin/bash

timeout()
{
waitfor=5
command=$*
$command &
commandpid=$!
( sleep $waitfor ; kill -9 $commandpid > /dev/null 2>&1 ) &
watchdog=$!
sleeppid=$PPID
wait $commandpid > /dev/null 2>&1
kill $sleeppid > /dev/null 2>&1
}

timeout telnet slcc29-scan1.us.oracle.com 1521 >> $output

Also, we can use expect and set timeout for expect. When telnet is integrated with expect, we can fulfill timeout for telnet through using expect's timeout value:

#!/usr/bin/expect

set timeout 30

send "<put telnet command here>\r"

remove duplicate images using fdupes and expect in linux

December 13th, 2013 Comments off

I've got several thousands of pictures, but most of them had several exact copies of themselves. So I had to remove duplicate ones by hand firstly.

Later, I thought of that in linux we had md5sum which will give the same string for files with exact same contents. Then I tried to write some program, and that toke me some while.

I searched google and found that in linux, we had fdupes which can do the job very well. fdupes will calculate duplicate files based on file size/md5 value, and will prompt you to reserve one copy or all copies of the duplicates and remove others if you gave -d parameter to it. You can read more about fdupes here http://linux.die.net/man/1/fdupes

As all the pictures were on a windows machine, so I installed cygwin and installed fdupes and expect. Later I wrote a small script to reserve only one copy of the duplicate pictures for me(you will have to enter your option either reserving one copy or all copies by hand if you do not use expect, as there's no option for reserve one copy by the author of fdupes). Here's my program:

$ cat fdupes.expect
#!/usr/bin/expect
set timeout 1000000
spawn /home/andy/fdupes.sh
expect "preserve files" {
send "1\r";exp_continue
}

$ cat /home/andy/fdupes.sh
fdupes.exe -d /cygdrive/d/pictures #yup, my pictures are all on this directory on windows, i.e. d:\pictures

After this, you can just run fdupes.expect, and it will reserve only one copy and remove other duplicates for you.

PS: Here's man page of fdupes https://github.com/adrianlopezroche/fdupes

quick configuration of python httpd server

November 28th, 2013 Comments off

Let's assume that you want to copy files from server A to server B, and you have found that no scp available, but wget is there for use. Then you can try run one python command and use wget to download files from server A.

Here's the steps:

On server A:

cd <directory of files you want to copy>

python -m SimpleHTTPServer 8080 #notice the output of this command, for example, "Serving HTTP on 0.0.0.0 port 8080 ..."

Now you can open browser and visit http://<hostname of server A>:8080. You will notice files are there now.

On server B:

wget http://<hostname of server A>:8080/<files to copy>

After you've copied files, you can press control+c to terminate that python http server on Server A. (Or you can press ctrl+z, and then %<job id> & to make that python httpd server run in background)

resolved – how to show all results in one page when searching your wordpress blog

November 13th, 2013 Comments off

Assume that you have your own wordpress blog, and you note down everything you met in daily work.

Now you have some trouble again in work, and remembered that you've noted similar issue before. So you tried searching your wordpress blog with a keyword such as "trouble". Later, wordpress returned a result of 30 pages, each page had 10 articles. Now you scrolled and click "next page" a lot and that really frustrated you. What if you have all the searching result in one page? Thus you just need scroll the page and no waiting for loading pages of next, next, next page. (You may worry that the page load time will disappoint other guys searching your blog, but this proves to be little to worry, as no body will search your blog except yourself. Believe me buddy!)

Here goes the way to fulfill this functionality:

  1. Go to wordpress admin page, then click "Appearance" -> "Editor";
  2. Click archive.php in the right to edit this file(search.php refers to archive.php, so you should edit archive.php);
  3. Search for "have_posts()", and add one line above that line. The line to be added is like this: <?php query_posts($query_string . '&showposts=30'); ?> You may change 30 here to any number you want. As you guessed, this is the number that will show after searching.
  4. Save the change and try searching again. You'll notice the change.

PS:

  1. Note that every time you upgrade wordpress or your wordpress theme you may need to do above steps again;
  2. The idea is from http://wordpress.org/support/topic/show-all-content-on-search-page

make tee to copy stdin as well as stderr & prevent ESC output of script

October 30th, 2013 Comments off
  • Make tee to copy stdin as well as stderr

As said by manpage of tee:

read from standard input and write to standard output and files

So if you have error messages in your script, then the error messages will not copied and write to file.

Here's one workaround for this:

./aaa.sh 2>&1 | tee -a log

Or you can use the more complicated one:

command > >(tee stdout.log) 2> >(tee stderr.log >&2)

  • Prevent ESC output of script

script literally captures every type of output that was sent to the screen. If you have colored or bold output, this shows up as esc characters within the output file. These characters can significantly clutter the output and are not usually useful. If you set the TERM environmental variable to dumb (using setenv TERM dumb for csh-based shells and export TERM=dumb for sh-based shells), applications will not output the escape characters. This provides a more readable output.

In addition, the timing information provided by script clutters the output. Although it can be useful to have automatically generated timing information, it may be easier to not use script’s timing, and instead just time the important commands with the time command mentioned in the previous chapter.

PS:

  1. Here's the full version http://stackoverflow.com/questions/692000/how-do-i-write-stderr-to-a-file-while-using-tee-with-a-pipe
  2. Some contents of this article is excerpted from <Optimizing Linux® Performance: A Hands-On Guide to Linux® Performance Tools>.

use batch script to start up & shutdown Virtualbox VMs

October 28th, 2013 Comments off

I woke up before 8 every day on weekdays, and want to poweron two VMs in virtualbox named "xp" and "win2008". So I can write a script and put it in "startup" folder, then these two VMs will startup with system automatically:

@echo off
date /t | find "Mon" && goto 1
date /t | find "Tue" && goto 1
date /t | find "Wed" && goto 1
date /t | find "Thu" && goto 1
date /t | find "Fri" && goto 1
exit

:1
if %time:~0,2% leq 8 (
c:\VirtualBox\VBoxManage startvm win2008 --type gui
c:\VirtualBox\VBoxManage startvm xp --type gui
) else exit

And I also want to shutdown these two VMs in one run:

c:\VirtualBox\VBoxManage controlvm win2008 acpipowerbutton
c:\VirtualBox\VBoxManage controlvm xp acpipowerbutton

PS:

You may also consider group policy(gpedit.msc -> Computer Configuration -> Windows Settings -> Scripts -> Shutdown) in windows so that when you shutdown your pc, all VMs will turned off automatically if you have a GPO for shutdown.

bash & expect tips

August 19th, 2013 Comments off


/u02/0-sync.sh

#!/usr/bin/expect
set timeout 10800
spawn /u02/sync.sh
expect "*assword*" {send "test\r";exp_continue
expect "*#*" {close}
}

/u02/sync.sh

#! /bin/sh
logpathroot=/u02/logs
synclist=/u02/sync.list

num=1
allnum=`cat $synclist | wc -l`
while [ $num -le $allnum ];
do
line=`awk '{if(NR=='$num') print}' $synclist`
component=`echo $line | awk '{print $2}'`
logtype=`echo $line | awk '{print $3}'`
host=`echo $line | awk '{print $4}'`
source=`echo $line | awk '{print $5}'`
env=`echo $line | awk '{print $1}'`
if [[ "$env" =~ "#" ]]; then
env=`echo $env | sed "s/#//"`
lnenv=`ls /u01/logs | grep -i "$env"`
exist=`ls "/u01/logs/${lnenv}/${component}_${logtype}-$host"`
if [ "$exist" != "" ]; then
rm -f /u01/logs/${lnenv}/${component}_${logtype}-$host
fi
else
lnenv=`ls /u01/logs | grep -i "$env"`
/bin/mkdir -p "${logpathroot}/${env}/${component}/${logtype}-$host"

#rsync -avz --exclude "aime/work/CLOUDTOP/*" /scratch/ /scratch_new/
/usr/bin/rsync -ave ssh --exclude '.zfs' --delete-excluded logs@${host}:${source}/ ${logpathroot}/${env}/${component}/${logtype}-$host
#chmod 755 -R "${logpathroot}/${env}/${component}/"
exist=`ls "/u01/logs/${lnenv}/${component}_${logtype}-$host"`
if [ "$exist" = "" ]; then
# rm -f /u01/logs/${lnenv}/${component}_${logtype}-$host
component1=`echo $component | tr "[:upper:]" "[:lower:]"`
logtype1=`echo $logtype | sed "s/-/_/"`
ln -sf ${logpathroot}/${env}/${component}/${logtype}-$host/ /u01/logs/${lnenv}/${component1}_${host}_${logtype1}_log
fi
fi
num=`expr $num + 1`
done

rm -f /u02/sync_at_*
echo > /u02/sync_at_"`date +%Y-%m-%d` `date +%H:%M` (UTC)"

/u02/sync.list

DC1 apex AdminServer_adr test27vmf4053 /u01/local/config/m_domains/base_domain/servers/AdminServer/adr
DC1 emgc EMGC_ADMINSERVER_adr test27cn05 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/adr
DC1 emgc EMGC_OMS1_adr test27cn05 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr
DC1 emgc EMGC_OMS2_adr test27cn06 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS2/adr
#DC1 emgc EMGC_OMS3_adr test27cn07 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS3/adr

PS:

  • If you want to match empty space in expect, you can use the ascii code of space which is 040, and expect -re "\040" would work.
  • About special characters in tcl/expect/shell:

Some characters have special meanings in Tcl or shells (e.g., sh, tcsh, etc.). In particular, these characters may provide security holes for shells.
The characters listed here are escaped by preceeding each one with a backslash (\).
The following characters are dissallowed or have special meanings in Tcl and so are escaped: &;`'"|*?~<>^()[]{}$\-
The following characters are dissallowed or have special meanings in command shells and so are escaped: &;`'"|*?~<>^()[]{}$\\n
The only difference between the two tables is the addition of \n to the shell escape table.

  • To execute multiple commands in one run, you can do below:
[root@vm1 ~]# cat conf.andy
sudo su -
echo $UID >/tmp/e
for i in {1..10}
do
touch /tmp/$i.txt
done
ls /tmp/

[root@vm1 ~]# cat /bin/autoexcute.andy
#!/usr/bin/expect --
set timeout 60
set user [lindex $argv 0]
set ip [lindex $argv 1]
set password [lindex $argv 2]
set com [lindex $argv 3]

spawn ssh -t -o "StrictHostKeyChecking no" $user@$ip
expect {
  "assword:" {
    send "$password\r"
    expect -re "(\\\$|\\\$\040)$" {
        send "/usr/bin/sudo su -\r"
        sleep 2
        expect -re "(\->\040|#|#\040)$" {
            send "$com\r"
            sleep 1
            expect -re "(\->\040|#|#\040)$" {close}
        }
    }
  }
}

[root@vm1 ~]# /bin/autoexcute.andy root testvm password1 "`cat conf.andy`"
spawn ssh -t -o StrictHostKeyChecking no root@testvm
root@testvm's password:
Last login: Tue Sep 15 19:06:49 2015 from vm1.us.oracle.com
-bash-4.1$ /usr/bin/sudo su -
[root@testvm-luis ~]# sudo su -
[root@testvm-luis ~]# echo $UID >/tmp/e
[root@testvm-luis ~]# for i in {1..10}
> do
> touch /tmp/$i.txt
> done
[root@testvm-luis ~]# ls /tmp/
1.txt  
10.txt 
2.txt  
3.txt  
4.txt  
5.txt  
6.txt  
7.txt  
8.txt  
9.txt  
  • Here's something about expect_out in tcp/expect:

expect_feedback_01

expect_feedback_02

quick configuration of httpd /var/tmp

July 8th, 2013 Comments off

yum -y install httpd

vi /etc/httpd/conf/httpd.conf

Alias /test /var/tmp
<Directory /var/tmp>
Options +Indexes
Order allow,deny
Allow from all
</Directory>

apachectl restart

Now copy files to /var/tmp and visit http://<ip address>/test

PS:

1. If you the installation of httpd server is not possibe, you can try the easier way of python httpd server here quick configuration of python httpd server.

2. If you want to share /var/tmp so that others can mount it as NFS share, you can do the following:

echo "/var/tmp *(rw)" >> /etc/exports

exportfs -a

/etc/init.d/nfs start

Then on the host you want to mount the NFS share, do the following:

mount -t nfs slc03nsv:/var/tmp /mnt

3. Want to write .htaccess for 301 redirect? Then you can refer to Three types of 301 rewrite/redirect for apache httpd server and apache rewrite with and without ending slash.

4. Here's a guide for nginx installation with a bash script: Install nginx reverse proxy server under centos(bash shell) and the way to get the real user ip address when using nginx as reverse proxy(from backend apache server) & logrotate linux log files.

5. Now your boss asked you to have a monitor of enterprise's website, than you may find the following article useful - Auto-monitor your server status and restart or reboot server(bash shell script) and awstats installation and configuration guide on linux centos.

6. Your httpd server is now under DDos Attack? I've undergone this, you can read the stories resolved – port 53 dns flooding attack.

7. You'll be sure to met some errors when you handled linux httpd server a lot, here're several of them Resolved – ld.so.1: httpd: fatal: libaprutil-0.so.0: open failed: No such file or directory & Solved:Cannot load /etc/httpd/modules/mod_auth_mysql.so into server: libmysqlclient.so.15: cannot open shared object file: No such file or directory & resolved awstats 500 internal server error after installation on centos linux.

Categories: IT Architecture, Linux, Python, Systems Tags:

parallel execution of script using perl fork()

July 3rd, 2013 Comments off

Let's assume that we want to test 100 hosts to see whether they're pingable or not. You may think of bash style for ... do ... done loop, but as that is executed one by one, so it's not efficient. And in this article, I'm going to create parallel execution of this task using perl's fork().

Firstly, here's one script to get whether one host is pingable or not:

[root@centos-doxer pping_doxer]# cat script.sh
#!/usr/bin/expect
log_user 0
match_max -d 2000000
set host [lindex [lrange $argv 0 0] 0]
spawn -noecho ping $host
set timeout 2
expect "bytes from" {exit 0} default {puts "not pingable from $host";exit 1}

Let's run it and see the result:

[root@centos-doxer pping_doxer]# ./script.sh test-host
not pingable from test-host

So now, let's write one perl script to wrapper this expect script, and also add parallel run to it:

#!/usr/bin/perl
our @hosts=("testhost0001","testhost00022","testhost00033","testhost0007");
#you can read a file to perl array too
#open($fh,"/doxer/projects/pping_doxer/servers.txt");
#our @hosts=<$fh>;
our $zombies = 0;
our $kid_proc_num = 0;

$SIG{CHLD} = sub { $zombies++ };
for(my $i=0; $i<@hosts; $i++) {
my $pid = fork();
if( !defined($pid) ) { exit 1; }
unless($pid) {
system("/doxer/projects/pping_doxer/script.sh $hosts[$i]"); #change the path according to your env
exit 0;
}
$kid_proc_num++;
}

while (1) {
if($zombies > 0) {
$zombies = 0;
my $collect;
while(($collect = waitpid(-1, WNOHANG)) > 0) {
$kid_proc_num--;
}
}
if($kid_proc_num==0) { last; }
else { next; }
}

And now let's have a test:

[root@centos-doxer pping_doxer]# ./pping_doxer
not pingable from testhost0001
not pingable from testhost00033
not pingable from testhost0007
not pingable from testhost00022

You can modify script.sh to fulfill your own aim except for ping of course.

That's it, enjoy!

PS:

1.I've wrote an updated version of pping_doxer, which allows you to specify a maximum number of parallel running processes and also puts hosts in a file. I've put the package in http://www.doxer.org/projects-doxer/, the package name is pping_doxer.

2.Thanks very much to http://www.ibm.com/developerworks/cn/linux/l-cn-perlmp/.

Categories: IT Architecture, Perl, Programming Tags: ,

understanding perl fork

July 2nd, 2013 Comments off

Let's go through perl fork() by example:

#!/usr/bin/perl -w
use strict;

defined(my $pid=fork()) or die "Fork process failured:$!\n";  #two processes are running now, one parent and child process($pid is zero)
unless($pid)
{
# This is the child process.
system "date";
sleep(3);
print ("Exit child after 3 seconds wait!\n");
exit();
}
# This is the parent process.
waitpid($pid,0);
system "date";
print ("exit parent!\n");

Here's the result:

Tue Jul 2 18:29:33 CST 2013
Exit child after 3 seconds wait!
Tue Jul 2 18:29:36 CST 2013
exit parent!

And now let's test when waitpid() is commented out:

#!/usr/bin/perl -w
use strict;

defined(my $pid=fork()) or die "Fork process failured:$!\n";
unless($pid)
{
# This is the child process.
system "date";
sleep(3);
print ("Exit child after 3 seconds wait!\n");
exit();
}
# This is the parent process.
#waitpid($pid,0);  #parent no longer wait until child process exit
system "date";
print ("exit parent!\n");

And here's the result:

[root@centos-doxer test]# ./perl.test
Tue Jul 2 18:30:05 CST 2013
Tue Jul 2 18:30:05 CST 2013
exit parent!

Categories: IT Architecture, Perl, Programming Tags:

regular expression tips

July 1st, 2013 Comments off
if($reply =˜ m/ˆ[0-9]+$/ig) # i is modifier
$var =˜ s/\bJeff\b/Jeff/i; #replace
$text =˜s/ˆ$/<p>/mg; #m treat string as multiple lines
$/ = ".\n"; #paragraph, string
#greedy

Source:aa<div>test1</div>bb<div>test2</div>cc
Greedy:<div>.*</div>
result:<div>test1</div>bb<div>test2</div>
Non-Greedy:<div>.*?</div>
result<div>test1</div>

 /<(.*?)>.*?<\/\1>/ #inside the

(?:xxx) #?: do not take into variables
file-slurp mode special

word boundaries in egrep

June 17th, 2013 Comments off

Here's one example:

[root@doxer_#1]# cat a
mcat me
here cat is
ok concaten

[root@doxer_#1]# egrep '\<cat\>' a
here cat is

[root@doxer_#1]# egrep '\<cat' a
here cat is

And here goes the details:

word-boundaries

perl tips – most useful perl command line options switches

June 9th, 2013 Comments off

Here's all perl command line options, red ones are the most useful ones from me:

[root@centos-doxer test]# perl --help
Usage: perl [switches] [--] [programfile] [arguments]

-c check syntax only (runs BEGIN and CHECK blocks) #perl -c ./perl.pl

-e program one line of program (several -e's allowed, omit programfile) #e.g. perl -e 'print "hi\n";print "hi again\n";'

-n assume "while (<>) { ... }" loop around program #like -p, but do not print

-p assume loop like -n but print line also, like sed #perl -p -e "tr/[a-z]/[A-Z]/" a.txt

-i[extension] edit <> files in place (makes backup if extension supplied) #perl -pi.bak -e "tr/[a-z]/[A-Z]/" a.txt

-[mM][-]module execute "use/no module..." before executing program #perl -MCPAN -e 'install xxx:xxx'

-l[octal] enable line ending processing, specifies line terminator #man ascii to see all ascii - octal mapping

-F/pattern/ split() pattern for -a switch (//'s are optional)

-a autosplit mode with -n or -p (splits $_ into @F) #perl -n -l072 -a -F/:/ -e 'print $F[0]' a.txt

 

-s enable rudimentary parsing for switches after programfile 

[root@centos-doxer test]# cat perl.test
#!/usr/bin/perl -s
print "yes\n" if defined($r);
print "no\n" unless defined($r);

[root@centos-doxer test]# ./perl.test -a
no
[root@centos-doxer test]# ./perl.test test
no
[root@centos-doxer test]# ./perl.test -r
yes

xm list|awk '{print $1}'|perl -p -e "s/^.*(slce.*?)_.*/\1/g" #will return slce09vm001

ovm vm ls|awk '{print $1}'|perl -p -e "s/^[0-9]{2,}_(.*)/\1/g"|perl -p -e "s/^.*(slc.*?)[_-].*/\1/g"|egrep -v 'wlsaas|virt'|grep slc|sort #will return slce09vm001

-C[number/list] enables the listed Unicode features
-0[octal] specify record separator (\0, if no argument)
-d[:debugger] run program under debugger
-D[number/list] set debugging flags (argument is a bit mask or alphabets)
-f don't do $sitelib/sitecustomize.pl at startup
-Idirectory specify @INC/#include directory (several -I's allowed)
-P run program through C preprocessor before compilation

-S look for programfile using PATH environment variable
-t enable tainting warnings
-T enable tainting checks
-u dump core after parsing program
-U allow unsafe operations
-v print version, subversion (includes VERY IMPORTANT perl info)
-V[:variable] print configuration summary (or a single Config.pm variable)
-w enable many useful warnings (RECOMMENDED)
-W enable all warnings
-x[directory] strip off text before #!perl line and perhaps cd to directory
-X disable all warnings

Categories: IT Architecture, Perl, Programming Tags:

perl – match fixed number of spaces or other characters

June 6th, 2013 Comments off

Sometimes you want to match lines beginning with exactly four spaces("<space><space><space><space>") :

<space><space><space><space><space>Line1

<space><space><space><space>Line2  --only this line!

<space><space>Line3

So here's the perl script to fulfill this:

#!/usr/bin/perl
open($fdme, '/doxer/test/a.txt');
foreach(<$fdme>){
if($_ =~ /^\s{4}\w+.*/){
print $_."\n";
}
}

Categories: IT Architecture, Perl, Programming Tags:

install perl module using tarball or cpan

June 2nd, 2013 Comments off
To install a perl module using cpan:
Method 1:
PERL_MM_USE_DEFAULT=1
export PERL_MM_USE_DEFAULT=1
cpan> o conf prerequisites_policy follow
cpan> o conf build_requires_install_policy yes
cpan> o conf commit
cpan > o conf init /proxy/
cpan> force install Net::SSH::Perl
Method 2:
PERL_MM_USE_DEFAULT=1
perl -MCPAN -e 'install Net::SSH::Perl'
To install a perl module using yum in Linux:
yum install -y "perl(X11::Protocol)"
To manually install a Perl module:
1. Download the Perl module from CPAN or other site.
2. Extract the tarball.
3. Run perl Makefile.PL
4. Run make
5. Run make test
6. Run make install

 

To test whether the module has been installed or not, run this:

perl -e 'use Module::Metadata', if nothing returned, then it's ok

PS:
  • yum install perl-CPAN if you found that cpan was not installed by default with perl
  • Here's help message for cpan:
cpan> help #o conf init to reconfigure cpan
Display Information
 command  argument          description
 a,b,d,m  WORD or /REGEXP/  about authors, bundles, distributions, modules
 i        WORD or /REGEXP/  about anything of above
 r        NONE              reinstall recommendations
 ls       AUTHOR            about files in the author's directory
 
Download, Test, Make, Install...
 get                        download
 make                       make (implies get)
 test      MODULES,         make test (implies make)
 install   DISTS, BUNDLES   make install (implies test)
 clean                      make clean
 look                       open subshell in these dists' directories
 readme                     display these dists' README files
 
Other
 h,?           display this menu       ! perl-code   eval a perl command
 o conf [opt]  set and query options   q             quit the cpan shell
 reload cpan   load CPAN.pm again      reload index  load newer indices
 autobundle    Snapshot                force cmd     unconditionally do cmd
Categories: IT Architecture, Perl, Programming Tags:

resolved – perl get only package/file names and not including package version number using regular expression

May 9th, 2013 Comments off

I got a list of packages installed on my centos linux box via:

rpm -qa|sort|uniq|sort

The result is like:

acl-2.2.39-8.el5
acpid-1.0.4-12.el5
alacarte-0.10.0-1.fc6
alsa-lib-1.0.17-1.el5
alsa-lib-devel-1.0.17-1.el5
amtu-1.0.6-2.el5
anacron-2.3-45.el5.centos
apr-1.2.7-11.el5_6.5
apr-util-1.2.7-11.el5_5.2

But now I only want the package names without version number, that is like:

acl
acpid
alacarte
alsa-lib
alsa-lib-devel
amtu
anacron
apr
apr-util

Here's the perl script to fulfill this:

#!/usr/bin/perl
open($temp1,"<","/doxer/test/package-names.txt") or die("error opening");
while(<$temp1>){
if($_ =~ /([a-zA-Z].*?)\-[0-9]{1,}\.[0-9]{1,}.*/){
print $1."\n";
}
}

Categories: IT Architecture, Perl, Programming Tags:

perl script for getting sun zfs head project and share usage info

April 26th, 2013 Comments off

Sometimes you would like to know on sun zfs head, which project occupies most of the space, and which shares of that occupies most space of that project.

Here's a perl script to fulfill this(it's a little cumbersome, but it works anyway)

#!/usr/bin/perl
use strict;
use warnings;
use Net::SSH::Perl;
use List::Util qw/sum/;
my $host = 'test-zfs-host';
my $user = 'root';
my $password = 'password';

my $ssh = Net::SSH::Perl->new($host);
$ssh->login($user,$password);
my ($stdout,$stderr,$exit) = $ssh->cmd("shares show");
my @std_arr=split(/:/,$stdout);
my @projects_arr = split(/\n/, $std_arr[2]);

foreach(@projects_arr){
$_ =~ s/^\s+|\s+$//g;
}
shift @projects_arr;
pop @projects_arr;
pop @projects_arr;

my @space_projects;
foreach(@projects_arr){
my ($stdout2,$stderr2,$exit2) = $ssh->cmd("shares select $_ get");
my @stdout_arr=split(/\n/,$stdout2);
my $space_temp=join("\n",grep(/space_total/,@stdout_arr));
my @space_total_project = "project ".$_.$space_temp;
push(@space_projects,@space_total_project);
}
my @space_projects3;
foreach(@projects_arr){
my ($stdout3,$stderr3,$exit3) = $ssh->cmd("shares select $_ ls");
my @stdout_arr3=split(/\n/,$stdout3);
my @space_temp3=grep(/\/export\//,@stdout_arr3);
my @space_temp4=grep(!/mountpoint/,@space_temp3);
push(@space_projects3,@space_temp4);
}
open(my $temp1,'>','/var/tmp/temp1') or die("cannot open file temp1");
open(my $temp2,'>','/var/tmp/temp2') or die("cannot open file temp2");
open(my $temp3,'>','/var/tmp/temp3') or die("cannot open file temp3");
open(my $temp4,'>','/var/tmp/temp4') or die("cannot open file temp4");
my @var_T;
my @var_G;
my @var_M;
my @var_K;

foreach(@space_projects){
if($_ =~ /.+space_total\s+(.*)K/){
push(@var_K,$_);
}
elsif($_ =~ /.+space_total\s+(.*)M/){
push(@var_M,$_);
}
elsif($_ =~ /.+space_total\s+(.*)G/){
push(@var_G,$_);
}
elsif($_ =~ /.+space_total\s+(.*)T/){
push(@var_T,$_);
}
}

select $temp1;
foreach(@var_T){
print $_."\n";
}
close $temp1;

select $temp2;
foreach(@var_G){
print $_."\n";
}
close $temp2;
select $temp3;
foreach(@var_M){
print $_."\n";
}
close $temp3;
select $temp4;
foreach(@var_K){
print $_."\n";
}
close $temp4;

system("echo \"======zfs project usage info(Descending)======\"");
system("sort -r -n -k 5 /var/tmp/temp1;sort -r -n -k 5 /var/tmp/temp2;sort -r -n -k 5 /var/tmp/temp3;sort -r -n -k 5 /var/tmp/temp4");
open(my $temp5,'>','/var/tmp/temp5') or die("cannot open file temp5");
open(my $temp6,'>','/var/tmp/temp6') or die("cannot open file temp6");
open(my $temp7,'>','/var/tmp/temp7') or die("cannot open file temp7");
open(my $temp8,'>','/var/tmp/temp8') or die("cannot open file temp8");
my @var_T_2;
my @var_G_2;
my @var_M_2;
my @var_K_2;

foreach(@space_projects3){
if($_ =~ /\s+.*K\s+.*/){
push(@var_K_2,$_);
}
elsif($_ =~ /\s+.*M\s+.*/){
push(@var_M_2,$_);
}
elsif($_ =~ /\s+.*G\s+.*/){
push(@var_G_2,$_);
}
elsif($_ =~ /\s+.*T\s+.*/){
push(@var_T_2,$_);
}
}

select $temp5;
foreach(@var_T_2){
print $_."\n";
}
close $temp5;

select $temp6;
foreach(@var_G_2){
print $_."\n";
}
close $temp6;
select $temp7;
foreach(@var_M_2){
print $_."\n";
}
close $temp7;
select $temp8;
foreach(@var_K_2){
print $_."\n";
}
close $temp8;

system("echo \"\n\n\n======zfs share usage info(Descending)======\"");
system("sort -r -n -k 2 /var/tmp/temp5;sort -r -n -k 2 /var/tmp/temp6;sort -r -n -k 2 /var/tmp/temp7;sort -r -n -k 2 /var/tmp/temp8");

The output would be like:

======zfs project usage info(Descending)======
project DC2_DMZ space_total = 7.68T
project dc2_c9testga space_total = 1.10T
project fa_trialadcf space_total = 277G
project fa_rehydration space_total = 266G
project common space_total = 10.0G
project NODE_8 space_total = 93K
project default space_total = 31K

 

======zfs share usage info(Descending)======
Service_Mid-2 1.44T /export/DC2_DMZ/Service_Mid-2
Service_Web 1.22T /export/DC2_DMZ/Service_Web
dc2_shared_idm 743G /export/DC2_DMZ/dc2_shared_idm
Infra_Web 400G /export/DC2_DMZ/Infra_Web
nuviaq_local02 988M /export/DC2_DMZ/nuviaq_local02
sftp_staging 127M /export/DC2_DMZ/sftp_staging
sftp_local03 14.8M /export/DC2_DMZ/sftp_local03
sftp_manager_local01 85K /export/DC2_DMZ/sftp_manager_local01

GNU Posix Standards Conformance guide – take diffutils for example

January 30th, 2013 Comments off

In a few cases, the GNU utilities' default behavior is incompatible with the POSIX standard. To suppress these incompatibilities, define the POSIXLY_CORRECT environment variable. Unless you are checking for POSIX conformance, you probably do not need to define POSIXLY_CORRECT.

Normally options and operands can appear in any order, and programs act as if all the options appear before any operands. For example, ‘diff lao tzu -C 2’ acts like ‘diff -C 2 lao tzu’, since ‘2’ is an option-argument of -C. However, if the POSIXLY_CORRECT environment variable is set, options must appear before operands, unless otherwise specified for a particular command.

Newer versions of POSIX are occasionally incompatible with older versions. For example, older versions of POSIX allowed the command ‘diff -c -10’ to have the same meaning as ‘diff -C 10’, but POSIX 1003.1-2001 ‘diff’ no longer allows digit-string options like -10.

The GNU utilities normally conform to the version of POSIX that is standard for your system. To cause them to conform to a different version of POSIX, define the _POSIX2_VERSION environment variable to a value of the form yyyymm specifying the year and month the standard was adopted.

Two values are currently supported for _POSIX2_VERSION: ‘199209’ stands for POSIX 1003.2-1992, and ‘200112’ stands for POSIX 1003.1-2001. For example, if you are running older software that assumes an older version of POSIX and uses ‘diff -c -10’, you can work around the compatibility problems by setting ‘_POSIX2_VERSION=199209’ in your environment.

Categories: IT Architecture, Programming, SHELL Tags:

a perl script to get all projects space usage in sun zfs heads

January 23rd, 2013 Comments off

#!/usr/bin/perl
use strict;
use warnings;
use Net::SSH::Perl;
use List::Util qw/sum/;
my $host = 'zfs-host';
my $user = 'root';
my $password = 'test';

my $ssh = Net::SSH::Perl->new($host);
$ssh->login($user,$password);
my ($stdout,$stderr,$exit) = $ssh->cmd("shares show");
my @std_arr=split(/:/,$stdout);
my @projects_arr = split(/\n/, $std_arr[2]);

foreach(@projects_arr){
$_ =~ s/^\s+|\s+$//g;
}
shift @projects_arr;
pop @projects_arr;
pop @projects_arr;

my @space_projects;
foreach(@projects_arr){
my ($stdout2,$stderr2,$exit2) = $ssh->cmd("shares select $_ get");
my @stdout_arr=split(/\n/,$stdout2);
my @space_total_project = grep(/space_total/,@stdout_arr);
push(@space_projects,@space_total_project);
}

foreach(@space_projects){
if ($_ =~ /.+=\s+(.+)M/){
$_=0;
}
elsif($_ =~ /.+=\s+(.+)G/){
$_=$1/1024;
}
elsif($_ =~ /.+=\s+(.+)K/){
$_=0;
}
elsif($_ =~ /.+=\s+(.+)T/){
$_=$1;
}
}
my $space_used_sum=sum(@space_projects);

print $space_used_sum."\n";

Categories: IT Architecture, Perl, Programming Tags: