resolved - sudo: sorry, you must have a tty to run sudo

The error message below sometimes will occur when you run a sudo <command>:

sudo: sorry, you must have a tty to run sudo

To resolve this, you may comment out "Defaults requiretty" in /etc/sudoers(revoked by running visudo). Here is more info about this method.

However, sometimes it's not convenient or even not possible to modify /etc/sudoers, then you can consider the following:

echo -e "<password>\n"|sudo -S <sudo command>

For -S parameter of sudo, you may refer to sudo man page:

-S' The -S (stdin) option causes sudo to read the password from the standard input instead of the terminal device. The password must be followed by a newline character.

So here -S bypass tty(terminal device) to read the password from the standard input. And by this, we can now pipe password to sudo.

PS:

From comments, you may also try below:

1. Comment out Defaults requiretty in /etc/sudoers

2. Defaults:[username] !requiretty #change [username]

3. You can use ssh -t to force pseudo-tty allocation. e.g. ssh -t user1@hostname1 "sudo df -h"

4. If you met error "PTY allocation request failed on channel 0" when SSH, then you can increase pty number

sysctl -a|grep -i pty

kernel.pty.max = 4096
kernel.pty.nr = 237

vi /etc/sysctl.conf #kernel.pty.max = 10000

sysctl -p;sysctl -a|grep pty

wget and curl tips

Imagine you want to download all files under http://www.example.com/2013/downloads, and not files under http://www.example.com/2013 except for directory 'downloads', then you can do this:

wget -r --level 100 -nd --no-proxy --no-parent --reject "index.htm*" --reject "*gif" 'http://www.example.com/2013/downloads/' #--level 100 is large enough, as I've seen no site has more than 100 levels of sub-directories so far.

wget -p -k --no-proxy --no-check-certificate --post-data 'id=username&passwd=password' <url> -O output.html

wget --no-proxy --no-check-certificate --save-cookies cookies.txt <url>

wget --no-proxy --no-check-certificate --load-cookies cookies.txt <url>

curl -k -u 'username:password' <url>

curl -k -L -d id=username -d passwd=password <url>

curl --data "loginform:id=username&loginform:passwd=password" -k -L <url>

curl -i -u username:password -H X-Oracle-UserId:myname@example.com -H X-Oracle-IdentityDomain:domainname -X GET "https://login.example.com:443/api/v1/users?userLogin"

Here's one curl example to get SSL certs info on LTM:

#!/bin/bash
path="/var/tmp"
path_root="/var/tmp"

agent="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.2)"

curl -v -L -k -A "$agent" -c ${path}/cookie "https://ltm-url/tmui/login.jsp?msgcode=1&"

curl -v -L -k -A "$agent" -b ${path}/cookie -c ${path}/cookie -e "https://ltm-url/tmui/login.jsp?msgcode=1&" -d "username=myusername&passwd=mypassword" "https://ltm-url/tmui/logmein.html?msgcode=1&"

curl -v -L -k -A "$agent" -b ${path}/cookie -c ${path}/cookie -o ${path_root}/certs-env.html "https://ltm-url/tmui/Control/jspmap/tmui/locallb/ssl_certificate/list.jsp?&startListIndex=0&showAll=true"

Now you can have a check of /var/tmp/certs-env.html for SSL certs info of Big IP VIPs.

PS:

To use private CA/public cert/private key, you should use below in curl:

curl -v --cacert your-root-ca.crt --cert your-public-cert.crt --key your-private.key --pass mypass -u "username:password" https://url
In this command, your-public-cert.crt is the public cert that you have trusted,your-private.key is the private RSA key portion of the cert that is used to sign the request, and “username:password” should be replaced with the correct username and password.

Also, if you’re using an intermediate cert, you can provide it in one command like so:

curl -v --cacert your-root-ca.crt --cert <(cat your-public-cert.crt  intermediate.crt ) --key your-private.key --pass mypass -u “username:password" https://url

avoid putty ssh connection sever or disconnect

After sometime, ssh will disconnect itself. If you want to avoid this, you can try run the following command:

while [ 1 ];do echo hi;sleep 60;done &

This will print message "hi" every 60 seconds on the standard output.

PS:

You can also set some parameters in /etc/ssh/sshd_config, you can refer to http://www.doxer.org/make-ssh-on-linux-not-to-disconnect-after-some-certain-time/

self defined timeout for telnet on Linux

telnet's default timeout value is relative high, so you may want to change timeout value to lower value such as 5 seconds. Here's the way that we can fulfill this:

#!/bin/bash

timeout()
{
waitfor=5
command=$*
$command &
commandpid=$!
( sleep $waitfor ; kill -9 $commandpid > /dev/null 2>&1 ) &
watchdog=$!
sleeppid=$PPID
wait $commandpid > /dev/null 2>&1
kill $sleeppid > /dev/null 2>&1
}

timeout telnet slcc29-scan1.us.oracle.com 1521 >> $output

Also, we can use expect and set timeout for expect. When telnet is integrated with expect, we can fulfill timeout for telnet through using expect's timeout value:

#!/usr/bin/expect

set timeout 30

send "<put telnet command here>\r"

remove duplicate images using fdupes and expect in linux

I've got several thousands of pictures, but most of them had several exact copies of themselves. So I had to remove duplicate ones by hand firstly.

Later, I thought of that in linux we had md5sum which will give the same string for files with exact same contents. Then I tried to write some program, and that toke me some while.

I searched google and found that in linux, we had fdupes which can do the job very well. fdupes will calculate duplicate files based on file size/md5 value, and will prompt you to reserve one copy or all copies of the duplicates and remove others if you gave -d parameter to it. You can read more about fdupes here http://linux.die.net/man/1/fdupes

As all the pictures were on a windows machine, so I installed cygwin and installed fdupes and expect. Later I wrote a small script to reserve only one copy of the duplicate pictures for me(you will have to enter your option either reserving one copy or all copies by hand if you do not use expect, as there's no option for reserve one copy by the author of fdupes). Here's my program:

$ cat fdupes.expect
#!/usr/bin/expect
set timeout 1000000
spawn /home/andy/fdupes.sh
expect "preserve files" {
send "1\r";exp_continue
}

$ cat /home/andy/fdupes.sh
fdupes.exe -d /cygdrive/d/pictures #yup, my pictures are all on this directory on windows, i.e. d:\pictures

After this, you can just run fdupes.expect, and it will reserve only one copy and remove other duplicates for you.

PS: Here's man page of fdupes https://github.com/adrianlopezroche/fdupes

make tee to copy stdin as well as stderr & prevent ESC output of script

  • Make tee to copy stdin as well as stderr

As said by manpage of tee:

read from standard input and write to standard output and files

So if you have error messages in your script, then the error messages will not copied and write to file.

Here's one workaround for this:

./aaa.sh 2>&1 | tee -a log

Or you can use the more complicated one:

command > >(tee stdout.log) 2> >(tee stderr.log >&2)

  • Prevent ESC output of script

script literally captures every type of output that was sent to the screen. If you have colored or bold output, this shows up as esc characters within the output file. These characters can significantly clutter the output and are not usually useful. If you set the TERM environmental variable to dumb (using setenv TERM dumb for csh-based shells and export TERM=dumb for sh-based shells), applications will not output the escape characters. This provides a more readable output.

In addition, the timing information provided by script clutters the output. Although it can be useful to have automatically generated timing information, it may be easier to not use script’s timing, and instead just time the important commands with the time command mentioned in the previous chapter.

PS:

  1. Here's the full version http://stackoverflow.com/questions/692000/how-do-i-write-stderr-to-a-file-while-using-tee-with-a-pipe
  2. Some contents of this article is excerpted from <Optimizing Linux® Performance: A Hands-On Guide to Linux® Performance Tools>.

bash & expect tips


/u02/0-sync.sh

#!/usr/bin/expect
set timeout 10800
spawn /u02/sync.sh
expect "*assword*" {send "test\r";exp_continue
expect "*#*" {close}
}

/u02/sync.sh

#! /bin/sh
logpathroot=/u02/logs
synclist=/u02/sync.list

num=1
allnum=`cat $synclist | wc -l`
while [ $num -le $allnum ];
do
line=`awk '{if(NR=='$num') print}' $synclist`
component=`echo $line | awk '{print $2}'`
logtype=`echo $line | awk '{print $3}'`
host=`echo $line | awk '{print $4}'`
source=`echo $line | awk '{print $5}'`
env=`echo $line | awk '{print $1}'`
if [[ "$env" =~ "#" ]]; then
env=`echo $env | sed "s/#//"`
lnenv=`ls /u01/logs | grep -i "$env"`
exist=`ls "/u01/logs/${lnenv}/${component}_${logtype}-$host"`
if [ "$exist" != "" ]; then
rm -f /u01/logs/${lnenv}/${component}_${logtype}-$host
fi
else
lnenv=`ls /u01/logs | grep -i "$env"`
/bin/mkdir -p "${logpathroot}/${env}/${component}/${logtype}-$host"

#rsync -avz --exclude "aime/work/CLOUDTOP/*" /scratch/ /scratch_new/
/usr/bin/rsync -ave ssh --exclude '.zfs' --delete-excluded logs@${host}:${source}/ ${logpathroot}/${env}/${component}/${logtype}-$host
#chmod 755 -R "${logpathroot}/${env}/${component}/"
exist=`ls "/u01/logs/${lnenv}/${component}_${logtype}-$host"`
if [ "$exist" = "" ]; then
# rm -f /u01/logs/${lnenv}/${component}_${logtype}-$host
component1=`echo $component | tr "[:upper:]" "[:lower:]"`
logtype1=`echo $logtype | sed "s/-/_/"`
ln -sf ${logpathroot}/${env}/${component}/${logtype}-$host/ /u01/logs/${lnenv}/${component1}_${host}_${logtype1}_log
fi
fi
num=`expr $num + 1`
done

rm -f /u02/sync_at_*
echo > /u02/sync_at_"`date +%Y-%m-%d` `date +%H:%M` (UTC)"

/u02/sync.list

DC1 apex AdminServer_adr test27vmf4053 /u01/local/config/m_domains/base_domain/servers/AdminServer/adr
DC1 emgc EMGC_ADMINSERVER_adr test27cn05 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/adr
DC1 emgc EMGC_OMS1_adr test27cn05 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr
DC1 emgc EMGC_OMS2_adr test27cn06 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS2/adr
#DC1 emgc EMGC_OMS3_adr test27cn07 /u01/local/exalogic/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS3/adr

PS:

  • If you want to match empty space in expect, you can use the ascii code of space which is 040, and expect -re "\040" would work.
  • About special characters in tcl/expect/shell:

Some characters have special meanings in Tcl or shells (e.g., sh, tcsh, etc.). In particular, these characters may provide security holes for shells.
The characters listed here are escaped by preceeding each one with a backslash (\).
The following characters are dissallowed or have special meanings in Tcl and so are escaped: &;`'"|*?~<>^()[]{}$\-
The following characters are dissallowed or have special meanings in command shells and so are escaped: &;`'"|*?~<>^()[]{}$\\n
The only difference between the two tables is the addition of \n to the shell escape table.

  • To execute multiple commands in one run, you can do below:
[root@vm1 ~]# cat conf.andy
sudo su -
echo $UID >/tmp/e
for i in {1..10}
do
touch /tmp/$i.txt
done
ls /tmp/

[root@vm1 ~]# cat /bin/autoexcute.andy
#!/usr/bin/expect --
set timeout 60
set user [lindex $argv 0]
set ip [lindex $argv 1]
set password [lindex $argv 2]
set com [lindex $argv 3]

spawn ssh -t -o "StrictHostKeyChecking no" $user@$ip
expect {
  "assword:" {
    send "$password\r"
    expect -re "(\\\$|\\\$\040)$" {
        send "/usr/bin/sudo su -\r"
        sleep 2
        expect -re "(\->\040|#|#\040)$" {
            send "$com\r"
            sleep 1
            expect -re "(\->\040|#|#\040)$" {close}
        }
    }
  }
}

[root@vm1 ~]# /bin/autoexcute.andy root testvm password1 "`cat conf.andy`"
spawn ssh -t -o StrictHostKeyChecking no root@testvm
root@testvm's password:
Last login: Tue Sep 15 19:06:49 2015 from vm1.us.oracle.com
-bash-4.1$ /usr/bin/sudo su -
[root@testvm-luis ~]# sudo su -
[root@testvm-luis ~]# echo $UID >/tmp/e
[root@testvm-luis ~]# for i in {1..10}
> do
> touch /tmp/$i.txt
> done
[root@testvm-luis ~]# ls /tmp/
1.txt  
10.txt 
2.txt  
3.txt  
4.txt  
5.txt  
6.txt  
7.txt  
8.txt  
9.txt  
  • To read a file and determine if it's empty in TCL/expect:

set fp_pass [open filename.txt r]
set pass [read $fp_pass]

if {$pass eq ""} {
set account 0

puts "empty"
}

  • Here's something about expect_out in tcp/expect:

expect_feedback_01

expect_feedback_02

GNU Posix Standards Conformance guide - take diffutils for example

In a few cases, the GNU utilities' default behavior is incompatible with the POSIX standard. To suppress these incompatibilities, define the POSIXLY_CORRECT environment variable. Unless you are checking for POSIX conformance, you probably do not need to define POSIXLY_CORRECT.

Normally options and operands can appear in any order, and programs act as if all the options appear before any operands. For example, ‘diff lao tzu -C 2’ acts like ‘diff -C 2 lao tzu’, since ‘2’ is an option-argument of -C. However, if the POSIXLY_CORRECT environment variable is set, options must appear before operands, unless otherwise specified for a particular command.

Newer versions of POSIX are occasionally incompatible with older versions. For example, older versions of POSIX allowed the command ‘diff -c -10’ to have the same meaning as ‘diff -C 10’, but POSIX 1003.1-2001 ‘diff’ no longer allows digit-string options like -10.

The GNU utilities normally conform to the version of POSIX that is standard for your system. To cause them to conform to a different version of POSIX, define the _POSIX2_VERSION environment variable to a value of the form yyyymm specifying the year and month the standard was adopted.

Two values are currently supported for _POSIX2_VERSION: ‘199209’ stands for POSIX 1003.2-1992, and ‘200112’ stands for POSIX 1003.1-2001. For example, if you are running older software that assumes an older version of POSIX and uses ‘diff -c -10’, you can work around the compatibility problems by setting ‘_POSIX2_VERSION=199209’ in your environment.