Home > Linux > linux curl to determin whether the site has a 301 redirection(with high efficiency by bash shell)

linux curl to determin whether the site has a 301 redirection(with high efficiency by bash shell)

October 20th, 2010


Requirements as follows:

I’ve got a task to determine whether the tons of urls has 301 redirection.Tons of urls,yes,that’s what it means,cause you know,about 350 urls.I’m first mad at the task,but then,I tried and get the much more high efficiency way by using bash shell.

The theory of the method is:Using curl with option -I,we can get the header response of the url.If the response contains a line of ‘Location:’,then we can conclude that the url 301 redirects to another site.

The follows are the procedures of the implementation:

1.touch domain.txt,and copy in it the urls(note:url must begin with ‘www.’)
www.xx1.jp
www.xx2.jp
www.xx3.jp
www.xx4.jp

2.Then touch if.sh,and chmod +x ./if.sh to endow it with the execution priviledge :
#!/bin/bash
#copyright:doxer.org
rm -rf ./redirect.txt
for i in `cat ./domain.txt`
do
echo -e “———————-Begin $i———————-\n” >>./redirect.txt
result=`curl -I –max-time 30 $i|grep -F ‘Location:’`
[ $? = 0 ] && echo -e “$i\n” >>./redirect.txt && echo $result >>./redirect.txt
echo -e “———————-End $i————————\n” >>./redirect.txt
done

3.OK!Now use cat ./redirect.txt to see the result:
———————-Begin www.xx1.jp———————-

———————-End www.xx1.jp————————

———————-Begin www.xx2.jp———————-

———————-End www.xx2.jp————————

———————-Begin www.xx3.jp———————-

———————-End www.xx3.jp————————

———————-Begin www.xx4.jp———————-
www.xx4.jp

Location: http://xx5.jp/
———————-End www.xx4.jp————————

It’s easy and with high efficiency,isn’t that?


support_me
Amounts
Categories: Linux Tags:
  1. No comments yet.
  1. No trackbacks yet.