Archive

Archive for the ‘Networking Security’ Category

resolved – net/core/dev.c:1894 skb_gso_segment+0x298/0x370()

April 19th, 2016 Comments off

Today on one of our servers, there were a lot of errors in /var/log/messages like below:

║Apr 14 21:50:25 test01 kernel: WARNING: at net/core/dev.c:1894
║skb_gso_segment+0x298/0x370()
║Apr 14 21:50:25 test01 kernel: Hardware name: SUN FIRE X4170 M3
║Apr 14 21:50:25 test01 kernel: : caps=(0x60014803, 0x0) len=255
║data_len=215 ip_summed=1
║Apr 14 21:50:25 test01 kernel: Modules linked in: dm_nfs nfs fscache
║auth_rpcgss nfs_acl xen_blkback xen_netback xen_gntdev xen_evtchn lockd
║ @ sunrpc 8021q garp bridge stp llc bonding be2iscsi iscsi_boot_sysfs ib_iser
║ @ rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic uio
║ @ ipv6 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp dm_round_robin libiscsi
║ @ dm_multipath scsi_transport_iscsi xenfs xen_privcmd dm_mirror video sbs sbshc
║acpi_memhotplug acpi_ipmi ipmi_msghandler parport_pc lp parport sr_mod cdrom
║ixgbe hwmon dca snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq
║snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd soundcore
║snd_page_alloc iTCO_wdt iTCO_vendor_support pcspkr ghes i2c_i801 hed i2c_core
║dm_region_hash dm_log dm_mod usb_storage ahci libahci sg shpchp megaraid_sas
║sd_mod crc_t10dif ext3 jbd mbcache
║Apr 14 21:50:25 test01 kernel: Pid: 0, comm: swapper Tainted: G W
║ 2.6.39-400.264.4.el5uek #1
║Apr 14 21:50:25 test01 kernel: Call Trace:
║Apr 14 21:50:25 test01 kernel: <IRQ> [<ffffffff8143dab8>] ?
║skb_gso_segment+0x298/0x370
║Apr 14 21:50:25 test01 kernel: [<ffffffff8106f300>]
║warn_slowpath_common+0x90/0xc0
║Apr 14 21:50:25 test01 kernel: [<ffffffff8106f42e>]
║warn_slowpath_fmt+0x6e/0x70
║Apr 14 21:50:25 test01 kernel: [<ffffffff810d73a7>] ?
║irq_to_desc+0x17/0x20
║Apr 14 21:50:25 test01 kernel: [<ffffffff812faf0c>] ?
║notify_remote_via_irq+0x2c/0x40
║Apr 14 21:50:25 test01 kernel: [<ffffffff8100a820>] ?
║xen_clocksource_read+0x20/0x30
║Apr 14 21:50:25 test01 kernel: [<ffffffff812faf4c>] ?
║xen_send_IPI_one+0x2c/0x40
║Apr 14 21:50:25 test01 kernel: [<ffffffff81011f10>] ?
║xen_smp_send_reschedule+0x10/0x20
║Apr 14 21:50:25 test01 kernel: [<ffffffff81056e0b>] ?
║ttwu_queue_remote+0x4b/0x60
║Apr 14 21:50:25 test01 kernel: [<ffffffff81509a7e>] ?
║_raw_spin_unlock_irqrestore+0x1e/0x30
║Apr 14 21:50:25 test01 kernel: [<ffffffff8143dab8>]
║skb_gso_segment+0x298/0x370
║Apr 14 21:50:25 test01 kernel: [<ffffffff8143dba6>]
║dev_gso_segment+0x16/0x50
║Apr 14 21:50:25 test01 kernel: [<ffffffff8143dfb5>]
║dev_hard_start_xmit+0x3d5/0x530
║Apr 14 21:50:25 test01 kernel: [<ffffffff8145a074>]
║sch_direct_xmit+0xc4/0x1d0
║Apr 14 21:50:25 test01 kernel: [<ffffffff8143e811>]
║dev_queue_xmit+0x161/0x410
║Apr 14 21:50:25 test01 kernel: [<ffffffff815099de>] ?
║_raw_spin_lock+0xe/0x20
║Apr 14 21:50:25 test01 kernel: [<ffffffffa045820c>]
║br_dev_queue_push_xmit+0x6c/0xa0 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffff81076e77>] ?
║local_bh_enable+0x27/0xa0
║Apr 14 21:50:25 test01 kernel: [<ffffffffa045e7ba>]
║br_nf_dev_queue_xmit+0x2a/0x90 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffffa045f668>]
║br_nf_post_routing+0x1f8/0x2e0 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffff81467428>]
║nf_iterate+0x78/0x90
║Apr 14 21:50:25 test01 kernel: [<ffffffff8146777c>]
║nf_hook_slow+0x7c/0x130
║Apr 14 21:50:25 test01 kernel: [<ffffffffa04581a0>] ?
║br_forward_finish+0x70/0x70 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffffa04581a0>] ?
║br_forward_finish+0x70/0x70 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffffa0458130>] ?
║br_flood_deliver+0x20/0x20 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffffa0458186>]
║br_forward_finish+0x56/0x70 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffffa045eba4>]
║br_nf_forward_finish+0xb4/0x180 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffffa045f36f>]
║br_nf_forward_ip+0x26f/0x370 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffff81467428>]
║nf_iterate+0x78/0x90
║Apr 14 21:50:25 test01 kernel: [<ffffffff8146777c>]
║nf_hook_slow+0x7c/0x130
║Apr 14 21:50:25 test01 kernel: [<ffffffffa0458130>] ?
║br_flood_deliver+0x20/0x20 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffff81467428>] ?
║nf_iterate+0x78/0x90
║Apr 14 21:50:25 test01 kernel: [<ffffffffa0458130>] ?
║br_flood_deliver+0x20/0x20 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffffa04582c8>]
║__br_forward+0x88/0xc0 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffffa0458356>]
║br_forward+0x56/0x60 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffffa04591fc>]
║br_handle_frame_finish+0x1ac/0x240 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffffa045ee1b>]
║br_nf_pre_routing_finish+0x1ab/0x350 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffff8115bfe9>] ?
║kmem_cache_alloc_trace+0xc9/0x1a0
║Apr 14 21:50:25 test01 kernel: [<ffffffffa045fc55>]
║br_nf_pre_routing+0x305/0x370 [bridge]
║Apr 14 21:50:25 test01 kernel: [<ffffffff8100122a>] ?
║xen_hypercall_xen_version+0xa/0x20
║Apr 14 21:50:25 test01 kernel: [<ffffffff81467428>]
║nf_iterate+0x78/0x90
║Apr 14 21:50:25 test01 kernel: [<ffffffff8146777c>]
║nf_hook_slow+0x7c/0x130

To fix this, we should disable LRO(large receive offload) first:

for i in eth0 eth1 eth2 eth3;do /sbin/ethtool -K $i lro off;done

And if the NICs are of Intel 10G, the we should disable GRO(generic receive offload) too:

for i in eth0 eth1 eth2 eth3;do /sbin/ethtool -K $i gro off;done

Here's the command to disable both of LRO/GRO:

for i in eth0 eth1 eth2 eth3;do /sbin/ethtool -K $i gro off;/sbin/ethtool -K $i lro off;done

 

resolved – yum Error performing checksum Trying other mirror and finally No more mirrors to try

August 27th, 2015 Comments off

Today when I was installing one package in Linux, below error prompted:

[root@testhost yum.repos.d]# yum list --disablerepo=* --enablerepo=yumpaas
Loaded plugins: rhnplugin, security
This system is not registered with ULN.
ULN support will be disabled.
yumpaas | 2.9 kB 00:00
yumpaas/primary_db | 30 kB 00:00
http://yumrepo.example.com/paas_oel5/repodata/b8e385ebfdd7bed69b7619e63cd82475c8bacc529db7b8c145609b64646d918a-primary.sqlite.bz2: [Errno -3] Error performing checksum
Trying other mirror.
yumpaas/primary_db | 30 kB 00:00
http://yumrepo.example.com/paas_oel5/repodata/b8e385ebfdd7bed69b7619e63cd82475c8bacc529db7b8c145609b64646d918a-primary.sqlite.bz2: [Errno -3] Error performing checksum
Trying other mirror.
Error: failure: repodata/b8e385ebfdd7bed69b7619e63cd82475c8bacc529db7b8c145609b64646d918a-primary.sqlite.bz2 from yumpaas: [Errno 256] No more mirrors to try.

The repo "yumpaas" is hosted on OEL 6 VM, which by default use sha2 for checksum. However, for OEL5 VMs(the VM running yum), yum uses sha1 by default. So I've WA this by install python-hashlib to extend yum's capability to handle sha2(python-hashlib from external repo EPEL).

[root@testhost yum.repos.d]# yum install python-hashlib

And after this, the problematic repo can be used. But to resolve this issue permanently without WA on OEL5 VMs, we should recreate the repo with sha1 for checksum algorithm(createrepo -s sha1).

resolved – Unable to locally verify the issuer’s authority

August 18th, 2015 Comments off

Today we found below error in weblogic server log:

[2015-08-16 14:38:21,866] [WARN] [N/A] [testhost.example.com] [Thrown exception when ...... javax.net.ssl.SSLHandshakeException: General SSLEngine problem
......
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
......
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:323)

And we had a test on the linux server:

[root@testhost1 ~]# wget https://testurl.example.com/
--2015-08-17 10:15:56-- https://testurl.example.com/
Resolving testurl.example.com... 192.168.77.230
Connecting to testurl.example.com|192.168.77.230|:443... connected.
ERROR: cannot verify testurl.example.com's certificate, issued by `/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4':
Unable to locally verify the issuer's authority.
To connect to testurl.example.com insecurely, use `--no-check-certificate'.
Unable to establish SSL connection.

And tested using openssl s_client:

[root@testhost ~]# openssl s_client -connect testurl.example.com:443 -showcerts
CONNECTED(00000003)
depth=1 /C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
verify error:num=2:unable to get issuer certificate
issuer= /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
verify return:0
---
Certificate chain
0 s:/C=US/ST=California/L=Redwood Shores/O=Test Corporation/OU=FOR TESTING PURPOSES ONLY/CN=*.testurl.example.com
i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
-----BEGIN CERTIFICATE-----
MIIFOTCCBCGgAwIBAgIQB1z+1jMRPPv/TrBA2qh5YzANBgkqhkiG9w0BAQsFADB+
MQswCQYDVQQGEwJVUzEdMBsGA
......
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=California/L=Redwood Shores/O=Test Corporation/OU=FOR TESTING PURPOSES ONLY/CN=*.testurl.example.com
issuer=/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
---
No client certificate CA names sent
---
SSL handshake has read 1510 bytes and written 447 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : AES128-SHA
Session-ID: 2B5E228E4705C12D7EE7B5C608AE5E3781C5626C1A99A8D60D21DB2350D4F4DE
Session-ID-ctx:
Master-Key: F06632FD67E22EFB1134CDC0EEE8B8B44972747D804D5C7969095FCB2A692E90F111DC4FC6081F4D94C7561BB556DA16
Key-Arg : None
Krb5 Principal: None
Start Time: 1439793731
Timeout : 300 (sec)
Verify return code: 2 (unable to get issuer certificate)
---

From above, the issue should be caused by CA file missing, as the SSL certificate is relatively new using sha2. So I determined to download CA file from symantec, both "Symantec Class 3 Secure Server CA - G4" and "VeriSign Class 3 Public Primary Certification Authority - G5".

[root@testhost ~]# wget --no-check-certificate 'http://symantec.tbs-certificats.com/SymantecSSG4.crt'

[root@testhost ~]# openssl version -a|grep OPENSSLDIR

OPENSSLDIR: "/etc/pki/tls"

[root@testhost ~]# cp /etc/pki/tls/certs/ca-bundle.crt /etc/pki/tls/certs/ca-bundle.crt.bak

[root@testhost ~]# openssl x509 -text -in "SymantecSSG4.crt" >> /etc/pki/tls/certs/ca-bundle.crt

And test wget again, but still the same error:

[root@testhost1 ~]# wget https://testurl.example.com/
--2015-08-17 10:38:59-- https://testurl.example.com/
Resolving myproxy.example.com... 192.168.19.20
Connecting to myproxy.example.com|192.168.19.20|:80... connected.
ERROR: cannot verify testurl.example.com's certificate, issued by `/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4':
unable to get issuer certificate
To connect to testurl.example.com insecurely, use `--no-check-certificate'.
Unable to establish SSL connection.

Then I determined to install another CA indicated in the error log(G4 above may not be needed):

[root@testhost1 ~]# wget --no-check-certificate 'https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem'

[root@testhost1 ~]# openssl x509 -text -in "VeriSign-Class 3-Public-Primary-Certification-Authority-G5.pem" >> /etc/pki/tls/certs/ca-bundle.crt

This time, wget passed https:

[root@testhost1 ~]# wget https://testurl.example.com/
--2015-08-17 10:40:03-- https://testurl.example.com/
Resolving myproxy.example.com... 192.168.19.20
Connecting to myproxy.example.com|192.168.19.20|:80... connected.
Proxy request sent, awaiting response... 200 OK
Length: 11028 (11K) [text/html]
Saving to: `index.html'

100%[======================================>] 11,028 --.-K/s in 0.02s

2015-08-17 10:40:03 (526 KB/s) - `index.html' saved [11028/11028]

PS:
1. You can check more info here about adding trusted root certificates to the server - Mac OS X / Windows / Linux (Ubuntu, Debian) / Linux (CentOs 6) / Linux (CentOs 5)

2. To use private CA/public cert/private key, you should use below in curl:

curl -v --cacert your-root-ca.crt --cert your-public-cert.crt --key your-private.key--pass mypass -u "username:password" https://url
In this command,your-public-cert.crt is the public cert that you have trusted,your-private.key is the private RSA key portion of the cert that is used to sign the request, and “username:password” should be replaced with the correct username and password.

Also, if you’re using an intermediate cert, you can provide it in one command like so:

curl -v --cacert your-root-ca.crt --cert <(cat your-public-cert.crt  intermediate.crt ) --key your-private.key--pass mypass -u “username:password" https://url

change NIC configuration to make new VLAN tag take effect

April 2nd, 2015 Comments off

Sometimes you may want to add vlan tag to existing NIC, and after the addition, you'll need to change DNS names bound to the old tag with new IPs in the newly added vlan tag. After all these two steps done, you'll need to make changes on the hosts(take linux for example) to make these changes into effect.

In this example, I'm going to move the old v118_FE to the new VLAN v117_FE.

ifconfig v118_FE down
ifconfig bond0.118 down
cd /etc/sysconfig/network-scripts
mv ifcfg-bond0.118 ifcfg-bond0.117
vi ifcfg-bond0.117
    DEVICE=bond0.117
    BOOTPROTO=none
    USERCTL=no
    ONBOOT=yes
    BRIDGE=v117_FE
    VLAN=yes
mv ifcfg-v118_FE ifcfg-v117_FE
vi ifcfg-v117_FE
    DEVICE=v117_FE
    BOOTPROTO=none
    USERCTL=no
    ONBOOT=yes
    STP=off
    TYPE=Bridge
    IPADDR=10.119.236.13
    NETMASK=255.255.248.0
    NETWORK=10.119.232.0
    BROADCAST=10.119.239.255
ifup v117_FE
ifup bond0.117
reboot

resolved – VPN Service not available, The VPN agent service is not responding. Please restart this application after a while.

March 30th, 2015 13 comments

Today when I tried to connect to VPN through Cisco AnyConnect Secure Mobility Client, the following error dialog prompted:

 

VPN Service not available

VPN Service not available

And after I clicked "OK" button, the following dialog prompted:

The VPN agent service is not responding

The VPN agent service is not responding

So all of the two dialogs were complaining about "VPN service" not available/not responding. So I ran "services.msc" in windows run and found below:

vpn service

vpn service

When I checked, the service "Cisco AnyConnect Secure Mobility Agent" was stopped, and the "Startup type" was "Manual". So I changed "Startup type" to "Automatic", click "Start", then "OK" to save.

After this, Cisco AnyConnect Secure Mobility Client was running ok and I can connect through it to VPN.

TCP Window Scaling – values about TCP buffer size

February 4th, 2015 Comments off

TCP Window Scaling(TCP socket buffer size, TCP window size)

/proc/sys/net/ipv4/tcp_window_scaling #1 is to enable window scaling
/proc/sys/net/ipv4/tcp_rmem - memory reserved for TCP rcv buffers. minimum, initial and maximum buffer size
/proc/sys/net/ipv4/tcp_wmem - memory reserved for TCP send buffers
/proc/sys/net/core/rmem_max - maximum receive window
/proc/sys/net/core/wmem_max - maximum send window

The following values (which are the defaults for 2.6.17 with more than 1 GByte of memory) would be reasonable for all paths with a 4MB BDP or smaller:

echo 1 > /proc/sys/net/ipv4/tcp_moderate_rcvbuf #autotuning enabled. The receiver buffer size (and TCP window size) is dynamically updated (autotuned) for each connection. (Sender side autotuning has been present and unconditionally enabled for many years now).
echo 108544 > /proc/sys/net/core/wmem_max
echo 108544 > /proc/sys/net/core/rmem_max
echo "4096 87380 4194304" > /proc/sys/net/ipv4/tcp_rmem
echo "4096 16384 4194304" > /proc/sys/net/ipv4/tcp_wmem

Advanced TCP features

cat /proc/sys/net/ipv4/tcp_timestamps #more is here(allow more accurate RTT measurements for deriving the retransmission timeout estimator; protect against old segments from the previous incarnations of the TCP connection; allow detection of unnecessary retransmissions. But enabling it will also allow you to guess the uptime of a target system.)
cat /proc/sys/net/ipv4/tcp_sack

Here are some background knowledge:

  • The throughput of a communication is limited by two windows: the congestion window and the receive window(TCP congestion window is maintained by the sender, and TCP window size is maintained by the receiver). The former tries not to exceed the capacity of the network (congestion control) and the latter tries not to exceed the capacity of the receiver to process data (flow control). The receiver may be overwhelmed by data if for example it is very busy (such as a Web server). Each TCP segment contains the current value of the receive window. If for example a sender receives an ack which acknowledges byte 4000 and specifies a receive window of 10000 (bytes), the sender will not send packets after byte 14000, even if the congestion window allows it.
  • TCP uses what is called the "congestion window", or CWND, to determine how many packets can be sent at one time. The larger the congestion window size, the higher the throughput. The TCP "slow start" and "congestion avoidance" algorithms determine the size of the congestion window. The maximum congestion window is related to the amount of buffer space that the kernel allocates for each socket. For each socket, there is a default value for the buffer size, which can be changed by the program using a system library call just before opening the socket. There is also a kernel enforced maximum buffer size. The buffer size can be adjusted for both the send and receive ends of the socket.
  • To get maximal throughput it is critical to use optimal TCP send and receive socket buffer sizes for the link you are using. If the buffers are too small, the TCP congestion window will never fully open up. If the receiver buffers are too large, TCP flow control breaks and the sender can overrun the receiver, which will cause the TCP window to shut down. This is likely to happen if the sending host is faster than the receiving host. Overly large windows on the sending side is not usually a problem as long as you have excess memory; note that every TCP socket has the potential to request this amount of memory even for short connections, making it easy to exhaust system resources.
  • More about TCP Buffer Sizing is here.
  • More about /proc/sys/net/ipv4/* Variables is here.

resolved – openssl error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm

December 17th, 2014 Comments off

Today when I tried using curl to get url info, error occurred like below:

[root@centos-doxer ~]# curl -i --user username:password -H "Content-Type: application/json" -X POST --data @/u01/shared/addcredential.json https://testserver.example.com/actions -v

* About to connect() to testserver.example.com port 443

*   Trying 10.242.11.201... connected

* Connected to testserver.example.com (10.242.11.201) port 443

* successfully set certificate verify locations:

*   CAfile: /etc/pki/tls/certs/ca-bundle.crt

  CApath: none

* SSLv2, Client hello (1):

SSLv3, TLS handshake, Server hello (2):

SSLv3, TLS handshake, CERT (11):

SSLv3, TLS alert, Server hello (2):

error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm

* Closing connection #0

After some searching, I found that it's caused by the current version of openssl(openssl-0.9.8e) does not support SHA256 Signature Algorithm. To resolve this, there are two ways:

1. add -k parameter to curl to ignore the SSL error

2. update openssl to "OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008", just try "yum update openssl".

# openssl version
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

3. upgrade openssl to at least openssl-0.9.8o. Here's the way to upgrade openssl: --- this may not be needed, try method 2 above

wget --no-check-certificate 'https://www.openssl.org/source/old/0.9.x/openssl-0.9.8o.tar.gz'
tar zxvf openssl-0.9.8o.tar.gz
cd openssl-0.9.8o
./config --prefix=/usr --openssldir=/usr/openssl
make
make test
make install

After this, run openssl version to confirm:

[root@centos-doxer openssl-0.9.8o]# /usr/bin/openssl version
OpenSSL 0.9.8o 01 Jun 2010

PS:

  • If you installed openssl from rpm package, then you'll find the openssl version is still the old one even after you install the new package. This is expected so don't rely too much on rpm:

[root@centos-doxer openssl-0.9.8o]# /usr/bin/openssl version
OpenSSL 0.9.8o 01 Jun 2010

Even after rebuilding rpm DB(rpm --rebuilddb), it's still the old version:

[root@centos-doxer openssl-0.9.8o]# rpm -qf /usr/bin/openssl
openssl-0.9.8e-26.el5_9.1
openssl-0.9.8e-26.el5_9.1

[root@centos-doxer openssl-0.9.8o]# rpm -qa|grep openssl
openssl-0.9.8e-26.el5_9.1
openssl-devel-0.9.8e-26.el5_9.1
openssl-0.9.8e-26.el5_9.1
openssl-devel-0.9.8e-26.el5_9.1

  • If you met error "Unknown SSL protocol error in connection to xxx" (curl) or "write:errno=104" (openssl) on OEL5, then it's due to curl/openssl comes with OEL5 does not support TLSv1.2 (you can run "curl --help|grep -i tlsv1.2" to confirm). You need manually compile & install curl/openssl to fix this. More info here. (you can softlink /usr/local/bin/curl to /usr/bin/curl, but can leave openssl as it is)
    • OpenSSL to /usr/local/ssl (won't overwirte default openssl)
      • wget --no-check-certificate 'https://www.openssl.org/source/openssl-1.0.2l.tar.gz'
        tar zxvf openssl-1.0.2l.tar.gz
        cd openssl-1.0.2l
        export PATH=/usr/bin:/u01/Allshared/JAVA7/jdk1.7.0_80/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin #to avoid /usr/local readonly error (autofs,perl "/usr/local/bin/perl5: error while loading shared libraries: libdb.so.3: cannot open shared object file: No such file or directory", then export PATH to exclude /usr/local/bin/)
        ./config -fpic shared && make && make install
        echo "/usr/local/ssl/lib" >> /etc/ld.so.conf
        ldconfig
        ls -l /usr/bin/openssl*
        openssl version
      • You will need specify openssl lib path if you compile other packages (such as python):
        • ./configure --prefix=/usr/local/ --enable-shared LDFLAGS="-L/usr/local/ssl/lib" #if you are using non-standard Lib path
  • Curl to /usr/local/curl/bin/curl
    • wget --no-check-certificate 'http://curl.haxx.se/download/curl-7.53.1.tar.gz'
      tar zxvf curl-7.53.1.tar.gz
      cd curl-7.53.1.tar.gz
      ./configure --prefix=/usr/local/curl --with-ssl=/usr/local/ssl LDFLAGS="-L/usr/local/ssl/lib" --disable-ldap
      ls -l /usr/bin/curl*
      mv /usr/bin/curl /usr/bin/curl.old; ln -s /usr/local/curl/bin/curl /usr/bin/curl
      curl -V
  • You will need specify openssl lib path if you compile other packages:
    • ./configure --prefix=/usr/local/ --enable-shared LDFLAGS="-L/usr/local/ssl/lib" #if you are using non-standard Lib path

resolved – high value of RX overruns in ifconfig

November 13th, 2014 Comments off

Today we tried ssh to one host, it stuck soon there. And we observed that RX overruns in ifconfig output was high:

[root@test /]# ifconfig bond0
bond0 Link encap:Ethernet HWaddr 00:10:E0:0D:AD:5E
inet6 addr: fe80::210:e0ff:fe0d:ad5e/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:140234052 errors:0 dropped:0 overruns:12665 frame:0
TX packets:47259596 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:34204561358 (31.8 GiB) TX bytes:21380246716 (19.9 GiB)

Receiver overruns usually occur when packets come in faster than the kernel can service the last interrupt. But in our case, we are seeing increasing inbound errors on interface Eth105/1/7 of device bcd-c1z1-swi-5k07a/b, did shut and no shut but no change. And after some more debugging, we found one bad SFP/cable(uplink port, e.g. WAN port, more info here). After replaced that, the server came back to normal.

example-swi-5k07a# sh int Eth105/1/7 | i err
4065 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 output error 0 collision 0 deferred 0 late collision

example-swi-5k07a# sh int Eth105/1/7 | i err
4099 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 output error 0 collision 0 deferred 0 late collision

example-swi-5k07a# sh int Eth105/1/7 counters errors

--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
Eth105/1/7 3740 483 0 4223 0 0

--------------------------------------------------------------------------------
Port Single-Col Multi-Col Late-Col Exces-Col Carri-Sen Runts
--------------------------------------------------------------------------------
Eth105/1/7 0 0 0 0 0 3740

--------------------------------------------------------------------------------
Port Giants SQETest-Err Deferred-Tx IntMacTx-Er IntMacRx-Er Symbol-Err
--------------------------------------------------------------------------------
Eth105/1/7 0 -- 0 0 0 0

example-swi-5k07a# sh int Eth105/1/7 counters errors

--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
Eth105/1/7 4386 551 0 4937 0 0

--------------------------------------------------------------------------------
Port Single-Col Multi-Col Late-Col Exces-Col Carri-Sen Runts
--------------------------------------------------------------------------------
Eth105/1/7 0 0 0 0 0 4386

--------------------------------------------------------------------------------
Port Giants SQETest-Err Deferred-Tx IntMacTx-Er IntMacRx-Er Symbol-Err
--------------------------------------------------------------------------------
Eth105/1/7 0 -- 0 0 0 0

PS:

During debugging, we also found on server side, the interface eth0 is half duplex and speed at 100Mb/s:

-bash-3.2# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Half
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000003 (3)
Link detected: yes

However, it should be full duplex and 1000Mb/s. So we also changed the speed, duplex to auto auto on switch and after that, the OS side is now showing the expected value:

-bash-3.2# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000003 (3)
Link detected: yes

resolved – auditd STDERR: Error deleting rule Error sending enable request (Operation not permitted)

September 19th, 2014 Comments off

Today when I try to restart auditd, the following error message prompted:

[2014-09-18T19:26:41+00:00] ERROR: service[auditd] (cookbook-devops-kernelaudit::default line 14) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /sbin/service auditd restart ----
STDOUT: Stopping auditd: [  OK  ]
Starting auditd: [FAILED]
STDERR: Error deleting rule (Operation not permitted)
Error sending enable request (Operation not permitted)
---- End output of /sbin/service auditd restart ----
Ran /sbin/service auditd restart returned 1

After some reading of manpage auditd, I realized that when audit "enabled" was set to 2(locked), any attempt to change the configuration in this mode will be audited and denied. And that maybe the reason of "STDERR: Error deleting rule (Operation not permitted)", "Error sending enable request (Operation not permitted)". Here's from man page of auditctl:

-e [0..2] Set enabled flag. When 0 is passed, this can be used to temporarily disable auditing. When 1 is passed as an argument, it will enable auditing. To lock the audit configuration so that it can't be changed, pass a 2 as the argument. Locking the configuration is intended to be the last command in audit.rules for anyone wishing this feature to be active. Any attempt to change the configuration in this mode will be audited and denied. The configuration can only be changed by rebooting the machine.

You can run auditctl -s to check the current setting:

[root@centos-doxer ~]# auditctl -s
AUDIT_STATUS: enabled=1 flag=1 pid=3154 rate_limit=0 backlog_limit=320 lost=0 backlog=0

And you can run auditctl -e <0|1|2> to change this attribute on the fly, or you can add -e <0|1|2> in /etc/audit/audit.rules. Please note after you modify this, a reboot is a must to make this into effect.

PS:

Here's more about linux audit.

arping in linux for getting MAC address and update ARP caches by broadcast

August 27th, 2014 Comments off

Suppose we want to know MAC address of 10.182.120.210. then we can log on one linux host which is in the same subnet of 10.182.120.210, e.g. 10.182.120.188:

[root@centos-doxer ~]#arping -U -c 3 -I bond0 -s 10.182.120.188 10.182.120.210
ARPING 10.182.120.210 from 10.182.120.188 bond0
Unicast reply from 10.182.120.210 [00:21:CC:B7:1F:EB] 1.397ms
Unicast reply from 10.182.120.210 [00:21:CC:B7:1F:EB] 1.378ms
Sent 3 probes (1 broadcast(s))
Received 2 response(s)

So 00:21:CC:B7:1F:EB is the MAC address of 10.182.120.210. And from here we can see that IP address 10.182.120.210 is now used in local network.

Another use of arping is to update ARP cache. One scene is that, you assign a new machine with one being used IP address, then you will not able to log on the old machine with the IP address. Even after you shutdown the new machine, you may still not able to access the old machine. And here's the resolution:

Suppose we have configured the new machine NIC eth0 with IP address 192.168.0.2 which is already used by one old machine. Log on the new machine and run the following commands:

arping -A 192.168.0.2 -I eth0 192.168.0.2
arping -U -s 192.168.0.2 -I eth0 192.168.0.1 #this is sending ARP broadcast, and 192.168.0.1 is the gateway address.
/sbin/arping -I eth0 -c 3 -s 192.168.0.2 192.168.0.3 #update neighbours' ARP caches

PS:

  1. You can run 'arp -nae'(linux) or 'arp -a'(windows) to get arp table.
  2. Here is more about arp sproof prevention (in Chinese. statistic binding/arp firewall/small vlan/PPPoE/immune network).
  3. Here is about Proxy ARP(join broadcast LAN with serial link on router).

resolved – check backend OHS httpd servers for BIG ip F5 LTM VIP

May 23rd, 2014 Comments off

Assume you want to check the OHS or httpd servers one LTM VIP example.vip.com is routing traffic to. Then here's the steps:

  1. get the ip address of VIP example.vip.com;
  2. log on LTM's BUI. Local traffic-> virtual servers -> virtual server list, search ip
  3. click "edit" below column "resource"
  4. note down default pool
  5. search pool name in local traffic -> virtual servers -> pools -> pool list
  6. click the number below column members. Then you'll find the OHS servers and ports the VIP will route traffic to.

PS:

  • To check connections including one specific IP, run below
    • show /sys connection |grep -w <IP>

test telnet from VLAN on cisco router device

May 22nd, 2014 Comments off

If you want to test telnet/ping connection from one vlan to specific destination IP, here is the howto:

test-router# telnet 10.200.244.14 80 source vlan 125
Trying 10.200.244.14...
Connected to 10.200.244.14.
Escape character is '^]'.

ucf-c1z1-rtr-1# ping 10.180.220.71 source 10.180.200.2

PING 10.180.220.71 (10.180.220.71) from 10.240.200.2: 56 data bytes
Request 0 timed out
Request 1 timed out

Good luck.

tcpdump & wireshark tips

March 13th, 2014 Comments off

tcpdump [ -AdDefIKlLnNOpqRStuUvxX ] [ -B buffer_size ] [ -c count ]

[ -C file_size ] [ -G rotate_seconds ] [ -F file ]
[ -i interface ] [ -m module ] [ -M secret ]
[ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]
[ -W filecount ]
[ -E spi@ipaddr algo:secret,... ]
[ -y datalinktype ] [ -z postrotate-command ] [ -Z user ] [ expression ]

#general format of a tcp protocol line

src > dst: flags data-seqno ack window urgent options
Src and dst are the source and destination IP addresses and ports.
Flags are some combination of S (SYN), F (FIN), P (PUSH), R (RST), W (ECN CWR) or E (ECN-Echo), or a single '.'(means no flags were set)
Data-seqno describes the portion of sequence space covered by the data in this packet.
Ack is sequence number of the next data expected the other direction on this connection.
Window is the number of bytes of receive buffer space available the other direction on this connection.
Urg indicates there is 'urgent' data in the packet.
Options are tcp options enclosed in angle brackets (e.g., <mss 1024>).

tcpdump -D #list of the network interfaces available
tcpdump -e #Print the link-level header on each dump line
tcpdump -S #Print absolute, rather than relative, TCP sequence numbers
tcpdump -s <snaplen> #Snarf snaplen bytes of data from each packet rather than the default of 65535 bytes
tcpdump -i eth0 -S -nn -XX vlan
tcpdump -i eth0 -S -nn -XX arp
tcpdump -i bond0 -S -nn -vvv udp dst port 53
tcpdump -i bond0 -S -nn -vvv host testhost
tcpdump -nn -S -vvv "dst host host1.example.com and (dst port 1521 or dst port 6200)"

tcpdump -vv -x -X -s 1500 -i eth0 'port 25' #traffic on SMTP. -xX to print data in addition to header in both hex/ASCII. use -s 192 to watch NFS traffic(NFS requests are very large and much of the detail won't be printed unless snaplen is increased).

tcpdump -nn -S udp dst port 111 #note that telnet is based on tcp protocol, NOT udp. So if you want to test UDP connection(udp is connection-less), then you must start up the app, then use tcpdump to test.

tcpdump -nn -S udp dst portrange 1-1023

Wireshark Capture Filters (in Capture -> Options)

Wireshark DisplayFilters (in toolbar)

Here is another example of TCP 3-way handshake & 4-way handshake & sync flood

EVENT DIAGRAM
Host A sends a TCP SYNchronize packet to Host BHost B receives A's SYNHost B sends a SYNchronize-ACKnowledgementHost A receives B's SYN-ACKHost A sends ACKnowledge

Host B receives ACK.
TCP socket connection is ESTABLISHED.

3-way-handshake
TCP Three Way Handshake
(SYN,SYN-ACK,ACK)

TCP-CLOSE_WAIT

 

The upper part shows the states on the end-point initiating the termination.

The lower part the states on the other end-point.

So the initiating end-point (i.e. the client) sends a termination request to the server and waits for an acknowledgement in state FIN-WAIT-1. The server sends an acknowledgement and goes in state CLOSE_WAIT. The client goes into FIN-WAIT-2 when the acknowledgement is received and waits for an active close. When the server actively sends its own termination request, it goes into LAST-ACK and waits for an acknowledgement from the client. When the client receives the termination request from the server, it sends an acknowledgement and goes into TIME_WAIT and after some time into CLOSED. The server goes into CLOSED state once it receives the acknowledgement from the client.

A socket can be in CLOSE_WAIT state indefinitely until the application closes it. Faulty scenarios would be like filedescriptor leak, server not being execute close() on socket leading to pile up of close_wait sockets. At java level, this manifests as "Too many open files" error. The value cannot be changed.

TIME_WAIT is just a time based wait on socket before closing down the connection permanently. Under most circumstances, sockets in TIME_WAIT is nothing to worry about. The value can be changed(tcp_time_wait_interval).

More info about time_wait & close_wait can be found here.

PS:

You can refer to this article for a detailed explanation of tcp three-way handshake establishing/terminating a connection. And for tcpdump one, you can check below:

[root@host2 ~]# telnet host1 14100
Trying 10.240.249.139...
Connected to host1.us.oracle.com (10.240.249.139).
Escape character is '^]'.
^]
telnet> quit
Connection closed.

[root@host1 ~]# tcpdump -vvv -S host host2
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
03:16:39.188951 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) host1.us.oracle.com.14100 > host2.us.oracle.com.18890: S, cksum 0xa806 (correct), 3445765853:3445765853(0) ack 3946095098 win 5792 <mss 1460,sackOK,timestamp 854077220 860674218,nop,wscale 7> #2. host1 ack SYN package by host2, and add it by 1 as the number to identify this connection(3946095098). Then host1 send a SYN(3445765853).
03:16:41.233807 IP (tos 0x0, ttl 64, id 6650, offset 0, flags [DF], proto: TCP (6), length: 52) host1.us.oracle.com.14100 > host2.us.oracle.com.18890: F, cksum 0xdd48 (correct), 3445765854:3445765854(0) ack 3946095099 win 46 <nop,nop,timestamp 854079265 860676263> #5. host1 Ack F(3946095099), and then it send a F just as host2 did(3445765854 unchanged). 

[root@host2 ~]# tcpdump -vvv -S host host1
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
03:16:39.188628 IP (tos 0x10, ttl 64, id 31059, offset 0, flags [DF], proto: TCP (6), length: 60) host2.us.oracle.com.18890 > host1.us.oracle.com.14100: S, cksum 0x265b (correct), 3946095097:3946095097(0) win 5792 <mss 1460,sackOK,timestamp 860674218 854045985,nop,wscale 7> #1. host2 send a SYN package to host1(3946095097)
03:16:39.188803 IP (tos 0x10, ttl 64, id 31060, offset 0, flags [DF], proto: TCP (6), length: 52) host2.us.oracle.com.18890 > host1.us.oracle.com.14100: ., cksum 0xed44 (correct), 3946095098:3946095098(0) ack 3445765854 win 46 <nop,nop,timestamp 860674218 854077220> #3. host2 ack the SYN sent by host1, and add 1 to identify this connection. The tcp connection is now established(3946095098 unchanged, ack 3445765854).
03:16:41.233397 IP (tos 0x10, ttl 64, id 31061, offset 0, flags [DF], proto: TCP (6), length: 52) host2.us.oracle.com.18890 > host1.us.oracle.com.14100: F, cksum 0xe546 (correct), 3946095098:3946095098(0) ack 3445765854 win 46 <nop,nop,timestamp 860676263 854077220> #4. host2 send a F(in) with a Ack, F will inform host1 that no more data needs sent(3946095098 unchanged), and ack is uded to identify the connection previously established(3445765854 unchanged)
03:16:41.233633 IP (tos 0x10, ttl 64, id 31062, offset 0, flags [DF], proto: TCP (6), length: 52) host2.us.oracle.com.18890 > host1.us.oracle.com.14100: ., cksum 0xdd48 (correct), 3946095099:3946095099(0) ack 3445765855 win 46 <nop,nop,timestamp 860676263 854079265> #6. host2 ack host1's F(3445765855), and the empty flag to identify the connection(3946095099 unchanged).

checking MTU or Jumbo Frame settings with ping

February 14th, 2014 Comments off

You may set your linux box's MTU to jumbo frame sized 9000 bytes or larger(can be between 1500 and 9000 for a 1GbE NIC, and 1500 and 64000 for a 10GbE NIC), but if the switch your box connected to does not have jumbo frame enabled, then your linux box may met problems when sending & receiving packets.

So how can we get an idea of whether Jumbo Frame enabled on switch or linux box?

Of course you can log on switch and check, but we can also verify this from linux box that connects to switch.

On linux box, you can see the MTU settings of each interface using ifconfig:

[root@centos-doxer ~]# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 08:00:27:3F:C5:08
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:50502 errors:0 dropped:0 overruns:0 frame:0
TX packets:4579 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9835512 (9.3 MiB) TX bytes:1787223 (1.7 MiB)
Base address:0xd010 Memory:f0000000-f0020000

As stated above, 9000 here doesn't mean that Jumbo Frame enabled on your box to switch. As you can verify with below command:

[root@testbox ~]# ping -c 2 -M do -s 1472 testbox2
PING testbox2.example.com (192.168.29.184) 1472(1500) bytes of data. #so here 1500 bytes go through the network
1480 bytes from testbox2.example.com (192.168.29.184): icmp_seq=1 ttl=252 time=0.319 ms
1480 bytes from testbox2.example.com (192.168.29.184): icmp_seq=2 ttl=252 time=0.372 ms

--- testbox2.example.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.319/0.345/0.372/0.032 ms

[root@testbox ~]# ping -c 2 -M do -s 1473 testbox2
PING testbox2.example.com (192.168.29.184) 1473(1501) bytes of data. #so here 1501 bytes can not go through. From here we can see that MTU for this box is 1500, although ifconfig says it's 9000
From testbox.example.com (192.168.28.40) icmp_seq=1 Frag needed and DF set (mtu = 1500)
From testbox.example.com (192.168.28.40) icmp_seq=1 Frag needed and DF set (mtu = 1500)

--- testbox2.example.com ping statistics ---
0 packets transmitted, 0 received, +2 errors

Also, if your the switch is Cisco one, you can verify whether the switch port connecting server has enabled jumbo frame or not by sniffing CDP (Cisco discover protocol) packet. Here's one example:

-bash-4.1# tcpdump -i eth0 -nn -v -s 0 -c 1 ether[20:2] == 0x2000 #ether[20:2] == 0x2000 means capture only packets that have a 2 byte value of hex 2000 starting at byte 20
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
03:44:14.221022 CDPv2, ttl: 180s, checksum: 692 (unverified), length 287
Device-ID (0x01), length: 46 bytes: 'ucf-test-swi-5k01b.example.com(SSI16010QJH)'
Address (0x02), length: 13 bytes: IPv4 (1) 192.168.0.242
Port-ID (0x03), length: 16 bytes: 'Ethernet111/1/12'
Capability (0x04), length: 4 bytes: (0x00000228): L2 Switch, IGMP snooping
Version String (0x05), length: 66 bytes:
Cisco Nexus Operating System (NX-OS) Software, Version 5.2(1)N1(4)
Platform (0x06), length: 11 bytes: 'N5K-C5548UP'
Native VLAN ID (0x0a), length: 2 bytes: 123
AVVID trust bitmap (0x12), length: 1 byte: 0x00
AVVID untrusted ports CoS (0x13), length: 1 byte: 0x00
Duplex (0x0b), length: 1 byte: full
MTU (0x11), length: 4 bytes: 1500 bytes #so here MTU size was set to 1500 bytes
System Name (0x14), length: 18 bytes: 'ucf-c1z3-swi-5k01b'
System Object ID (not decoded) (0x15), length: 14 bytes:
0x0000: 060c 2b06 0104 0109 0c03 0103 883c
Management Addresses (0x16), length: 13 bytes: IPv4 (1) 10.131.144.17
Physical Location (0x17), length: 13 bytes: 0x00/snmplocation
1 packets captured
1 packets received by filter
0 packets dropped by kernel
110 packets dropped by interface

PS:

 1. As for "-M do" parameter for ping, you may refer to man ping for more info. And as for DF(don't fragment) and Path MTU Discovery mentioned in the manpage, you may read more on http://en.wikipedia.org/wiki/Path_MTU_discovery and http://en.wikipedia.org/wiki/IP_fragmentation

 2. Maximum packet size is the MTU plus the data-link header length. Packets are not always transmitted at the Maximum packet size. As we can see from output of iptraf -z eth0. To set MTU, you can use ip link set <device> mtu <size>. Here's more about "ip":

ip [ OPTIONS ] OBJECT { COMMAND | help }

OBJECT := { link | addr | addrlabel | route | rule | neigh | tunnel | maddr | mroute | monitor }

OPTIONS := { -V[ersion] | -s[tatistics] | -r[esolve] | -f[amily] { inet | inet6 | ipx | dnet | link } | -o[neline] }

-s, -stats, -statistics

output more information. If the option appears twice or more, the amount of information increases. As a rule, the information is statistics or some time values.

 3. If you find no related data returned:

[root@test ~]# tcpdump -i eth0 -nn -vvv -c 1 ether[20:2] == 0x2000

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
06:17:41.347122 IP (tos 0x0, ttl 64, id 8455, offset 0, flags [+], proto: UDP (17), length: 1500) 10.240.19.231.513 > 10.240.23.255.513: UDP, length 7020
1 packets captured
1 packets received by filter
0 packets dropped by kernel

Then you can increase the size of data from each packet. Setting snaplen to 0 sets it to the default of 65535, for backwards compatibility with recent older versions of tcpdump:

[root@test ~]# tcpdump -i eth0 -nn -vvv -s 0 -c 1 ether[20:2] == 0x2000

Or if CDP totally cannot help you get the info, you can try LLDP(Link Layer Discovery Protocol) which replaced CDP(or you can run lldpctl after installing package lldpd, install method here):

[root@test ~]# tcpdump -i eth3 -s 1500 -XX -c 1 'ether proto 0x88cc'

[root@test ~]# tcpdump -i eth3 -v -s 1500 -c 1 '(ether[12:2]=0x88cc or ether[20:2]=0x2000)'

# lldpctl
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface: eth4, via: LLDP, RID: 2, Time: 0 day, 00:21:15
Chassis:
ChassisID: local test-cz01-o13-swi-2
Port:
PortID: ifalias Ex0/12
-------------------------------------------------------------------------------
Interface: eth5, via: LLDP, RID: 2, Time: 0 day, 07:17:52
Chassis:
ChassisID: local test-cz01-o13-swi-2
Port:
PortID: ifalias Ex0/11
-------------------------------------------------------------------------------
Interface: eth0, via: LLDP, RID: 1, Time: 0 day, 07:18:15
Chassis:
ChassisID: mac 70:ca:9b:d2:f2:3f
SysName: test-cz01-g13-swi-1.ucf.oracle.com
SysDescr: Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-IPBASEK9-M), Version 15.0(2)SG6, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Wed 31-Oct-12 13:38 by prod_r
MgmtIP: 192.168.80.75
Capability: Bridge, on
Capability: Router, on
Port:
PortID: ifname Gi1/19
PortDescr: GigabitEthernet1/19
PMD autoneg: supported: yes, enabled: yes
Adv: 10Base-T, HD: yes, FD: yes
Adv: 100Base-TX, HD: yes, FD: yes
Adv: 1000Base-T, HD: yes, FD: yes
MAU oper type: 1000BaseTFD - Four-pair Category 5 UTP, full duplex mode
VLAN: 125, pvid: yes
-------------------------------------------------------------------------------

PS:

 1. Here's more on tcpdump tips http://dazdaztech.wordpress.com/2013/05/17/using-tcpdump-to-see-cdp-or-lldp-packets/ and http://the-welters.com/professional/tcpdump.html

 2. For EtherType, you can refer to wiki page here https://en.wikipedia.org/wiki/EtherType

 4. Here's more about MTU:

The link layer, which is typically Ethernet, sends information into the network as a series of frames. Even though the layers above may have pieces of information much larger than the frame size, the link layer breaks everything up into frames(which in payload encloses IP packet such as TCP/UDP/ICMP) to send them over the network. This maximum size of data in a frame is known as the maximum transfer unit (MTU). You can use network configuration tools such as ip or ifconfig to set the MTU.

The size of the MTU has a direct impact on the efficiency of the network. Each frame in the link layer has a small header, so using a large MTU increases the ratio of user data to overhead (header). When using a large MTU, however, each frame of data has a higher chance of being corrupted or dropped. For clean physical links, a high MTU usually leads to better performance because it requires less overhead; for noisy links, however, a smaller MTU may actually enhance performance because less data has to be re-sent when a single frame is corrupted.

Here's one image of layers of network frames:

layers-of-network-frames

 

Add static routes in linux which will survive reboot and network bouncing

December 24th, 2013 Comments off

We can see that in linux, the file /etc/sysconfig/static-routes is revoked by /etc/init.d/network:

[root@test-linux ~]# grep static-routes /etc/init.d/network
# Add non interface-specific static-routes.
if [ -f /etc/sysconfig/static-routes ]; then
grep "^any" /etc/sysconfig/static-routes | while read ignore args ; do

So we can add rules in /etc/sysconfig/static-routes to let network routes survive reboot and network bouncing. The format of /etc/sysconfig/static-routes is like:

any net 10.247.17.0 netmask 255.255.255.192 gw 10.247.10.1
any net 10.247.11.128 netmask 255.255.255.192 gw 10.247.10.1

To make route in effect immediately, you can use route add:

route add -net 192.168.62.0 netmask 255.255.255.0 gw 192.168.1.1

But remember that to change the default gateway, we need modify /etc/sysconfig/network(modify GATEWAY=).

After the modification, bounce the network using service network restart to make the changes in effect.

PS: 

  • You need make sure network id follows -net, or you'll see error "route: netmask doesn't match route address".
  • To reload all static routes in /etc/sysconfig/static-routes, you can do the following:
      # Add non interface-specific static-routes.
        if [ -f /etc/sysconfig/static-routes ]; then
           grep "^any" /etc/sysconfig/static-routes | while read ignore args ; do
              /sbin/route add -$args
           done
        fi

resolved – mount clntudp_create: RPC: Program not registered

December 2nd, 2013 Comments off

When I did a showmount -e localhost, error occured:

[root@centos-doxer ~]# showmount -e localhost
mount clntudp_create: RPC: Program not registered

So I checked what RPC program number of showmount was using:

[root@centos-doxer ~]# grep showmount /etc/rpc
mountd 100005 mount showmount

As this indicated, we should startup mountd daemon to make showmount -e localhost work. And mountd is part of nfs, so I started up nfs:

[root@centos-doxer ~]# /etc/init.d/nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]

Now as mountd was running, showmount -e localhost should work.

 

VLAN in windows hyper-v

November 26th, 2013 Comments off

Briefly, a virtual LAN (VLAN) can be regarded as a broadcast domain. It operates on the OSI
network layer 2. The exact protocol definition is known as 802.1Q. Each network packet belong-
ing to a VLAN has an identifier. This is just a number between 0 and 4095, with both 0 and 4095
reserved for other uses. Let’s assume a VLAN with an identifier of 10. A NIC configured with
the VLAN ID of 10 will pick up network packets with the same ID and will ignore all other IDs.
The point of VLANs is that switches and routers enabled for 802.1Q can present VLANs to dif-
ferent switch ports in the network. In other words, where a normal IP subnet is limited to a set
of ports on a physical switch, a subnet defined in a VLAN can be present on any switch port—if
so configured, of course.

Getting back to the VLAN functionality in Hyper-V: both virtual switches and virtual NICs
can detect and use VLAN IDs. Both can accept and reject network packets based on VLAN ID,
which means that the VM does not have to do it itself. The use of VLAN enables Hyper-V to
participate in more advanced network designs. One limitation in the current implementation is
that a virtual switch can have just one VLAN ID, although that should not matter too much in
practice. The default setting is to accept all VLAN IDs.

resolved – sshd: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost= user=

November 20th, 2013 Comments off

Today when I tried to log on one linux server with a normal account, errors were found in /var/log/secure:

Nov 20 07:43:39 test_linux sshd[11200]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.182.120.188 user=testuser
Nov 20 07:43:39 test_linux sshd[11200]: pam_ldap: error trying to bind (Invalid credentials)
Nov 20 07:43:42 test_linux sshd[11200]: nss_ldap: failed to bind to LDAP server ldaps://test.com:7501: Invalid credentials
Nov 20 07:43:42 test_linux sshd[11200]: nss_ldap: failed to bind to LDAP server ldap://test.com: Invalid credentials
Nov 20 07:43:42 test_linux sshd[11200]: nss_ldap: could not search LDAP server - Server is unavailable
Nov 20 07:43:42 test_linux sshd[11200]: nss_ldap: failed to bind to LDAP server ldaps://test.com:7501: Invalid credentials
Nov 20 07:43:43 test_linux sshd[11200]: nss_ldap: failed to bind to LDAP server ldap://test.com: Invalid credentials
Nov 20 07:43:43 test_linux sshd[11200]: nss_ldap: could not search LDAP server - Server is unavailable
Nov 20 07:43:55 test_linux sshd[11200]: pam_ldap: error trying to bind (Invalid credentials)
Nov 20 07:43:55 test_linux sshd[11200]: Failed password for testuser from 10.182.120.188 port 34243 ssh2
Nov 20 07:43:55 test_linux sshd[11201]: fatal: Access denied for user testuser by PAM account configuration

After some attempts on linux PAM(sshd, system-auth), I still got nothing. Later, I checked /etc/ldap.conf with one other box, and found the configuration on the problematic host was not right.

I copied the right ldap.conf and tried log on later, and the issue resolved.

PS:

You can read more about linux PAM here http://www.linux-pam.org/Linux-PAM-html/ (I recommend having a reading on the System Administrators' Guide as that may be the only one linux administrators can reach. You can also get a detailed info on some commonly used PAM modules such as pam_tally2.so, pam_unix.so, pam_cracklib, etc.)

Here's one configuration in /etc/pam.d/sshd:

#%PAM-1.0
auth required pam_tally2.so deny=3 onerr=fail unlock_time=1200 #lock account after 3 failed logins. The accounts will be automatically unlocked after 20 minutes
auth include system-auth
account required pam_nologin.so
account include system-auth
password include system-auth
session optional pam_keyinit.so force revoke
session include system-auth
session required pam_loginuid.so

And here is from /etc/pam.d/system-auth:

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth required pam_env.so
auth required pam_tally2.so onerr=fail deny=3 audit silent
#auth required pam_tally2.so onerr=fail deny=3 unlock_time=300 audit silent
auth sufficient pam_fprintd.so
auth sufficient pam_unix.so try_first_pass
auth requisite pam_succeed_if.so uid >= 500 quiet
auth required pam_deny.so

account required pam_tally2.so onerr=fail
account required pam_unix.so
account sufficient pam_localuser.so
account sufficient pam_succeed_if.so uid < 500 quiet
account required pam_permit.so

#password required pam_cracklib.so retry=3 minlen=14 dcredit=-1 ucredit=-1 ocredit=-1 lcredit=-1 difok=3 enforce_for_root
password sufficient pam_unix.so sha512 shadow try_first_pass remember=5
#password sufficient pam_unix.so sha512 shadow try_first_pass use_authtok remember=5
password required pam_deny.so

session optional pam_keyinit.so revoke
session required pam_limits.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so

You'll get error message "pam_tally2(sshd:auth): user test (502) tally 4, deny 3" in /var/log/secure when you try to log on the after the third time you entered wrong password. And "pam_tally2 --user test" will return 0 Failures after 20 minutes as you configured. You can run pam_tally2 --user test --reset to reset the number to 0.

You can disable pam_tally2 fully with below:

grep pam_tally2 /etc/pam.d/{sshd,login,system-auth}
cp /etc/pam.d/sshd /etc/pam.d/sshd.bak;cp /etc/pam.d/login /etc/pam.d/login.bak;cp /etc/pam.d/system-auth /etc/pam.d/system-auth.bak

sed -i '/pam_tally2/ s/^/# /' /etc/pam.d/sshd;sed -i '/pam_tally2/ s/^/# /' /etc/pam.d/login;sed -i '/pam_tally2/ s/^/# /' /etc/pam.d/system-auth;

To change user password expiry infomation, use chage:

chage -d <last_day> username #Set the number of days since January 1st, 1970 when the password was last changed.
chage -M -1 username #-1 will remove checking a password's validity
chage -E -1 username #remove an account expiration date
chage -E -1 username #remove an account's inactivity
chage -l username #show account aging information
chage -m 0 username #minimum number of days between password changes. 0 indicates that the user may change his/her password at any time
chage -W <days> username #number of days of warning before a password change is required
chage username #If none of the options are selected, chage operates in an interactive fashion, prompting the user with the current values for all of the fields. Enter the new value to change the field, or leave the line blank to use the current value. The current value is displayed between a pair of [ ] marks.

To make a user password never expire, run below

chage -I -1 -m 0 -M 99999 -E -1 username

 

 

make sudo asking for no password on linux

November 1st, 2013 Comments off

Assuming that you have a user named 'test', and he belongs to 'admin' group. So you want user test can sudo to root, and don't want linux prompting for password. Here's the way you can do it:

cp /etc/sudoers{,.bak}
sed -i '/%admin/ s/^/# /' /etc/sudoers
echo '%admin ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers

Enjoy!

disable linux strong password policy

November 1st, 2013 Comments off

You may enable strong password policy for linux, and can disable it of course. So here's the way if you want to disable it:

egrep 'pam_cracklib.so|use_authtok|enforce=/etc/pam.d/system-auth

/bin/cp /etc/pam.d/system-auth{,.bak}
sed -i '/pam_cracklib.so/ s/^/# /' /etc/pam.d/system-auth
sed -i 's/use_authtok//' /etc/pam.d/system-auth

sed -i 's/enforce=everyone/enforce=none/g' /etc/pam.d/system-auth
echo "password" | passwd --stdin username

PS:

  1. To enable strong password for linux, you can have a try on this http://goo.gl/uwdbN
  2. You can read more about linux pam here http://www.linux-pam.org/Linux-PAM-html/
  3. You can also disable pam_tally2 for not locking user accounts after several times of login failures. You can read this article for more info:

grep pam_tally2 /etc/pam.d/{sshd,login,system-auth}
/bin/cp /etc/pam.d/sshd /etc/pam.d/sshd.bak;/bin/cp /etc/pam.d/login /etc/pam.d/login.bak;/bin/cp /etc/pam.d/system-auth /etc/pam.d/system-auth.bak

sed -i '/pam_tally2/ s/^/# /' /etc/pam.d/sshd;sed -i '/pam_tally2/ s/^/# /' /etc/pam.d/login;sed -i '/pam_tally2/ s/^/# /' /etc/pam.d/system-auth;

4. To change user password expiry information, use chage:

chage -d <last_day> username #Set the number of days since January 1st, 1970 when the password was last changed.
chage -M -1 username #-1 will remove checking a password's validity
chage -E -1 username #remove an account expiration date
chage -E -1 username #remove an account's inactivity
chage -l username #show account aging information
chage -m 0 username #minimum number of days between password changes. 0 indicates that the user may change his/her password at any time
chage -W <days> username #number of days of warning before a password change is required
chage username #If none of the options are selected, chage operates in an interactive fashion, prompting the user with the current values for all of the fields. Enter the new value to change the field, or leave the line blank to use the current value. The current value is displayed between a pair of [ ] marks.

F5 big-ip LTM iRULE to redirect http requests to https

October 25th, 2013 Comments off

Here's the irule script:

when HTTP_REQUEST {
HTTP::redirect "https://[HTTP::host][HTTP::uri]"
}

PS:

1.You can read more about F5 LTM docs here http://support.f5.com/kb/en-us/products/big-ip_ltm.html <select a version of big ip software from the left side first>

2.Here's one diagram shows a logical configuration example of the F5 solution for Oracle Database, Applications, Middleware, Servers and Storage:

oracle_f5

PS:

Here is about "Persistence profile types".

tcp flags explanation in details – SYN ACK FIN RST URG PSH and iptables for sync flood

October 11th, 2013 Comments off

This is from wikipedia:

To establish a connection, TCP uses a three-way handshake. Before a client attempts to connect with a server, the server must first bind to and listen at a port to open it up for connections: this is called a passive open. Once the passive open is established, a client may initiate an active open. To establish a connection, the three-way (or 3-step) handshake occurs:

  1. SYN: The active open is performed by the client sending a SYN to the server. The client sets the segment's sequence number to a random value A.
  2. SYN-ACK: In response, the server replies with a SYN-ACK. The acknowledgment number is set to one more than the received sequence number i.e. A+1, and the sequence number that the server chooses for the packet is another random number, B.
  3. ACK: Finally, the client sends an ACK back to the server. The sequence number is set to the received acknowledgement value i.e. A+1, and the acknowledgement number is set to one more than the received sequence number i.e. B+1.

At this point, both the client and server have received an acknowledgment of the connection. The steps 1, 2 establish the connection parameter (sequence number) for one direction and it is acknowledged. The steps 2, 3 establish the connection parameter (sequence number) for the other direction and it is acknowledged. With these, a full-duplex communication is established.

You can read pdf document here http://www.nordu.net/development/2nd-cnnw/tcp-analysis-based-on-flags.pdf

H3C's implementations of sync flood solution http://www.h3c.com/portal/Products___Solutions/Technology/Security_and_VPN/Technology_White_Paper/200812/624110_57_0.htm

Using iptables to resolve sync flood issue http://pierre.linux.edu/2010/04/how-to-secure-your-webserver-against-syn-flooding-and-dos-attack/ and http://www.cyberciti.biz/tips/howto-limit-linux-syn-attacks.html

You may also consider using tcpkill to kill half open sessions(using ss -s/netstat -s<SYN_RECV>/tcptrack to see connection summary)

Output from netstat -atun:

The reason for waiting is that packets may arrive out of order or be retransmitted after the connection has been closed. CLOSE_WAIT indicates that the other side of the connection has closed the connection. TIME_WAIT indicates that this side has closed the connection. The connection is being kept around so that any delayed packets can be matched to the connection and handled appropriately.

more on http://kb.iu.edu/data/ajmi.html about FIN_wait (one error: 2MSL<Maximum Segment Lifetime>=120s, not 2ms)

All about tcp socket states: http://www.krenel.org/tcp-time_wait-and-ephemeral-ports-bad-friends/

And here's more about tcp connection(internet socket) states: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Protocol_operation

device bond0/bond1 does not seem to be present delaying

August 1st, 2013 Comments off

Today I encountered one problem when trying to start up network:

"device bond0 does not seem to be present delaying"

In my env, bond0 was bridged to v119_BR. And eth0 and eth1 that composed bond0 were not in any vlan in the switch side. Here's some more info:

-bash-3.2# ifconfig v119_BR

v119_BR Link encap:Ethernet HWaddr 00:21:28:EE:C5:37
inet addr:10.172.149.171 Bcast:10.172.151.255 Mask:255.255.255.0
inet6 addr: 2606:b400:2010:4051:221:28ff:feee:c537/64 Scope:Global
inet6 addr: fe80::221:28ff:feee:c537/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3830 errors:0 dropped:0 overruns:0 frame:0
TX packets:334 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:292812 (285.9 KiB) TX bytes:65740 (64.1 KiB)

-bash-3.2# cat ifcfg-v119_BR
DEVICE=v119_BR
BOOTPROTO=none
USERCTL=no
ONBOOT=yes
STP=off
TYPE=Bridge
IPADDR=10.172.149.171
NETMASK=255.255.255.0
NETWORK=10.172.144.0
BROADCAST=10.172.151.255

And I've bond0(eth0 and eth1) bridged to v119_BR:

-bash-3.2# cat ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
USERCTL=no
ONBOOT=yes
BRIDGE=v119_BR
BONDING_OPTS="primary=eth0 miimon=100 mode=1"
VLAN=no

PS:

As no multiple VLANs are defined on switch, so no alias interface needed for the bridge v119_BR. If multiple VLANs are defined on eth0 & eth1, then we have to define bond0.117(117 is vlan ID), which should be like the following:
[root@testhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0.117 #and then ifcfg-bond0.118 and more if these vlans are defined on switch
DEVICE=bond0.117
BOOTPROTO=none
USERCTL=no
ONBOOT=yes
BRIDGE=v119_BR
VLAN=yes

[root@testhost ~]# cat ifcfg-bond0.122
DEVICE=bond0.122
BOOTPROTO=none
USERCTL=no
ONBOOT=yes
BRIDGE=v122_MGT
VLAN=yes

[root@testhost network-scripts]# cat ifcfg-bond0 #there's no BRIDGE here, as multiple Bridges will be specified in alias config
DEVICE=bond0
BOOTPROTO=none
USERCTL=no
ONBOOT=yes
BONDING_OPTS="primary=eth0 miimon=100 mode=1"

So what about the problem:

"device bond0/bond0 does not seem to be present delaying"

After some searching, I found that it's caused by bond module not loaded. To resolve this, I tried:

-bash-3.2# uname -r
2.6.18-128.2.1.4.37.el5xen

-bash-3.2# find . -name "*bond*"
./kernel/drivers/net/tulip/winbond-840.ko
./kernel/drivers/net/bonding
./kernel/drivers/net/bonding/bonding.ko

insmod /lib/modules/2.6.18-128.2.1.4.37.el5xen/kernel/drivers/net/bonding/bonding.ko

Add the following lines in /etc/modprobe.conf

alias bond1 bonding
alias bond0 bonding

Then reboot the host. After these steps, the issue was resolved.

get linux server fingerprint through using ssh-keygen

July 31st, 2013 Comments off

Run ssh-keygen -lf  /etc/ssh/ssh_host_rsa_key.pub to get the server's fingerprint which respond to:

$ ssh <test-host>
The authenticity of host 'testhost[ip address]' can't be established.
RSA key fingerprint is <xx:xx:xx....>
Are you sure you want to continue connecting (yes/no)? yes

iptables rules after creating NAT to eth0 in virt-manager

July 18th, 2013 Comments off

During creating a new virtual network in virt-manager GUI, I specified the following parameters:

Network: 192.168.72.0/24

Start: 192.168.72.128

End: 192.168.72.254

Forwarding to physical network(Destination: Physical device eth0, Mode NAT)

After this, the iptables will automatically created:

[root@centos-doxer ~]# iptables-save #dump the contents of an IP Table. Use /etc/init.d/iptables save to save iptables rules to /etc/sysconfig/iptables
*filter
:INPUT ACCEPT [19172:2049589]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [16236:5446648]
-A INPUT -i virbr1 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr1 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr1 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr1 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -d 192.168.72.0/255.255.255.0 -i eth0 -o virbr1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.72.0/255.255.255.0 -i virbr1 -o eth0 -j ACCEPT
-A FORWARD -i virbr1 -o virbr1 -j ACCEPT
-A FORWARD -o virbr1 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr1 -j REJECT --reject-with icmp-port-unreachable
COMMIT

*nat
:PREROUTING ACCEPT [16665:4067972]
:POSTROUTING ACCEPT [197:13739]
:OUTPUT ACCEPT [197:13739]
-A POSTROUTING -s 192.168.72.0/255.255.255.0 -d ! 192.168.72.0/255.255.255.0 -o eth0 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.72.0/255.255.255.0 -d ! 192.168.72.0/255.255.255.0 -o eth0 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.72.0/255.255.255.0 -d ! 192.168.72.0/255.255.255.0 -o eth0 -j MASQUERADE
COMMIT

And here's the bridge info:

[root@centos-doxer ~]# brctl show
bridge name bridge id STP enabled interfaces
virbr0 8000.0800273fc508 no eth0
virbr1 8000.000000000000 yes #one vm will have one interface here, format will be like virbr1-nic

[root@centos-doxer ~]# ifconfig virbr1
virbr1 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet addr:192.168.72.1 Bcast:192.168.72.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:3656 (3.5 KiB)

NAT forwarding for ssh and vncviewer

July 11th, 2013 Comments off

*nat
:PREROUTING ACCEPT [1:88]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A PREROUTING -p tcp --dport 5911 -d 192.168.40.10 -j DNAT --to-destination 172.16.0.101:5904 #now ssh <ip of eth0, 192.168.40.10> -p 5911 is going to visit 172.16.0.101:5904
-A PREROUTING -p tcp --dport 222 -d 192.168.40.10 -j DNAT --to-destination 172.16.0.101:22 #now ssh <ip of eth0, 192.168.40.10> -p 222 is going to visit 172.16.0.101:22
-A POSTROUTING -o eth0 -j MASQUERADE #if eth0 is private ip, you can also do a NAT with one public ip.
COMMIT

*filter
:INPUT ACCEPT [247:16364]
:FORWARD ACCEPT [163:13692]
:OUTPUT ACCEPT [228:18664]
COMMIT

PPTP vpn configuration on linux

July 10th, 2013 Comments off
  • Configure pptp on remote machine:

vi /etc/sysctl.conf
net.ipv4.ip_forward=1

sysctl -p
yum install pptpd

vi /etc/pptpd.conf
option /etc/ppp/options.pptpd
ppp /usr/sbin/pppd
localip 192.168.8.1 #configure it first as an alias eth0:1 for example
remoteip 192.168.8.11-14
netmask 255.255.248.0
lcp-echo-interval 180
lcp-echo-failure 10

vi /etc/ppp/options.pptpd
name pptpd
refuse-pap
refuse-chap
refuse-mschap
require-mschap-v2
require-mppe-128
proxyarp
lock
nobsdcomp
novj
novjccomp
nologfd
idle 2592000
ms-dns 8.8.8.8 #you can also use OpenDNS's DNS such as 208.67.222.222/208.67.220.220
ms-dns 8.8.4.4
mtu 1416
mru 1416
noaccomp

vi /etc/ppp/chap-secrets
"username" pptpd "password" *

vi /etc/sysconfig/iptables
*nat
:PREROUTING ACCEPT [1:88]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A POSTROUTING -o eth0 -j MASQUERADE #eth0 is where public ip resides
#-A POSTROUTING -s 192.168.1.0/24 -j SNAT -to 192.168.8.1
COMMIT

*filter
:INPUT ACCEPT [247:16364]
-A INPUT -p tcp -m state --state NEW -m tcp --dport 1723 -j ACCEPT
-A INPUT -p gre -j ACCEPT #PPTP is a form of PPP between two hosts via GRE using encryption (MPPE) and compression (MPPC)
-A OUTPUT -p gre -j ACCEPT
:FORWARD ACCEPT [163:13692]
:OUTPUT ACCEPT [228:18664]
COMMIT

/etc/init.d/pptpd restart

  • Now add a new VPN connection in Windows.

PS:

  • Here's four types of vpn tunneling protocols:
  1. L2TP(Data Link Layer, Layer 2 Tunneling Protocol) over IPSec(IPSec is used to secure L2TP packets by providing confidentiality, authentication and integrity): xl2tpd. IPsec in tunneling mode does not create virtual physical interfaces at the end of the tunnel, since the tunnel is handled directly by the TCP/IP stack. L2TP can be used to provide these interfaces, this technique is called L2TP/IPsec. However, IPSec has a weakness that prevents it from being used all the time—IPSec can’t travel through a Network Address Translation (NAT) server(here is for l2tp/ipsec going through NAT). If you need to go through a NAT server, you can use PPTP or SSTP. Here is more info about IPSec transport/tunnel mode, and here is more about NAT/PAT.
  2. IPSec(Network Layer): openswan/strongswan
  3. PPTP(Session Layer): PoPTop(alias for pptpd)
  4. SSL based vpn(Session Layer): openvpn/openconnect/SSTP(windows)
  5. IKEv2(Internet Key Exchange Version 2): support VPN reconnect
  • After creating the vpn connection(assume you give it a name doxer-vpn), you'll find the following when you run 'route print':

C:\Users\liandy>route print
===========================================================================
Interface List
70...........................doxer-vpn

......

192.168.8.0    255.255.255.0      192.168.8.1     192.168.8.11     21

  • You may want to enlarge netmask to another one, such as 255.255.248.0. Here's one bat script to automatically connect and change route table:

rasdial doxer-vpn username password

for /f "delims=." %%i in ('route print ^| find /i "doxer-vpn"') do set if=%%i

route add 192.168.8.0 mask 255.255.248.0 0.0.0.0 if %if%

pause

  • And you'll also find that you can not visit the sites blocked by the remote site gateway's firewall. To change this, right click the vpn connection and navigate through:

Networking -> IPV4 -> Properties -> Advanced, uncheck "use default gateway on remote network".

After this, you should now be able to connect to sites blocked by remote firewall.

  • For Tun(layer 3) and Tap(layer 2), you can refer to here

NAT binding one priviate ip and one public ip together using linux as router

July 9th, 2013 Comments off

If you're using Linux as router, and you want to bind one public ip and one private ip together, you can do the following:

  • On linux router, define the public ip and iptables rules:

[root@router ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0:1
DEVICE="eth0:1"
BOOTPROTO="static"
ONBOOT="yes"
TYPE="Ethernet"
IPADDR=10.172.100.171
NETMASK=255.255.248.0

[root@router~]# cat /etc/sysconfig/iptables

*nat
:PREROUTING ACCEPT [1:88]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]

-A POSTROUTING -o eth0 -j MASQUERADE #eth0 is public ip, which acts as a gateway(not router) to internal network. For outbound internal package, eth0 will MASQUERADE their source address
-A PREROUTING -d 10.172.100.171 -j DNAT --to-destination 192.168.6.1 #or you can do port forwarding like "-A PREROUTING -p tcp --dport 5911 -d 192.168.40.10 -j DNAT --to-destination 172.16.0.101:5901"

-A POSTROUTING -s 192.168.6.1 -j SNAT --to-source 10.172.100.171

COMMIT

*filter
:INPUT ACCEPT [247:16364]
:FORWARD ACCEPT [163:13692]
:OUTPUT ACCEPT [228:18664]
COMMIT

PS:

Here're some other types of redirect(REDIRECT is used to redirect the traffic between services on the local machine):

/sbin/iptables -t nat -I PREROUTING -p tcp --dport 2222 -j REDIRECT --to-ports 22

/sbin/iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 2222 -j REDIRECT --to-ports 22

/sbin/iptables -t nat -I OUTPUT -p tcp -d 10.247.17.2 --dport 2222 -j REDIRECT --to-ports 22

*nat
:PREROUTING ACCEPT [2:98]
:POSTROUTING ACCEPT [8:840]
:OUTPUT ACCEPT [8:840]
-A PREROUTING -p tcp -m tcp --dport 389 -j REDIRECT --to-ports 4060
-A OUTPUT -s 10.247.18.143 -p tcp -m tcp --dport 389 -j REDIRECT --to-ports 4060
-A OUTPUT -s 127.0.0.1 -p tcp -m tcp --dport 389 -j REDIRECT --to-ports 4060

configure linux as a router firewall through iptables NAT

June 25th, 2013 Comments off
  • On the linux box that will act as router:

1.Turn on ip_forward:

vi /etc/sysctl.conf

net.ipv4.ip_forward = 1

sysctl -p

2.Edit /etc/sysconfig/iptables:

*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A POSTROUTING -o eth1 -j MASQUERADE #eth1 is the NIC connecting to outside network

#-A POSTROUTING -s 192.168.8.0/255.255.248.0 -o eth0 -j MASQUERADE  #allow 192.168.8.0/21 to do NAT
COMMIT

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

3.Reload iptables:

[root@Router ~]# service iptables restart
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter nat [ OK ]
Unloading iptables modules: [ OK ]
Applying iptables firewall rules: [ OK ]
Loading additional iptables modules: ip_conntrack_netbios_n[ OK ]

[root@Router ~]# iptables -t nat -nL
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[root@Router ~]# iptables -t filter -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

On the linux box that will act as client:

1.Set default gateway to the ip address of linux router:

vi /etc/sysconfig/network

...

GATEWAY=192.168.6.1 #this is ip address of the linux router

...

2.restart network

Test

On router, the default gateway is:

0.0.0.0         10.124.184.1    0.0.0.0         UG        0 0          0 eth1

And on the linux client, we'll now be able to connect to outside network too:

[root@client ~]# ping 10.244.29.184
PING 10.244.29.184 (10.244.29.184) 56(84) bytes of data.
64 bytes from 10.244.29.184: icmp_seq=1 ttl=254 time=0.236 ms

PS:

  1. You can also make linux as firewall using NAT/iptables, more on this article: http://xinn.org/iptables-nat.html
  2. About the numbers in brackets, you can refer to the following: https://www.linuxquestions.org/questions/linux-networking-3/those-%5B-damn-brackets-%5D-in-iptables-must-be-there-for-a-reason-619556/
  3. You should turn on promiscuous mode before applying the configs in this article. If you're using Vsphere Esxi, this is the step:

promiscuous

mtr – combine traceroute and ping

May 28th, 2013 Comments off

mtr combines the functionality of the traceroute(use UDP packets)/tracert(use ping, sometimes better than traceroute) and ping programs in a single network diagnostic tool.

As mtr starts, it investigates the network connection between the host mtr runs on and HOSTNAME. by sending packets with purposly low TTLs. It continues to send packets with low TTL, noting the response time of the intervening routers. This allows mtr to print the response percentage and response times of the internet route to HOSTNAME. A sudden increase in packetloss or response time is often an indication of a bad (or simply overloaded) link.

 

mtr in linux

mtr in linux