resolved – ssh Read from socket failed: Connection reset by peer and Write failed: Broken pipe

March 13th, 2014

If you met following errors when ssh to linux box:

Read from socket failed: Connection reset by peer

Write failed: Broken pipe

Then there’s one possibility that the linux box’s filesystem was corrupted. As in my case there’s output to stdout:

EXT3-fs error ext3_lookup: deleted inode referenced

To resolve this, you need make linux go to single user mode and fsck -y <filesystem>. You can get corrupted filesystem names when booting:

[/sbin/fsck.ext3 (1) -- /usr] fsck.ext3 -a /dev/xvda2
/usr contains a file system with errors, check forced.
/usr: Directory inode 378101, block 0, offset 0: directory corrupted

(i.e., without -a or -p options)

[/sbin/fsck.ext3 (1) -- /oem] fsck.ext3 -a /dev/xvda5
/oem: recovering journal
/oem: clean, 8253/1048576 files, 202701/1048233 blocks
[/sbin/fsck.ext3 (1) -- /u01] fsck.ext3 -a /dev/xvdb
u01: clean, 36575/14548992 files, 2122736/29081600 blocks

So in this case, I did fsck -y /dev/xvda2 && fsck -y /dev/xvda5. Later reboot host, and then everything went well.


If two VMs are booted up in two hypervisors and these VMs shared the same filesystem(like NFS), then after fsck -y one FS and booted up the VM, the FS will corrupt soon as there’re other copies of itself is using that FS. So you need first make sure that only one copy of VM is running on hypervisors of the same server pool.

Categories: Kernel, Linux Tags:

tcpdump tips

March 13th, 2014

tcpdump [ -AdDefIKlLnNOpqRStuUvxX ] [ -B buffer_size ] [ -c count ]

[ -C file_size ] [ -G rotate_seconds ] [ -F file ]
[ -i interface ] [ -m module ] [ -M secret ]
[ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]
[ -W filecount ]
[ -E spi@ipaddr algo:secret,... ]
[ -y datalinktype ] [ -z postrotate-command ] [ -Z user ] [ expression ]

#general format of a tcp protocol line

src > dst: flags data-seqno ack window urgent options
Src and dst are the source and destination IP addresses and ports.
Flags are some combination of S (SYN), F (FIN), P (PUSH), R (RST), W (ECN CWR) or E (ECN-Echo), or a single ‘.’(means no flags were set)
Data-seqno describes the portion of sequence space covered by the data in this packet.
Ack is sequence number of the next data expected the other direction on this connection.
Window is the number of bytes of receive buffer space available the other direction on this connection.
Urg indicates there is ‘urgent’ data in the packet.
Options are tcp options enclosed in angle brackets (e.g., <mss 1024>).

tcpdump -D #list of the network interfaces available
tcpdump -e #Print the link-level header on each dump line
tcpdump -S #Print absolute, rather than relative, TCP sequence numbers
tcpdump -s <snaplen> #Snarf snaplen bytes of data from each packet rather than the default of 65535 bytes
tcpdump -i eth0 -nn -XX vlan
tcpdump -i eth0 -nn -XX arp
tcpdump -i bond0 -nn -vvv udp dst port 53
tcpdump -i bond0 -nn -vvv host testhost
tcpdump -nn -vvv “dst host and (dst port 1521 or dst port 6200)”

Categories: Life Tags:

psftp through a proxy

March 5th, 2014

You may know that, we can set proxy in putty for ssh to remote host, as shown below:

putty_proxyAnd if you want to scp files from remote site to your local box, you can use putty’s psftp.exe. There’re many options for psftp.exe:

C:\Users\test>d:\PuTTY\psftp.exe -h
PuTTY Secure File Transfer (SFTP) client
Release 0.62
Usage: psftp [options] [user@]host
-V print version information and exit
-pgpfp print PGP key fingerprints and exit
-b file use specified batchfile
-bc output batchfile commands
-be don’t stop batchfile processing if errors
-v show verbose messages
-load sessname Load settings from saved session
-l user connect with specified username
-P port connect to specified port
-pw passw login with specified password
-1 -2 force use of particular SSH protocol version
-4 -6 force use of IPv4 or IPv6
-C enable compression
-i key private key file for authentication
-noagent disable use of Pageant
-agent enable use of Pageant
-batch disable all interactive prompts

Although there’s proxy setting option for putty.exe, there’s no proxy setting for psftp.exe! So what should you do if you want to copy files back to local box, and there’s firewall blocking you from doing this directly, and you must use a proxy?

As you may notice, there’s “-load sessname” option in psftp.exe:

-load sessname Load settings from saved session

This option means that, if you have session opened by putty.exe, then you can use psftp.exe -load <session name> to copy files from remote site. For example, suppose you opened one session named mysession in putty.exe in which you set proxy there, then you can use “psftp.exe -load mysession” to copy files from remote site(no need for username/password, as you must have entered that in putty.exe session):

C:\Users\test>d:\PuTTY\psftp.exe -load mysession
Using username “root”.
Remote working directory is /root
psftp> ls
Listing directory /root
drwx—— 3 ec2-user ec2-user 4096 Mar 4 09:27 .
drwxr-xr-x 3 root root 4096 Dec 10 23:47 ..
-rw——- 1 ec2-user ec2-user 388 Mar 5 05:07 .bash_history
-rw-r–r– 1 ec2-user ec2-user 18 Sep 4 18:23 .bash_logout
-rw-r–r– 1 ec2-user ec2-user 176 Sep 4 18:23 .bash_profile
-rw-r–r– 1 ec2-user ec2-user 124 Sep 4 18:23 .bashrc
drwx—— 2 ec2-user ec2-user 4096 Mar 4 09:21 .ssh
psftp> help
! run a local command
bye finish your SFTP session
cd change your remote working directory
chmod change file permissions and modes
close finish your SFTP session but do not quit PSFTP
del delete files on the remote server
dir list remote files
exit finish your SFTP session
get download a file from the server to your local machine
help give help
lcd change local working directory
lpwd print local working directory
ls list remote files
mget download multiple files at once
mkdir create directories on the remote server
mput upload multiple files at once
mv move or rename file(s) on the remote server
open connect to a host
put upload a file from your local machine to the server
pwd print your remote working directory
quit finish your SFTP session
reget continue downloading files
ren move or rename file(s) on the remote server
reput continue uploading files
rm delete files on the remote server
rmdir remove directories on the remote server

Now you can get/put files as we used to now.


If you do not need proxy connecting to remote site, then you can use psftp.exe CLI to get remote files directly. For example:

d:\PuTTY\psftp.exe [email protected] -i d:\PuTTY\aws.ppk -b d:\PuTTY\script.scr -bc -be -v

And in d:\PuTTY\script.scr is script for put/get files:

cd /backup
lcd c:\
mget *.tar.gz

Categories: Linux, Systems Tags: ,

notes on Ten Steps to ITSM Success

February 25th, 2014
  • Step 1 – Setting the stage

1.   Draft a creditable Business Plan, complete with:

1.1   Clear executive sponsorship

1.2   Rudimentary financial analysis

1.3   Risk analysis

1.4   Organizational impact

1.5   Analysis of alternatives

1.6   Assumptions and constraints

1.7   Recommended implementation approach.

2.   Offer a proposed execution plan.

3.   Identify required resources.

4.   Execute a training and awareness campaign.

  • Step 2 - Inventory the current service offering

1. Gain agreement on current service offerings.
2. Develop cost types and categories.
3. Quantify the cost of each service.
4. Interview key stakeholders.
5. Validate findings with the business sponsor and stakeholders.

  • Step 3 - Validate the current service model

1. Identify and engage with key stakeholders across functional areas.
2. Develop a needs/services questionnaire jointly with customer representatives.
3. Decide on a “best means to an end” – i.e. conduct one-on-one interviews, or facilitate group workshops.
4. Agree and document business value-based, rank-ordered IT service requirements using a tool such as CTQ Tree.
5. Analyze results and develop Heat Maps and service maps.
6. Discuss results with the business, highlighting cost/trade-off areas.

  • Step 4 - Establish an itsm steering committee

1. Assemble an ITSM Steering Committee with cross-organizational representation.
2. Draft a charter outlining the Committee’s role, responsibilities, and scope of authority.
3. Educate the Committee members on their duties and areas of responsibility.
4. Formalize a standardized, repeatable communication strategy, and use it consistently.
5. Create a repository for housing and maintaining a historical record of Committee decisions and issues.
6. Ensure the ITSM Steering Committee is properly aligned with the organization’s enterprise governance model, including other groups with which it must interact.

  • Step 5 - Define the ideal target state

1. Articulate the company’s vision and mission statements. If they do not exist, engage your senior leadership and create them.
2. List the organization’s strategic goals.
3. If one does not already exist, create an organizational strategic plan that incorporates the ITSM Transformation effort.
4. Define specific, measurable, achievable, realistic and time-bound objectives that will achieve the articulated goals.
5. Plan the tasks necessary to achieve the objectives.
6. Create an IT Ecosystem detailing the interactions and relationships in the target state you wish to achieve.
7. Validate that your service management system:
— a. Supports delivery of target state services and agreed service levels
— b. Conforms to architectural policies, principles and guidelines

—c. Defines interfaces and integration points (people, processes, tools and information).

  • Step 6 - Create the IT strategic and tactical plans

1. Negotiate the order in which prioritized capabilities will be developed.
2. Achieve an optimal outcome for your ITSM Transformation by advising on trade-offs and emphasizing shared services, infrastructure and processes.
3. Issue a Notice of Decision when negotiations are complete.
4. Produce an IT Strategic Plan that addresses how IT will build, operate and sustain the capabilities required to deliver customer requirements.
5. Build a Goal Linking matrix.
6. Develop a Program Management Plan that defines the portfolio of projects required to execute the ITSM Steering Committee’s priorities.
7. Generate and publish the ITSM Transformation Roadmap to project staff and all relevant stakeholders, as well as to the Business Sponsor.
8. Construct, approve and publish tactical project plans.

  • Step 7 - Define organizational roles and responsibilities

1. Assess staff skills, functions, authority, accountability, roles, responsibilities and required level of supervision.
2. Schedule executive off-site session(s) with clearly established and enforced “rules of the game.”
3. Build out a top-level enterprise RACI that aligns to enterprise governance.
4. Validate and continue Organizational Change Management activities.

  • Step 8 - Standardized development approach

1. Decide upon and implement a standard process design framework.
2. Agree upon – and then publicize – enterprise standards.
3. Reach consensus on the broad activities each service and underlying process must execute.
4. Charter and staff integrated development teams.
5. Document tool automation requirements and procure licenses for the selected suite of tools.
6. Incorporate project management practices into your design plans.

  • Step 9 - Strategy and planning

1. Assess the planned capability development and prioritization.
2. Update previous assumptions and constraints.
3. Validate the scope of each planned capability.
4. Review and validate stipulated timelines.
5. Combine development activities, where applicable.
6. Create working project plan with milestones and agreed-upon deliverables.
7. Ensure proper allocation and utilization of planned resources.
8. Build a fullyloaded operational project plan.

  • Step 10 - Logical and physical design

1. Construct a business-specific logical design for business users, processing systems and data.
2. Define an enterprise data classification scheme and governance model aligned to security policies.
3. Create policies controlling how and under what circumstances data may be accessed (by people, systems and other data elements).
4. Convene an Administrative Review Session with stakeholders to validate the logical model.
5. Initiate physical design activities.
6. Draft initial transition readiness plan.

  • Step 11 - Build and test

1. Prepare the test facility (development and test environments).
2. Configure, integrate and test the selected tool suite.
3. Create a representative bed of test data.
4. Build the unit and integration test plans.
5. Create an initial draft of user and operator training plans.
6. Design the deployment plan.
7. Conduct the formal Acceptance Testing criteria.

  • Step 12 - Conduct service and process health assessment

1. Pause development activities to take the pulse of your ITSM Improvement Initiative.
2. Approach the Business Sponsor and request an independent Third-Party Service and Process Health Assessment.
3. Validate/refine the list of pertinent questions and oversee data collection.
4. Analyze Assessment results and remediate any identified gaps.

  • Step 13 - Analysis and deployment

1. Remediate discrepancies discovered during Service and Process health assessment.
2. Conduct simulation exercises(if feasible).
3. Execute approved training plans.
4. Prepare the environment for the upcoming change.
5. Schedule the deployment date.
6. Review and validate the back-out plan.
7. Deploy the approved Release Package.

  • Step 14 - Operation and sustainment

1. Ensure operational staff can sustain the new or modified environment.
2. Monitor activities to validate services are performing as desired.
3. Address and correct unforeseen capacity and availability issues.
4. Add or update service and operational documentation to the enterprise Knowledge Repository.

  • Step 15 - Balanced scorecard and continual improvement

1. Verify the target operational steady state has been achieved.
2. Develop and implement an IT Balanced Scorecard (IT BSC):
— Create measures that show that IT’s tangible and intangible to the business.
— Define a balanced set of metrics across the four key areas.
3. Position Continual Improvement as an overall Quality Management approach focused on the voice of the customer (VOC).
4. Create an enterprise Continual Improvement (CSI) Strategy:
— Define a standardized, data-driven, cross-organizational approach to continual improvement across the enterprise.
— Define key goals and objectives in alignment with enterprise strategy.
— Establish strong executive sponsorship and stakeholder buy-in.
— Anticipate argument against the CSI effort and develop countermeasures.
5. Create an enterprise Continual Improvement (CSI) Plan:
— Define the detailed Scope, Stakeholders, and Standards (three S’s).
— Develop an integrated CSI model combining “best of breed” frameworks.
— Design an organizational construct for a dedicated CSI Office/Team.
— Define and staff key CSI roles to execute the Plan.
— Define the Maturity Model, Process Model, Project Selection, Metrics, Auditing and Tools.
6. Use standardized templates to document, analyze, prioritize and manage enterprise improvement opportunities.

  • Step 16 - Putting it all together


This article are excerpts from book <Ten Steps to ITSM Success>.

Categories: IT Architecture Tags: ,

ITIL – Looking at an Example of a Service Design Project

February 18th, 2014

Dummy Co. is a commercial IT service provider. Dummy Co. has implemented ITIL and allocated many of the service management roles to members of the technical teams, as Figure 12-4 illustrates.

figure 12-4

An account manager has found a new customer. The account manager is Henry, and one of his roles is to act as the business relationship manager. The customer requires an invoicing service and doesn’t want to acquire and run the system itself. It wants to utilise it as part of a cloud (see the earlier section ‘Considering system architecture’ for an explanation of a cloud) from a software as a service (SaaS) provider. This means that the users will access the invoicing service from their Internet browsers over a private Internet connection, and use Dummy Co’s software application and hardware.

Henry has established the customer’s business requirements. These have been reviewed using the service portfolio management process (see Chapter 4). Dummy Co. has an existing application that can be used to deliver the service, and this will fulfil most of the utility requirements. However, the warranty requirements are quite demanding and exceed any service levels offered to existing customers. A business case has been established and approved via service portfolio management with a little help from the financial management and demand management processes (see Chapter 4). There is a considerable return on investment (ROI) to be gained from the venture. The design project is initiated. Charlie is the service delivery manager, and she has been allocated the role of design coordination manager, so she takes charge of the project.

Henry has invited Charlie, who also acts as the service level manager, to a meeting with the customer to identify the service level requirements. The service level requirements are as follows:

  • The service will be used at several locations worldwide, so it must be available 24/7, five days a week. Any single failure leading to loss of service at a site must be fixed within 30 minutes, and it will not break more than twice in any one week.
  • The maximum number of people using the service at any one time is likely to be around 3,000. Because the users are spread across various sites, demand will not fluctuate much throughout the day. The response time of the system must be less than two seconds.
  • As the service provider, you need to have appropriate disaster recovery facilities in place in the event that you have a disaster. As the customer, in the event that we have a disaster and want to relocate our staff, we require the ability for the service to be accessible at alternative locations in less than one hour.
  • The data protection act requires us to protect our client’s information. All invoice data must be protected and must be restorable in the event of a cyber-attack. Our data must be backed up and recoverable in the event that you suffer a security breach.

Charlie takes these requirements and documents them as the SLRs(Service Level Requirements). She follows a process flow like the one in Figure 12-2. Charlie reviews the new requirements and compares them with the targets in the existing OLA(Operational Level Agreement) and UC(Underpinning Contract) to see whether there are other services that are delivered to these service levels. If it turns out that these service levels have not been offered before, or there is doubt as to whether they can be achieved, then the technical designers must be consulted.

Well, Dummy Co.’s IT department has just one technical architect, and he is called Fred. Fred does what he always does when he has a new design project to get to grips with: he looks at the architecture of the infrastructure. He looks to see whether there is enough network equipment to cope with the additional load that the new service will need. He looks at the servers in the data centre to see whether there are enough of the right type. He looks at the data storage to see how much is in use. Pretty much what any technical architect would do.

However, Fred has many roles. For the projects that he undertakes, he performs the roles of availability manager, capacity manager, security manager and IT service continuity manager. So when he is reviewing the infrastructure, he takes the SLRs and looks at the requirements from the four warranty points of view. In fact, Fred takes the SLRs and converts them into detailed requirements for availability, capacity, continuity and security, and then considers the requirements in his musings. The work that Fred is doing is sometimes called a capability review. The results of this are that Fred produces an outline design and proposal of how the new requirements will be met. This of course comes at a cost, so Fred will obtain costs and quotes for any additional resources that are needed.

The main content of Fred’s proposal is as follows:

  • To fulfil the 24/7 availability requirement, additional server equipment will be required to provide the necessary resilience. The network currently operates at this level, and no additional resilience is required.
  • To meet the capacity and performance requirement, additional data storage will be added to the existing storage area network. This will also contribute towards the security requirement.
  • In addition, to meet the security requirement, additional data backups are required, and an additional off-site storage facility is needed.
  • The continuity requirement can be met with the existing recovery facility, but the recovery plans must be updated to ensure that the new serv- ice is recovered within the target time.

Fred sends the proposal back to Charlie. The next step is for Dummy Co. to decide whether it wants to spend the money required by Fred’s proposal in order to upgrade the infrastructure, agree to the customer’s requirements and win the business. This involves Charlie, Henry and Dummy Co. senior management. The decision is made to go ahead. Hurrah!

Henry and Charlie arrange to see the customer, and get into negotiation about the finer points of the SLA and, of course, the commercial stuff. Once the SLA is agreed, the detailed design work is started and Charlie liaises with Fred and others in the technical management team to help with any issues as they arise. When the design work is complete, Fred creates an SDP(Service Design Package). This marks the end of the design project. Of course the next step is to build, test and implement the solution – but that’s another story that comes under service transition, which I cover in Chapters 7 and 13.

Figure 12-5 provides an overview of the example in this section. It is not complete as I need a much bigger piece of paper than a page of this book to show it all, but hopefully it gives you an idea.

A swimlane flow diagram like the one in Figure 12-5 is a really good way to visualise how the processes will work in your organisation.

figure 12-5

PS: This article is from book <ITIL For Dummies, 2011 Edition>.


Categories: IT Architecture Tags: ,

checking MTU or Jumbo Frame settings with ping

February 14th, 2014

You may set your linux box’s MTU to jumbo frame sized 9000 bytes or larger, but if the switch your box connected to does not have jumbo frame enabled, then your linux box may met problems when sending & receiving packets.

So how can we get an idea of whether Jumbo Frame enabled on switch or linux box?

Of course you can log on switch and check, but we can also verify this from linux box that connects to switch.

On linux box, you can see the MTU settings of each interface using ifconfig:

[root@centos-doxer ~]# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 08:00:27:3F:C5:08
RX packets:50502 errors:0 dropped:0 overruns:0 frame:0
TX packets:4579 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9835512 (9.3 MiB) TX bytes:1787223 (1.7 MiB)
Base address:0xd010 Memory:f0000000-f0020000

As stated above, 9000 here doesn’t mean that Jumbo Frame enabled on your box to switch. As you can verify with below command:

[root@testbox ~]# ping -c 2 -M do -s 1472 testbox2
PING ( 1472(1500) bytes of data. #so here 1500 bytes go through the network
1480 bytes from ( icmp_seq=1 ttl=252 time=0.319 ms
1480 bytes from ( icmp_seq=2 ttl=252 time=0.372 ms

— ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.319/0.345/0.372/0.032 ms
[root@testbox ~]#
[root@testbox ~]#
[root@testbox ~]# ping -c 2 -M do -s 1473 testbox2
PING ( 1473(1501) bytes of data. #so here 1501 bytes can not go through. From here we can see that MTU for this box is 1500, although ifconfig says it’s 9000
From ( icmp_seq=1 Frag needed and DF set (mtu = 1500)
From ( icmp_seq=1 Frag needed and DF set (mtu = 1500)

— ping statistics —
0 packets transmitted, 0 received, +2 errors

Also, if your the switch is Cisco one, you can verify whether the switch port connecting server has enabled jumbo frame or not by sniffing CDP (Cisco discover protocol) packet. Here’s one example:

-bash-4.1# tcpdump -i eth0 -nn -v -c 1 ether[20:2] == 0×2000 #ether[20:2] == 0×2000 means capture only packets that have a 2 byte value of hex 2000 starting at byte 20
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
03:44:14.221022 CDPv2, ttl: 180s, checksum: 692 (unverified), length 287
Device-ID (0×01), length: 46 bytes: ‘’
Address (0×02), length: 13 bytes: IPv4 (1)
Port-ID (0×03), length: 16 bytes: ‘Ethernet111/1/12′
Capability (0×04), length: 4 bytes: (0×00000228): L2 Switch, IGMP snooping
Version String (0×05), length: 66 bytes:
Cisco Nexus Operating System (NX-OS) Software, Version 5.2(1)N1(4)
Platform (0×06), length: 11 bytes: ‘N5K-C5548UP’
Native VLAN ID (0x0a), length: 2 bytes: 123
AVVID trust bitmap (0×12), length: 1 byte: 0×00
AVVID untrusted ports CoS (0×13), length: 1 byte: 0×00
Duplex (0x0b), length: 1 byte: full
MTU (0×11), length: 4 bytes: 1500 bytes #so here MTU size was set to 1500 bytes
System Name (0×14), length: 18 bytes: ‘ucf-c1z3-swi-5k01b’
System Object ID (not decoded) (0×15), length: 14 bytes:
0×0000: 060c 2b06 0104 0109 0c03 0103 883c
Management Addresses (0×16), length: 13 bytes: IPv4 (1)
Physical Location (0×17), length: 13 bytes: 0×00/snmplocation
1 packets captured
1 packets received by filter
0 packets dropped by kernel
110 packets dropped by interface


  1. As for “-M do” parameter for ping, you may refer to man ping for more info. And as for DF(don’t fragment) and Path MTU Discovery mentioned in the manpage, you may read more on and
  2. Here’s more on tcpdump tips and
  3. Maximum packet size is the MTU plus the data-link header length. Packets are not always transmitted at the Maximum packet size. As we can see from output of iptraf -z eth0.
  4. Here’s more about MTU:

The link layer, which is typically Ethernet, sends information into the network as a series of frames. Even though the layers above may have pieces of information much larger than the frame size, the link layer breaks everything up into frames(which in payload encloses IP packet such as TCP/UDP/ICMP) to send them over the network. This maximum size of data in a frame is known as the maximum transfer unit (MTU). You can use network configuration tools such as ip or ifconfig to set the MTU.

The size of the MTU has a direct impact on the efficiency of the network. Each frame in the link layer has a small header, so using a large MTU increases the ratio of user data to overhead (header). When using a large MTU, however, each frame of data has a higher chance of being corrupted or dropped. For clean physical links, a high MTU usually leads to better performance because it requires less overhead; for noisy links, however, a smaller MTU may actually enhance performance because less data has to be re-sent when a single frame is corrupted.

Here’s one image of layers of network frames: