Archive

Archive for the ‘Unix’ Category

Linux tips – Performance and Troubleshooting

April 10th, 2014 No comments

System CPU

top
procinfo #yum install procinfo
gnome-system-monitor #can also see network flow rate
mpstat
sar

System Memory

top
free
slabtop
sar
/proc/meminfo #provides the most complete view of system memory usage
procinfo
gnome-system-monitor #can also see network flow rate

Process-specific CPU

time
strace #traces the system calls that a program makes while executing
ltrace #traces the calls(functions) that an application makes to libraries rather than to the kernel. Then use ldd to display which libraries are used, and use objdump to search each of those libraries for the given function.
ps
ld.so #ld

Process-specific Memory

ps
/proc/<pid> #you can refer to http://www.doxer.org/proc-filesystem-day-1/ for more info.

/proc/<PID>/status #provides information about the status of a given process PID
/proc/<PID>/maps #how the process’s virtual address space is used

ipcs #more info on http://www.doxer.org/resolved-semget-failed-with-status-28-failed-oracle-database-starting-up/ and http://www.doxer.org/resolvedload-manager-shared-memory-error-is-28-no-space-left-on-devicefor-apache-pmserver-etc-running-on-linux-solaris-unix/

Disk I/O

vmstat #provides totals rather than the rate of change during the sample
sar
lsof
time sh -c “dd if=/dev/zero of=System2.img bs=1M count=10240 && sync” #10G
time dd if=ddfile of=/dev/null bs=8k
dd if=/dev/zero of=vm1disk bs=1M seek=10240 count=0 #10G

Network

ethtool
ifconfig
ip
iptraf
gkrellm
netstat
gnome-system-monitor #can also see network flow rate
sar #network statistics
/etc/cron.d/sysstat #/var/log/sa/

General Ideas & options & outputs

Run Queue Statistics
In Linux, a process can be either runnable or blocked waiting for an event to complete.

A blocked process may be waiting for data from an I/O device or the results of a system call.

When these processes are runnable, but waiting to use the processor, they form a line called the run queue.
The load on a system is the total amount of running and runnable process.

Context Switches
To create the illusion that a given single processor runs multiple tasks simultaneously, the Linux kernel constantly switches between different processes.
The switch between different processes is called a context switch.
To guarantee that each process receives a fair share of processor time, the kernel periodically interrupts the running process and, if appropriate, the kernel scheduler decides to start another process rather than let the current process continue executing. It is possible that your system will context switch every time this periodic interrupt or timer occurs. (cat /proc/interrupts | grep timer, and do this again after e.g. 10s interval)

Interrupts
In addition, periodically, the processor receives an interrupt by hardware devices.
/proc/interrupts can be examined to show which interrupts are firing on which CPUs

CPU Utilization
At any given time, the CPU can be doing one of seven things:
Idle
Running user code #user time
System time #executing code in the Linux kernel on behalf of the application code
Executing user code that has been “nice”ed or set to run at a lower priority than normal processes
iowait #waiting for I/O (such as disk or network) to complete
irq #means it is in high-priority kernel code handling a hardware interrupt
softirq #executing kernel code that was also triggered by an interrupt, but it is running at a lower priority


Buffers and cache
Alternatively, if your system has much more physical memory than required by your applications, Linux will cache recently used files in physical memory so that subsequent accesses to that file do not require an access to the hard drive. This can greatly speed up applications that access the hard drive frequently, which, obviously, can prove especially useful for frequently launched applications. The first time the application is launched, it needs to be read from the disk; if the application remains in the cache, however, it needs to be read from the much quicker physical memory. This disk cache differs from the processor cache mentioned in the previous chapter. Other than oprofile, valgrind, and kcachegrind, most tools that report statistics about “cache” are actually referring to disk cache.

In addition to cache, Linux also uses extra memory as buffers. To further optimize applications, Linux sets aside memory to use for data that needs to be written to disk. These set-asides are called buffers. If an application has to write something to the disk, which would usually take a long time, Linux lets the application continue immediately but saves the file data into a memory buffer. At some point in the future, the buffer is flushed to disk, but the application can continue immediately.
Active Versus Inactive Memory
Active memory is currently being used by a process. Inactive memory is memory that is allocated but has not been used for a while. Nothing is essentially different between the two types of memory. When required, the Linux kernel takes a process’s least recently used memory pages and moves them from the active to the inactive list. When choosing which memory will be swapped to disk, the kernel chooses from the inactive memory list.
Kernel Usage of Memory (Slabs)
In addition to the memory that applications allocate, the Linux kernel consumes a certain amount for bookkeeping purposes. This bookkeeping includes, for example, keeping track of data arriving from network and disk I/O devices, as well as keeping track of which processes are running and which are sleeping. To manage this bookkeeping, the kernel has a series of caches that contains one or more slabs of memory. Each slab consists of a set of one or more objects. The amount of slab memory consumed by the kernel depends on which parts of the Linux kernel are being used, and can change as the type of load on the machine changes.

slabtop

slabtop shows in real-time how the kernel is allocating its various caches and how full they are. Internally, the kernel has a series of caches that are made up of one or more slabs. Each slab consists of a set of one or more objects. These objects can be active (or used) or inactive (unused). slabtop shows you the status of the different slabs. It shows you how full they are and how much memory they are using.


time

time measures three types of time. First, it measures the real or elapsed time, which is the amount of time between when the program started and finished execution. Next, it measures the user time, which is the amount of time that the CPU spent executing application code on behalf of the program. Finally, time measures system time, which is the amount of time the CPU spent executing system or kernel code on behalf of the application.


Disk I/O

When an application does a read or write, the Linux kernel may have a copy of the file stored into its cache or buffers and returns the requested information without ever accessing the disk. If the Linux kernel does not have a copy of the data stored in memory, however, it adds a request to the disk’s I/O queue. If the Linux kernel notices that multiple requests are asking for contiguous locations on the disk, it merges them into a single big request. This merging increases overall disk performance by eliminating the seek time for the second request. When the request has been placed in the disk queue, if the disk is not currently busy, it starts to service the I/O request. If the disk is busy, the request waits in the queue until the drive is available, and then it is serviced.

iostat

iostat provides a per-device and per-partition breakdown of how many blocks are written to and from a particular disk. (Blocks in iostat are usually sized at 512 bytes.)

lsof
lsof can prove helpful when narrowing down which applications are generating I/O


 top output

S(or STAT) – This is the current status of a process, where the process is either sleeping (S), running (R), zombied (killed but not yet dead) (Z), in an uninterruptable sleep (D), or being traced (T).

TIME – The total amount CPU time (user and system) that this process has used since it started executing.

top options

-b Run in batch mode. Typically, top shows only a single screenful of information, and processes that don’t fit on the screen never display. This option shows all the processes and can be very useful if you are saving top’s output to a file or piping the output to another command for processing.

I This toggles whether top will divide the CPU usage by the number of CPUs on the system. For example, if a process was consuming all of both CPUs on a two-CPU system, this toggles whether top displays a CPU usage of 100% or 200%.

1 (numeral 1) This toggles whether the CPU usage will be broken down to the individual usage or shown as a total.

mpstat options

-P { cpu | ALL } This option tells mpstat which CPUs to monitor. cpu is the number between 0 and the total CPUs minus 1.

The biggest benefit of mpstat is that it shows the time next to the statistics, so you can look for a correlation between CPU usage and time of day.

mpstat can be used to determine whether the CPUs are fully utilized and relatively balanced. By observing the number of interrupts each CPU is handling, it is possible to find an imbalance.

 sar options

-I {irq | SUM | ALL | XALL} This reports the rates that interrupts have been occurring in the system.
-P {cpu | ALL} This option specifies which CPU the statistics should be gathered from. If this isn’t specified, the system totals are reported.
-q This reports information about the run queues and load averages of the machine.
-u This reports information about CPU utilization of the system. (This is the default output.)
-w This reports the number of context switches that occurred in the system.
-o filename This specifies the name of the binary output file that will store the performance statistics.
-f filename This specifies the filename of the performance statistics.

-B – This reports information about the number of blocks that the kernel swapped to and from disk. In addition, for kernel versions after v2.5, it reports information about the number of page faults.
-W – This reports the number of pages of swap that are brought in and out of the system.
-r – This reports information about the memory being used in the system. It includes information about the total free memory, swap, cache, and buffers being used.
-R Report memory statistics

-d –  reports disk activities

-n DEV – Shows statistics about the number of packets and bytes sent and received by each device.
-n EDEV – Shows information about the transmit and receive errors for each device.
-n SOCK – Shows information about the total number of sockets (TCP, UDP, and RAW) in use.
-n ALL – Shows all the network statistics.

sar output

runq-sz This is the size of the run queue when the sample was taken.
plist-sz This is the number of processes present (running, sleeping, or waiting for I/O) when the sample was taken.
proc/s This is the number of new processes created per second. (This is the same as the forks statistic from vmstat.)

tps – Transfers per second. This is the number of reads and writes to the drive/partition per second.
rd_sec/s – Number of disk sectors read per second.
wr_sec/s – Number of disk sectors written per second.


vmstat options

-n print header info only once

-a This changes the default output of memory statistics to indicate the active/inactive amount of memory rather than information about buffer and cache usage.
-s (procps 3.2 or greater) This prints out the vm table. This is a grab bag of different statistics about the system since it has booted. It cannot be run in sample mode. It contains both memory and CPU statistics.

-d – This option displays individual disk statistics at a rate of one sample per interval. The statistics are the totals since system boot, rather than just those that occurred between this sample and the previous sample.
-p partition – This displays performance statistics about the given partition at a rate of one sample per interval. The statistics are the totals since system boot, rather than just those that occurred between this sample and the previous sample.

vmstat output
si – The rate of memory (in KB/s) that has been swapped in from disk during the last sample.
so – The rate of memory (in KB/s) that has been swapped out to disk during the last sample.
pages paged in – The amount of memory (in pages) read from the disk(s) into the system buffers. (On most IA32 systems, a page is 4KB.)
pages paged out – The amount of memory (in pages) written to the disk(s) from the system cache. (On most IA32 systems, a page is 4KB.)
pages swapped in – The amount of memory (in pages) read from swap into system memory.
pages swapped in/out – The amount of memory (in pages) written from system memory to the swap.

bo – This indicates the number of total blocks written to disk in the previous interval. (In vmstat, block size for a disk is typically 1,024 bytes.)
bi – This shows the number of blocks read from the disk in the previous interval. (In vmstat, block size for a disk is typically 1,024 bytes.)
wa – This indicates the amount of CPU time spent waiting for I/O to complete. The rate of disk blocks written per second.
reads: ms – The amount of time (in ms) spent reading from the disk.
writes: ms – The amount of time (in ms) spent writing to the disk.
IO: cur – The total number of I/O that are currently in progress. Note that there is a bug in recent versions of vmstat in which this is incorrectly divided by 1,000, which almost always yields a 0.
IO: s – This is the number of seconds spent waiting for I/O to complete.

iostat options
-d – This displays only information about disk I/O rather than the default display, which includes information about CPU usage as well.
-k – This shows statistics in kilobytes rather than blocks.
-x – This shows extended-performance I/O statistics.
device – If a device is specified, iostat shows only information about that device.

iostat output
tps – Transfers per second. This is the number of reads and writes to the drive/partition per second.
Blk_read/s – The rate of disk blocks read per second.
Blk_wrtn/s – The rate of disk blocks written per second.
Blk_read – The total number of blocks read during the interval.
Blk_wrtn – The total number of blocks written during the interval.
rrqm/s – The number of reads merged before they were issued to the disk.
wrqm/s – The number of writes merged before they were issued to the disk.
r/s – The number of reads issued to the disk per second.
w/s – The number of writes issued to the disk per second.
rsec/s – Disk sectors read per second.
wsec/s – Disk sectors written per second.
avgrq-sz – The average size (in sectors) of disk requests.
avgqu-sz – The average size of the disk request queue.
await – The average time (in ms) for a request to be completely serviced. This average includes the time that the request was waiting in the disk’s queue plus the amount of time it was serviced by the disk.
svctm – The average service time (in ms) for requests submitted to the disk. This indicates how long on average the disk took to complete a request. Unlike await, it does not include the amount of time spent waiting in the queue.

lsof options
+D directory – This causes lsof to recursively search all the files in the given directory and report on which processes are using them.
+d directory – This causes lsof to report on which processes are using the files in the given directory.

lsof output
FD – The file descriptor of the file, or tex for a executable, mem for a memory mapped file.
TYPE – The type of file. REG for a regular file.
DEVICE – Device number in major, minor number.
SIZE – The size of the file.
NODE – The inode of the file.


free options

-s delay – This option causes free to print out new memory statistics every delay seconds.


 strace options

strace [-p <pid>] -s 200 <program>#attach to a process. -s 200 to make the maximum string size to print (the default is 32) to 200. Note that filenames are not considered strings and are always printed in full.

-c – This causes strace to print out a summary of statistics rather than an individual list of all the system calls that are made.

ltrace options
-c – This option causes ltrace to print a summary of all the calls after the command has completed.
-S – ltrace traces system calls in addition to library calls, which is identical to the functionality strace provides.
-p pid – This traces the process with the given PID.


ps options
vsz The virtual set size is the amount of virtual memory that the application is using. Because Linux only allocated physical memory when an application tries to use it, this value may be much greater than the amount of physical memory the application is using.
rss The resident set size is the amount of physical memory the application is currently using.
pmep The percentage of the system memory that the process is consuming.
command This is the command name.

/proc/<PID>/status output
VmSize This is the process’s virtual set size, which is the amount of virtual memory that the application is using. Because Linux only allocates physical memory when an application tries to use it, this value may be much greater than the amount of physical memory the application is actually using. This is the same as the vsz parameter provided by ps.
VmLck This is the amount of memory that has been locked by this process. Locked memory cannot be swapped to disk.
VmRSS This is the resident set size or amount of physical memory the application is currently using. This is the same as the rss statistic provided by ps.

ipcs
Because shared memory is used by multiple processes, it cannot be attributed to any particular process. ipcs provides enough information about the state of the system-wide shared memory to determine which processes allocated the shared memory, which processes are using it, and how often they are using it. This information proves useful when trying to reduce shared memory usage.

ipcs options

lsof –u oracle | grep <shmid> #shmid is from output of ipcs -m. lists the processes under the oracle user attached to the shared memory segment

-t – This shows the time when the shared memory was created, when a process last attached to it, and when a process last detached from it.
-u – This provides a summary about how much shared memory is being used and whether it has been swapped or is in memory.
-l – This shows the system-wide limits for shared memory usage.
-p – This shows the PIDs of the processes that created and last used the shared memory segments.
-c – creator


ifconfig output #more on http://www.thegeekscope.com/linux-ifconfig-command-output-explained/

Errors – Frames with errors (possibly because of a bad network cable or duplex mismatch).
Dropped – Frames that were discarded (most likely because of low amounts of memory or buffers).
Overruns – Frames that may have been discarded by the network card because the kernel or network card was overwhelmed with frames. This should not normally happen.
Frame – These frames were dropped as a result of problems on the physical level. This could be the result of cyclic redundancy check (CRC) errors or other low-level problems.
Compressed – Some lower-level interfaces, such as Point-to-Point Protocol (PPP) or Serial Line Internet Protocol (SLIP) devices compress frames before they are sent over the network. This value indicates the number of these compressed frames. (Compressed packets are usually present during SLIP or PPP connections)

carrier – The number of packets discarded because of link media failure (such as a faulty cable)

ip options
-s [-s] link – If the extra -s is provided to ip, it provides a more detailed list of low-level Ethernet statistics.

iptraf options
-d interface – Detailed statistics for an interface including receive, transmit, and error rates
-s interface – Statistics about which IP ports are being used on an interface and how many bytes are flowing through them
-t <minutes> – Number of minutes that iptraf runs before exiting
-z interface – shows packet counts by size on the specified interface

netstat options
-p – Displays the PID/program name responsible for opening each of the displayed sockets
-c – Continually updates the display of information every second
–interfaces=<name> – Displays network statistics for the given interface
–statistics|-s – IP/UDP/ICMP/TCP statistics
–tcp|-t – Shows only information about TCP sockets
–udp|-u – Shows only information about UDP sockets.
–raw|-w – Shows only information about RAW sockets (IP and ICMP)
–listening|-l – Show only listening sockets. (These are omitted by default.)
–all|-a – Show both listening and non-listening (for TCP this means established connections) sockets. With the –interfaces option, show interfaces that are not marked
–numeric|-n – Show numerical addresses instead of trying to determine symbolic host, port or user names.
–extend|-e – Display additional information. Use this option twice for maximum detail.

netstat output

Active Internet connections (w/o servers)
Proto - The protocol (tcp, udp, raw) used by the socket.
Recv-Q - The count of bytes not copied by the user program connected to this socket.
Send-Q - The count of bytes not acknowledged by the remote host.
Local Address - Address and port number of the local end of the socket. Unless the --numeric (-n) option is specified, the socket address is resolved to its canonical host name (FQDN), and the port number is translated into the corresponding service name.
Foreign Address - Address and port number of the remote end of the socket. Analogous to "Local Address."
State - The state of the socket. Since there are no states in raw mode and usually no states used in UDP, this column may be left blank. Normally this can be one of several values: #more on http://www.doxer.org/tcp-flags-explanation-in-details-syn-ack-fin-rst-urg-psh-and-iptables-for-sync-flood/
    ESTABLISHED
        The socket has an established connection.
    SYN_SENT
        The socket is actively attempting to establish a connection.
    SYN_RECV
        A connection request has been received from the network.
    FIN_WAIT1
        The socket is closed, and the connection is shutting down.
    FIN_WAIT2
        Connection is closed, and the socket is waiting for a shutdown from the remote end.
    TIME_WAIT
        The socket is waiting after close to handle packets still in the network.
    CLOSED
        The socket is not being used.
    CLOSE_WAIT
        The remote end has shut down, waiting for the socket to close.
    LAST_ACK
        The remote end has shut down, and the socket is closed. Waiting for acknowledgement.
    LISTEN
        The socket is listening for incoming connections. Such sockets are not included in the output unless you specify the --listening (-l) or --all (-a) option.
    CLOSING
        Both sockets are shut down but we still don't have all our data sent.
    UNKNOWN
        The state of the socket is unknown.
User - The username or the user id (UID) of the owner of the socket.
PID/Program name - Slash-separated pair of the process id (PID) and process name of the process that owns the socket. --program causes this column to be included. You will also need superuser privileges to see this information on sockets you don't own. This identification information is not yet available for IPX sockets.

Example

[ezolt@scrffy ~/edid]$ vmstat 1 | tee /tmp/output
procs -----------memory---------- ---swap-- -----io----  --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo    in    cs  us sy id wa
0  1 201060  35832  26532 324112    0    0     3     2     6     2  5  1  94  0
0  0 201060  35888  26532 324112    0    0    16     0  1138   358  0  0  99  0
0  0 201060  35888  26540 324104    0    0     0    88  1163   371  0  0 100  0

The number of context switches looks good compared to the number of interrupts. The scheduler is switching processes less than the number of timer interrupts that are firing. This is most likely because the system is nearly idle, and most of the time when the timer interrupt fires, the scheduler does not have any work to do, so it does not switch from the idle process.

[ezolt@scrffy manuscript]$ sar -w -c -q 1 2
Linux 2.6.8-1.521smp (scrffy)   10/20/2004

08:23:29 PM    proc/s
08:23:30 PM      0.00

08:23:29 PM   cswch/s
08:23:30 PM    594.00

08:23:29 PM   runq-sz  plist-sz   ldavg-1    ldavg-5  ldavg-15
08:23:30 PM         0       163      1.12       1.17      1.17

08:23:30 PM    proc/s
08:23:31 PM      0.00

08:23:30 PM   cswch/s
08:23:31 PM    812.87

08:23:30 PM   runq-sz  plist-sz   ldavg-1    ldavg-5  ldavg-15
08:23:31 PM         0       163      1.12       1.17      1.17

Average:       proc/s
Average:         0.00

Average:      cswch/s
Average:       703.98

Average:      runq-sz  plist-sz   ldavg-1    ldavg-5  ldavg-15
Average:            0       163      1.12       1.17      1.17

In this case, we ask sar to show us the total number of context switches and process creations that occur every second. We also ask sar for information about the load average. We can see in this example that this machine has 163 process that are in memory but not running. For the past minute, on average 1.12 processes have been ready to run.

bash-2.05b$ vmstat -a
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free  inact active   si   so    bi    bo   in    cs us sy id wa
 2  1 514004   5640 79816 1341208   33   31   204   247 1111  1548  8  5 73 14

The amount of inactive pages indicates how much of the memory could be swapped to disk and how much is currently being used. In this case, we can see that 1310MB of memory is active, and only 78MB is considered inactive. This machine has a large amount of memory, and much of it is being actively used.


bash-2.05b$ vmstat -s

      1552528  total memory
      1546692  used memory
      1410448  active memory
        11100  inactive memory
         5836  free memory
         2676  buffer memory
       645864  swap cache
      2097096  total swap
       526280  used swap
      1570816  free swap
     20293225 non-nice user cpu ticks
     18284715 nice user cpu ticks
     17687435 system cpu ticks
    357314699 idle cpu ticks
     67673539 IO-wait cpu ticks
       352225 IRQ cpu ticks
      4872449 softirq cpu ticks
    495248623 pages paged in
    600129070 pages paged out
     19877382 pages swapped in
     18874460 pages swapped out
   2702803833 interrupts
   3763550322 CPU context switches
   1094067854 boot time
     20158151 forks

It can be helpful to know the system totals when trying to figure out what percentage of the swap and memory is currently being used. Another interesting statistic is the pages paged in, which indicates the total number of pages that were read from the disk. This statistic includes the pages that are read starting an application and those that the application itself may be using.


[ezolt@wintermute tmp]$ ps -o etime,time,pcpu,cmd 10882
    ELAPSED     TIME %CPU CMD
      00:06 00:00:05 88.0 ./burn

This example shows a test application that is consuming 88 percent of the CPU and has been running for 6 seconds, but has only consumed 5 seconds of CPU time.


[ezolt@wintermute tmp]$ ps –o vsz,rss,tsiz,dsiz,majflt,minflt,cmd 10882
VSZ RSS TSIZ DSIZ MAJFLT MINFLT CMD
11124 10004 1 11122 66 2465 ./burn

The burn application has a very small text size (1KB), but a very large data size (11,122KB). Of the total virtual size (11,124KB), the process has a slightly smaller resident set size (10,004KB), which represents the total amount of physical memory that the process is actually using. In addition, most of the faults generated by burn were minor faults, so most of the memory faults were due to memory allocation rather than loading in a large amount of text or data from the program image on the disk.


[ezolt@wintermute tmp]$ cat /proc/4540/status
Name: burn
State: T (stopped)
Tgid: 4540
Pid: 4540
PPid: 1514
TracerPid: 0
Uid: 501 501 501 501
Gid: 501 501 501 501
FDSize: 256
Groups: 501 9 502
VmSize: 11124 kB
VmLck: 0 kB
VmRSS: 10004 kB
VmData: 9776 kB
VmStk: 8 kB
VmExe: 4 kB
VmLib: 1312 kB
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000000000000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000

The VmLck size of 0KB means that the process has not locked any pages into memory, making them unswappable. The VmRSS size of 10,004KB means that the application is currently using 10,004KB of physical memory, although it has either allocated or mapped the VmSize or 11,124KB. If the application begins to use the memory that it has allocated but is not currently using, the VmRSS size increases but leaves the VmSize unchanged.

[ezolt@wintermute test_app]$ cat /proc/4540/maps
08048000-08049000 r-xp 00000000 21:03 393730 /tmp/burn
08049000-0804a000 rw-p 00000000 21:03 393730 /tmp/burn
0804a000-089d3000 rwxp 00000000 00:00 0
40000000-40015000 r-xp 00000000 21:03 1147263 /lib/ld-2.3.2.so
40015000-40016000 rw-p 00015000 21:03 1147263 /lib/ld-2.3.2.so
4002e000-4002f000 rw-p 00000000 00:00 0
4002f000-40162000 r-xp 00000000 21:03 2031811 /lib/tls/libc-2.3.2.so
40162000-40166000 rw-p 00132000 21:03 2031811 /lib/tls/libc-2.3.2.so
40166000-40168000 rw-p 00000000 00:00 0
bfffe000-c0000000 rwxp fffff000 00:00 0

The burn application is using two libraries: ld and libc. The text section (denoted by the permission r-xp) of libc has a range of 0x4002f000 through 0×40162000 or a size of 0×133000 or 1,257,472 bytes.
The data section (denoted by permission rw-p) of libc has a range of 40162000 through 40166000 or a size of 0×4000 or 16,384 bytes. The text size of libc is bigger than ld’s text size of 0×15000 or 86,016 bytes. The data size of libc is also bigger than ld’s text size of 0×1000 or 4,096 bytes. libc is the big library that burn is linking in.


[ezolt@wintermute tmp]$ ipcs -u

------ Shared Memory Status --------
segments allocated 21
pages allocated 1585
pages resident 720
pages swapped 412
Swap performance: 0 attempts 0 successes

------ Semaphore Status --------
used arrays = 0
allocated semaphores = 0

------ Messages: Status --------
allocated queues = 0
used headers = 0
used space = 0 bytes

In this case, we can see that 21 different segments or pieces of shared memory have been allocated. All these segments consume a total of 1,585 pages of memory; 720 of these exist in physical memory and 412 have been swapped to disk.

[ezolt@wintermute tmp]$ ipcs

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 0 root 777 49152 1
0x00000000 32769 root 777 16384 1
0x00000000 65538 ezolt 600 393216 2 dest

we ask ipcs for a general overview of all the shared memory segments in the system. This indicates who is using each memory segment. In this case, we see a list of all the shared segments. For one in particular, the one with a share memory ID of 65538, the user (ezolt) is the owner. It has a permission of 600 (a typical UNIX permission), which in this case, means that only ezolt can read and write to it. It has 393,216 bytes, and 2 processes are attached to it.

[ezolt@wintermute tmp]$ ipcs -p

------ Shared Memory Creator/Last-op --------
shmid owner cpid lpid
0 root 1224 11954
32769 root 1224 11954
65538 ezolt 1229 11954

Finally, we can figure out exactly which processes created the shared memory segments and which other processes are using them. For the segment with shmid 32769, we can see that the PID 1229 created it and 11954 was the last to use it.


[ezolt@wintermute procps-3.2.0]$ ./vmstat 1 3

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 0 197020 81804 29920 0 0 236 25 1017 67 1 1 93 4
1 1 0 172252 106252 29952 0 0 24448 0 1200 395 1 36 0 63
0 0 0 231068 50004 27924 0 0 19712 80 1179 345 1 34 15 49

During one of the samples, the system read 24,448 disk blocks. As mentioned previously, the block size for a disk is 1,024 bytes(or 4,096 bytes), so this means that the system is reading in data at about 23MB per second. We can also see that during this sample, the CPU was spending a significant portion of time waiting for I/O to complete. The CPU waits on I/O 63 percent of the time during the sample in which the disk was reading at ~23MB per second, and it waits on I/O 49 percent for the next sample, in which the disk was reading at ~19MB per second.

[ezolt@wintermute procps-3.2.0]$ ./vmstat -D
3 disks
5 partitions
53256 total reads
641233 merged reads
4787741 read sectors
343552 milli reading
14479 writes
17556 merged writes
257208 written sectors
7237771 milli writing
0 inprogress IO
342 milli spent IO

In this example, a large number of the reads issued to the system were merged before they were issued to the device. Although there were ~640,000 merged reads, only ~53,000 read commands were actually issued to the drives. The output also tells us that a total of 4,787,741 sectors have been read from the disk, and that since system boot, 343,552ms (or 344 seconds) were spent reading from the disk. The same statistics are available for write performance.

[ezolt@wintermute procps-3.2.0]$ ./vmstat -p hde3 1 3
hde3 reads read sectors writes requested writes
18999 191986 24701 197608
19059 192466 24795 198360
- 19161 193282 24795 198360

Shows that 60 (19,059 – 18,999) reads and 94 writes (24,795 – 24,795) have been issued to partition hde3. This view can prove particularly useful if you are trying to determine which partition of a disk is seeing the most usage.


 

[ezolt@localhost sysstat-5.0.2]$ ./iostat -x -dk 1 5 /dev/hda2
Linux 2.4.22-1.2188.nptl (localhost.localdomain) 05/01/2004
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
hda2 11.22 44.40 3.15 4.20 115.00 388.97 57.50 194.49
68.52 1.75 237.17 11.47 8.43

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
hda2 0.00 1548.00 0.00 100.00 0.00 13240.00 0.00 6620.00
132.40 55.13 538.60 10.00 100.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
hda2 0.00 1365.00 0.00 131.00 0.00 11672.00 0.00 5836.00
89.10 53.86 422.44 7.63 100.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
hda2 0.00 1483.00 0.00 84.00 0.00 12688.00 0.00 6344.00
151.0 39.69 399.52 11.90 100.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
hda2 0.00 2067.00 0.00 123.00 0.00 17664.00 0.00 8832.00
143.61 58.59 508.54 8.13 100.00

you can see that the average queue size is pretty high (~237 to 538) and, as a result, the amount of time that a request must wait (~422.44ms to 538.60ms) is much greater than the amount of time it takes to service the request (7.63ms to 11.90ms). These high average service times, along with the fact that the utilization is 100 percent, show that the disk is completely saturated.


[ezolt@wintermute sysstat-5.0.2]$ sar -n SOCK 1 2

Linux 2.4.22-1.2174.nptlsmp (wintermute.phil.org) 06/07/04
21:32:26 totsck tcpsck udpsck rawsck ip-frag
21:32:27 373 118 8 0 0
21:32:28 373 118 8 0 0
Average: 373 118 8 0 0

We can see the total number of open sockets and the TCP, RAW, and UDP sockets. sar also displays the number of fragmented IP packets.

PS:

avoid putty ssh connection sever or disconnect

January 17th, 2014 2 comments

After sometime, ssh will disconnect itself. If you want to avoid this, you can try run the following command:

while [ 1 ];do echo hi;sleep 60;done &

This will print message “hi” every 60 seconds on the standard output.

PS:

You can also set some parameters in /etc/ssh/sshd_config, you can refer to http://www.doxer.org/learn-linux/make-ssh-on-linux-not-to-disconnect-after-some-certain-time/

Categories: Linux, SHELL, Unix Tags:

debugging nfs problem with snoop in solaris

December 3rd, 2013 No comments

Network analyzers are ultimately the most useful tools available when it comes to debugging NFS problems. The snoop network analyzer bundled with Solaris was introduced in Section 13.5. This section presents an example of how to use snoop to resolve NFS-related problems.

Consider the case where the NFS client rome attempts to access the contents of the filesystems exported by the server zeus through the /net automounter path:

rome% ls -la /net/zeus/export
total 5
dr-xr-xr-x   3 root     root           3 Jul 31 22:51 .
dr-xr-xr-x   2 root     root           2 Jul 31 22:40 ..
drwxr-xr-x   3 root     other        512 Jul 28 16:48 eng
dr-xr-xr-x   1 root     root           1 Jul 31 22:51 home
rome% ls /net/zeus/export/home
/net/zeus/export/home: Permission denied

 

The client is not able to open the contents of the directory /net/zeus/export/home, although the directory gives read and execute permissions to all users:

Code View: Scroll / Show All
rome% df -k /net/zeus/export/home
filesystem            kbytes    used   avail capacity  Mounted on
-hosts                     0       0       0     0%    /net/zeus/export/home

 

The df command shows the -hosts automap mounted on the path of interest. This means that the NFS filesystem rome:/export/home has not yet been mounted. To investigate the problem further, snoopis invoked while the problematic ls command is rerun:

Code View: Scroll / Show All
 rome# snoop -i /tmp/snoop.cap rome zeus
  1   0.00000      rome -> zeus      PORTMAP C GETPORT prog=100003 (NFS) vers=3 
proto=UDP
  2   0.00314      zeus -> rome      PORTMAP R GETPORT port=2049
  3   0.00019      rome -> zeus      NFS C NULL3
  4   0.00110      zeus -> rome      NFS R NULL3 
  5   0.00124      rome -> zeus      PORTMAP C GETPORT prog=100005 (MOUNT) vers=1 
proto=TCP
  6   0.00283      zeus -> rome      PORTMAP R GETPORT port=33168
  7   0.00094      rome -> zeus      TCP D=33168 S=49659 Syn Seq=1331963017 Len=0 
Win=24820 Options=<nop,nop,sackOK,mss 1460>
  8   0.00142      zeus -> rome      TCP D=49659 S=33168 Syn Ack=1331963018 
Seq=4025012052 Len=0 Win=24820 Options=<nop,nop,sackOK,mss 1460>
  9   0.00003      rome -> zeus      TCP D=33168 S=49659     Ack=4025012053 
Seq=1331963018 Len=0 Win=24820
 10   0.00024      rome -> zeus      MOUNT1 C Get export list
 11   0.00073      zeus -> rome      TCP D=49659 S=33168     Ack=1331963062 
Seq=4025012053 Len=0 Win=24776
 12   0.00602      zeus -> rome      MOUNT1 R Get export list 2 entries
 13   0.00003      rome -> zeus      TCP D=33168 S=49659     Ack=4025012173 
Seq=1331963062 Len=0 Win=24820
 14   0.00026      rome -> zeus      TCP D=33168 S=49659 Fin Ack=4025012173 
Seq=1331963062 Len=0 Win=24820
 15   0.00065      zeus -> rome      TCP D=49659 S=33168     Ack=1331963063 
Seq=4025012173 Len=0 Win=24820
 16   0.00079      zeus -> rome      TCP D=49659 S=33168 Fin Ack=1331963063 
Seq=4025012173 Len=0 Win=24820
 17   0.00004      rome -> zeus      TCP D=33168 S=49659     Ack=4025012174 
Seq=1331963063 Len=0 Win=24820
 18   0.00058      rome -> zeus      PORTMAP C GETPORT prog=100005 (MOUNT) vers=3 
proto=UDP
 19   0.00412      zeus -> rome      PORTMAP R GETPORT port=34582
 20   0.00018      rome -> zeus      MOUNT3 C Null
 21   0.00134      zeus -> rome      MOUNT3 R Null 
 22   0.00056      rome -> zeus      MOUNT3 C Mount /export/home
 23   0.23112      zeus -> rome      MOUNT3 R Mount Permission denied

 

Packet 1 shows the client rome requesting the port number of the NFS service (RPC program number 100003, Version 3, over the UDP protocol) from the server’s rpcbind (portmapper). Packet 2 shows the server’s reply indicating nfsd is running on port 2049. Packet 3 shows the automounter’s call to the server’s nfsd daemon to verify that it is indeed running. The server’s successful reply is shown in packet 4. Packet 5 shows the client’s request for the port number for RPC program number 100005, Version 1, over TCP (the RPC MOUNT program). The server replies with packet 6 with port=33168. Packets 7 through 9 are TCP hand shaking between our NFS client and the server’s mountd. Packet 10 shows the client’s call to the server’s mountd daemon (which implements the MOUNT program) currently running on port 33168. The client is requesting the list of exported entries. The server replies with packet 12 including the names of the two entries exported. Packets 18 and 19 are similar to packets 5 and 6, except that this time the client is asking for the port number of the MOUNT program version 3 running over UDP. Packet 20 and 21 show the client verifying that version 3 of the MOUNT service is up and running on the server. Finally, the client issues the Mount /export/home request to the server in packet 22, requesting the filehandle of the /export/home path. The server’s mountd daemon checks its export list, and determines that the host rome is not present in it and replies to the client with a “Permission Denied” error in packet 23.

The analysis indicates that the “Permission Denied” error returned to the ls command came from the MOUNT request made to the server, not from problems with directory mode bits on the client. Having gathered this information, we study the exported list on the server and quickly notice that the filesystem /export/home is exported only to the host verona:

rome$ showmount -e zeus
export list for zeus:
/export/eng  (everyone)
/export/home verona

 

We could have obtained the same information by inspecting the contents of packet 12, which contains the export list requested during the transaction:

Code View: Scroll / Show All
rome# snoop -i /tmp/cap -v -p 10,12
...
      Packet 10 arrived at 3:32:47.73
RPC:  ----- SUN RPC Header -----
RPC:  
RPC:  Record Mark: last fragment, length = 40
RPC:  Transaction id = 965581102
RPC:  Type = 0 (Call)
RPC:  RPC version = 2
RPC:  Program = 100005 (MOUNT), version = 1, procedure = 5
RPC:  Credentials: Flavor = 0 (None), len = 0 bytes
RPC:  Verifier   : Flavor = 0 (None), len = 0 bytes
RPC:  
MOUNT:----- NFS MOUNT -----
MOUNT:
MOUNT:Proc = 5 (Return export list)
MOUNT:
...
       Packet 12 arrived at 3:32:47.74
RPC:  ----- SUN RPC Header -----
RPC:  
RPC:  Record Mark: last fragment, length = 92
RPC:  Transaction id = 965581102
RPC:  Type = 1 (Reply)
RPC:  This is a reply to frame 10
RPC:  Status = 0 (Accepted)
RPC:  Verifier   : Flavor = 0 (None), len = 0 bytes
RPC:  Accept status = 0 (Success)
RPC:  
MOUNT:----- NFS MOUNT -----
MOUNT:
MOUNT:Proc = 5 (Return export list)
MOUNT:Directory = /export/eng
MOUNT:Directory = /export/home
MOUNT: Group = verona
MOUNT:

 

For simplicity, only the RPC and NFS Mount portions of the packets are shown. Packet 10 is the request for the export list, packet 12 is the reply. Notice that every RPC packet contains the transaction ID (XID), the message type (call or reply), the status of the call, and the credentials. Notice that the RPC header includes the string “This is a reply to frame 10″. This is not part of the network packet. Snoopkeeps track of the XIDs it has processed and attempts to match calls with replies and retransmissions. This feature comes in very handy during debugging. The Mount portion of packet 12 shows the list of directories exported and the group of hosts to which they are exported. In this case, we can see that /export/home was only exported with access rights to the host verona. The problem can be fixed by adding the host rome to the export list on the server.

PS:

explain solaris snoop network analyzer with example

December 2nd, 2013 No comments

Here’s the code:

# snoop -i /tmp/capture -v -p 3
ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 3 arrived at 15:08:43.35
ETHER:  Packet size = 82 bytes
ETHER:  Destination = 0:0:c:7:ac:56, Cisco
ETHER:  Source      = 8:0:20:b9:2b:f6, Sun
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:   
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:   Total length = 68 bytes
IP:   Identification = 35462
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 255 seconds/hops
IP:   Protocol = 17 (UDP)
IP:   Header checksum = 4503
IP:   Source address = 131.40.52.223, caramba
IP:   Destination address = 131.40.52.27, mickey
IP:   No options
IP:   
UDP:  ----- UDP Header -----
UDP:  
UDP:  Source port = 55559
UDP:  Destination port = 2049 (Sun RPC)
UDP:  Length = 48 
UDP:  Checksum = 3685 
UDP:  
RPC:  ----- SUN RPC Header -----
RPC:  
RPC:  Transaction id = 969440111
RPC:  Type = 0 (Call)
RPC:  RPC version = 2
RPC:  Program = 100003 (NFS), version = 3, procedure = 0
RPC:  Credentials: Flavor = 0 (None), len = 0 bytes
RPC:  Verifier   : Flavor = 0 (None), len = 0 bytes
RPC:  
NFS:  ----- Sun NFS -----
NFS:  
NFS:  Proc = 0 (Null procedure)
NFS:

And let’s analyze this:

The Ethernet header displays the source and destination addresses as well as the type of information embedded in the packet. The IP layer displays the IP version number, flags, options, and address of the sender and recipient of the packet. The UDP header displays the source and destination ports, along with the length and checksum of the UDP portion of the packet. Embedded in the UDP frame is the RPC data. Every RPC packet has a transaction ID used by the sender to identify replies to its requests, and by the server to identify duplicate calls. The previous example shows a request from the host caramba to the server mickey. The RPC version = 2 refers to the version of the RPC protocol itself, the program number 100003 and Version 3 apply to the NFS service. NFS procedure 0 is always the NULL procedure, and is most commonly invoked with no authentication information. The NFS NULL procedure does not take any arguments, therefore none are listed in the NFS portion of the packet.

PS:

  1. Here’s more usage about snoop on solaris:

The amount of traffic on a busy network can be overwhelming, containing many irrelevant packets to the problem at hand. The use of filters reduces the amount of noise captured and displayed, allowing you to focus on relevant data. A filter can be applied at the time the data is captured, or at the time the data is displayed. Applying the filter at capture time reduces the amount of data that needs to be stored and processed during display. Applying the filter at display time allows you to further refine the previously captured information. You will find yourself applying different display filters to the same data set as you narrow the problem down, and isolate the network packets of interest.

Snoop uses the same syntax for capture and display filters. For example, the host filter instructs snoop to only capture packets with source or destination address matching the specified host:

Code View: Scroll / Show All
# snoop host caramba
Using device /dev/hme (promiscuous mode)
     caramba -> schooner     NFS C GETATTR3 FH=B083
    schooner -> caramba      NFS R GETATTR3 OK
     caramba -> schooner     TCP D=2049 S=1023     Ack=3647506101 Seq=2611574902 Len=0 Win=24820

 

In this example the host filter instructs snoop to capture packets originating at or addressed to the host caramba. You can specify the IP address or the hostname, and snoop will use the name service switch to do the conversion. Snoop assumes that the hostname specified is an IPv4 address. You can specify an IPv6 address by using the inet6 qualifier in front of the host filter:

Code View: Scroll / Show All
# snoop inet6 host caramba
Using device /dev/hme (promiscuous mode)
     caramba -> 2100::56:a00:20ff:fea0:3390    ICMPv6 Neighbor advertisement
2100::56:a00:20ff:fea0:3390 -> caramba         ICMPv6 Echo request (ID: 1294 Sequence number: 0)
     caramba -> 2100::56:a00:20ff:fea0:3390    ICMPv6 Echo reply (ID: 1294 Sequence number: 0)

 

You can restrict capture of traffic addressed to the specified host by using the to or dst qualifier in front of the host filter:

# snoop to host caramba
Using device /dev/hme (promiscuous mode)
    schooner -> caramba      RPC R XID=1493500696 Success
    schooner -> caramba      RPC R XID=1493500697 Success
    schooner -> caramba      RPC R XID=1493500698 Success

 

Similarly you can restrict captured traffic to only packets originating from the specified host by using the from or src qualifier:

Code View: Scroll / Show All
# snoop from host caramba
Using device /dev/hme (promiscuous mode)
     caramba -> schooner     NFS C GETATTR3 FH=B083
     caramba -> schooner     TCP D=2049 S=1023     Ack=3647527137 Seq=2611841034 Len=0 Win=24820

 

Note that the host keyword is not required when the specified hostname does not conflict with the name of another snoop primitive.The previous snoop from host caramba command could have been invoked without the host keyword and it would have generated the same output:

Code View: Scroll / Show All
 
					# snoop from caramba 
Using device /dev/hme (promiscuous mode)
     caramba -> schooner     NFS C GETATTR3 FH=B083
     caramba -> schooner     TCP D=2049 S=1023     Ack=3647527137 Seq=2611841034 Len=0 Win=24820

 

For clarity, we use the host keyword throughout this book. Two or more filters can be combined by using the logical operators and and or :

# snoop -o /tmp/capture -c 20 from host caramba and rpc nfs 3
Using device /dev/hme (promiscuous mode)
20 20 packets captured

 

Snoop captures all NFS Version 3 packets originating at the host caramba. Here, snoop is invoked with the -c and -o options to save 20 filtered packets into the /tmp/capture file. We can later apply other filters during display time to further analyze the captured information. For example, you may want to narrow the previous search even further by only listing TCP traffic by using the proto filter:

# snoop -i /tmp/capture proto tcp
Using device /dev/hme (promiscuous mode)
  1   0.00000     caramba -> schooner    NFS C GETATTR3 FH=B083
  2   2.91969     caramba -> schooner    NFS C GETATTR3 FH=0CAE
  9   0.37944     caramba -> rea         NFS C FSINFO3 FH=0156
 10   0.00430     caramba -> rea         NFS C GETATTR3 FH=0156
 11   0.00365     caramba -> rea         NFS C ACCESS3 FH=0156 (lookup)
 14   0.00256     caramba -> rea         NFS C LOOKUP3 FH=F244 libc.so.1
 15   0.00411     caramba -> rea         NFS C ACCESS3 FH=772D (lookup)

 

Snoop reads the previously filtered data from /tmp/capture, and applies the new filter to only display TCP traffic. The resulting output is NFS traffic originating at the host caramba over the TCP protocol. We can apply a UDP filter to the same NFS traffic in the /tmp/capture file and obtain the NFS Version 3 traffic over UDP from host caramba without affecting the information in the /tmp/capture file:

# snoop -i /tmp/capture proto udp
Using device /dev/hme (promiscuous mode)
  1   0.00000      caramba -> rea          NFS C NULL3

 

So far, we’ve presented filters that let you specify the information you are interested in. Use the not operator to specify the criteria of packets that you wish to have excluded during capture. For example, you can use the not operator to capture all network traffic, except that generated by the remote shell:

Code View: Scroll / Show All
# snoop not port login
Using device /dev/hme (promiscuous mode)
      rt-086 -> BROADCAST        RIP R (25 destinations)
      rt-086 -> BROADCAST        RIP R (10 destinations)
     caramba -> schooner         NFS C GETATTR3 FH=B083
    schooner -> caramba          NFS R GETATTR3 OK
     caramba -> donald           NFS C GETATTR3 FH=00BD
    jamboree -> donald           NFS R GETATTR3 OK
     caramba -> donald           TCP D=2049 S=657     Ack=3855205229 Seq=2331839250 Len=0 Win=24820
     caramba -> schooner         TCP D=2049 S=1023    Ack=3647569565 Seq=2612134974 Len=0 Win=24820
     narwhal -> 224.2.127.254    UDP D=9875 S=32825 LEN=368

 

On multihomed hosts (systems with more than one network interface device), use the -d option to specify the particular network interface to snoop on:

snoop -d hme2

 

You can snoop on multiple network interfaces concurrently by invoking separate instances of snoop on each device. This is particularly useful when you don’t know what interface the host will use to generate or receive the requests. The -d option can be used in conjunction with any of the other options and filters previously described:

# snoop -o /tmp/capture-hme0 -d hme0 not port login &
# snoop -o /tmp/capture-hme1 -d hme1 not port login &

2.This article is from book <Managing NFS and NIS, Second Edition>

hpux tips

June 30th, 2013 No comments
tusc #like truss or strace
lsdev
swapinfo -tm #memory usage
/var/adm/syslog/syslog.log, /etc/shutdownlog,
/etc/rc.log
/var/adm/syslog/syslog.log #like /var/log/messages
/etc/shutdownlog
/var/adm/crash/crash.X
/etc/rc.config.d/netconf #the interfaces which are started at boot up
/opt/fcms/bin/fcdutil /dev/fcd0 #HP-ux HBA and driver info
swlist -l product | grep “Fibre Channel Driver” #HP-UX
/usr/sbin/swlist and swinstall
bdf #HP, report number of free disk blocks
Extend the logical volume: lvextend –L <new LV size in MB> /dev/vgxx/lvolXX
Extend the filesystem to use the space added: fsadm –F vxfs –b <new size in 1 KB sectors> <mount point>
/opt/fcms/bin/fcdutil /dev/fcd0 | grep “World Wide Name”
dlmsetconf, dlmcfgmgr
hrdconf #HP
model #find out what model of machine we’re on, like 9000/800/rp4440
/opt/ignite/bin/print_manifest #To display system information and configuration(model, memory, CPU, Storage, partitions, I/O devices, software installed, kernel parameters, ip address)
/usr/sbin/lanscan #lists all network adapters, deprecated(after this, use ifconfig <name of NIC> to check details). related commands lanadmin, linkloop, lan, nwmgr<this is recommended>
lanadmin -x 0 #PPA number
ioscan #HP, scan I/O system, scan newly added disks, check processor type etc
ioscan -f #all devices
ioscan –fknC fc #list HBA devices
/opt/fcms/bin/fcmsutil #Fibre Channel Mass Storage Utility Command, fcmsutil /dev/tdX -> Display HBA details
/sbin/rc3.d #run levels, All the scripts should take the appropriate action depending on the argument given. The stop script for a subsystem should be in the rc directory one run level below its start script, e.g. if the start script is in rc3.d then the stop script should be in rc2.d
/stand   #kernel and kernel configuration files
/usr/bin/bdf -l #FS, like df -k
/usr/sbin/sam #system admin tool
/usr/sbin/fsadm #linux belongs to lvm2(JBOD – Raid0)
/usr/sbin/{cstm,xstm,mstm} #Support Tools Manager,
/sbin/ipf #rules in /etc/opt/ipf/ipf.conf, ipf –Fa –f /etc/opt/ipf/ipf.conf to re-read rules file
/usr/lbin/modprpw #To unlock the account (if TCB is used) use: /usr/lbin/modprpw -l -k <loginid>
/opt/perf/bin/extract #performance monitoring
/etc/pam.conf
/etc/nsswitch.conf
/etc/opt/ldapux/ldapux_client.conf
/opt/ldapux/config/setup
nsquery passwd liandy
A Guide to HP-UX Document Collections HP documents
Categories: tips, Unix Tags:

AIX tips

June 30th, 2013 No comments
fuser -cuxk /oracle #kill all the process using filesystem /oracle
procstack #show current stack of a process
bootlist -m normal -o          # Lists the current bootlist
bootlist -m normal cd0 hdisk0  # To set cd0 and hdisk0 as first and second boot devices
bootlist -m service cd0 rmt0   # To change the bootlist for service mode
alog -L
alog -o -t boot
alog -L -t boot #find out the properties of boot log file
##Device Configuration Database(Predefined, Customized)
01. Available  - Device is ready and can be used
02. Defined    - Device is unavailable
03. Unknown    - Undefined
04. Stopped    - Configured but unavailable
lsdev
      -C  to list customized database
      -P  to list predefined database
      -c (class)
      -t (type)
      -s (subtype)
To list all customised devices ie installed
 # lsdev -C
To list all the Hard Drives in a system
 # lsdev -Cc disk
To list all the adapters in a sytem
 # lsdev -Cc adapter
lscfg -v  #list all installed devices in detail
lscfg -vpl fcs0<ent0> #find out the WWN, FRU #, firmware level of fibre adapter fcs0
entstat -d ent0 #link status, link speed and mac address and statistics of an Ethernet adapter ent0
##Setting multiple IP address for a single network card
 # ifconfig lo0 alias 195.60.60.1
 # ifconfig en0 alias <IPadress> netmask <net_mask>
/etc/rc.net, /etc/rc.tcpip #make the above permanent
lsattr -El ent0 -a media_speed -R #find out the possible media_speed values for ethernet card ent0
lsattr -El mem0 #find out the effective attribute of a device “mem0″
lsattr -El sys0 #list the defaults in the pre-defined db for device ent0
To change the maximum number of processes allowed per user
Find out the valid range of values using lsattr command
 # lsattr -l sys0 -a maxuproc -R
 40…131072 (+1)
Change the maxuproc value using chdev command
 # chdev -l sys0 -a maxuproc=10000
rmdev -l (device) -d #delete the device
To delete a static route manually
Syntax:- chdev -l inet0 -a delroute=<net>,<destination_address>,<Gate_way_address>,<Subnet_mask>
 # chdev -l inet0 -a delroute=’net’,’0.0.0.0′,’172.26.160.2′
To change the IP address of an interface manually
 # chdev -l en0 -a netaddr=192.168.123.1 -a netmask=255.255.255.0 -a state=up
To set the IP address initially
 # mktcpip -h <hostname> -a <ipaddress> -m <subnet_mask> -i <if_name> -n <NameServer_address>
   -d <domain_name> -g <gateway_address> -A no
##add device to system
To define a tape device
 # mkdev -d -c tape -t 8mm -s scsi -p scsi0  -w 5,0
To make the predefined rmt0 tape to available status
 # mkdev -l rmt0
##configure new devices using cfgmgr
cfgmgr -l fcs0 #configure detected devices attached to the fcs0 adapter
cfgmgr -i /tmp/drivers #cfgmgr -i /tmp/drivers
getconf -a
prtconf -c/m/s
bootinfo -K
mksysb/savevg/restore
sysdumpdev/sysdumpstart/snap/kdb
swapon/swapoff/lsps/chps/mkps/rmps, /etc/swapspaces, /etc/filesystems
LVM – lsvg/lspv/mkvg/mklv/logform/crfs/chfs/extendvg/mklvcopy/syncvf/bosboot/synclvodm/chpv
NIM – Network Installation Management
/etc/netsvc.conf #Name resolution order
##no command is used to change the network tuning parameters. ioo for IO tuning(aio, asynchronous IO), vmo for virtual memory manager parameters
To list the current network parameters / network options
 # no  -a
To enable IP forwarding
 # no -o “ipforwarding=1″
To make ipforwarding=1 permanent now and after reboot
 # no -p -o ipforwarding=1
###/etc/tunables/xxx, tuncheck/tunsave/tunrestore/tundefault
startsrc/lssrc, iptrace, tcpdump #The startsrc command sends the System Resource Controller (SRC) a request to start a subsystem or a group of subsystems, or to pass on a packet to the subsystem that starts a subserver.
##ODM, object data manager
/etc/objrepos
/usr/lib/objrepos
/usr/share/lib/objrepos
odmget CuDv #list all records with an Object Class CuDv
odmget -q “name=sys0 and attribute=maxuproc” CuAt
svmon -G #memory. pin(frames that cannot be swapped), pg space(paging space, ie swap)
pagesize
svmon -P 13548 -i 1 2 #monitor memory leak by  looking for processes whose working segment continually grows
trcon/filemon/trcstop #Most Active Logical/physical Volumes, most active Files
rmss #a means to simulate different sizes of real memory that are smaller than your actual machine
netpmon #network monitoring
##package management
oslevel -r/s/l xxxx/-g/-rq
###PATH
/etc/objrepos
/usr/lib/objrepos
/usr/share/lib/objrepos
lslpp -l [software name]
lslpp -f <fileset name> #display the names of all the files of fileset
lslpp -w /usr/sbin/nfsd #which fileset a file belongs to
##NFS
service portmap start
service nfs start
showmount -e localhost
/var/lib/nfs/
/etc/exports #/backup/downloads *(sync,ro,root_squash,wdelay), exportfs -a, exprotfs *:/backup/downloads
mount -fv -t nfs <xx> <dir> #check ports used
lslpp -ha #installation history of filesets
To list all installable software in media /dev/cd0
 installp [-L|-l] -d /dev/cd0
To cleanup all failed installtion
 installp -C
To install bos.net software (apply and commit) package with all pre-requisites from directory /tmp/net
 installp -acgx -d /tmp/net bos.net
To commit teh applied updates
 installp -cgx all
To remove bos.net package
 installp -ug bos.net
To find out whether a Fix is installed or not
 # instfix -i -k <APAR Number>
To list all the fixes that are installed on your system
 # instfix -i -v
To list filesets which are lesser than the specified maintenance level
 # instfix -ciqk 5100-04_AIX_ML | grep “:-:”
To install all filesets associated with fix Ix38794 from the tape
 # instfix  -k Ix38794  -d /dev/rmt0
To Display the entire list of fixes present on the media
 # instfix -T -d /dev/cd0
To confirm the AIX preventive maintenance level on your system
 # instfix -i | grep ML
 All filesets for 5.0.0.0_AIX_ML were found.
 All filesets for 5.1.0.0_AIX_ML were found.
 All filesets for 5.1.0.0_AIX_ML were found.
 All filesets for 5100-01_AIX_ML were found.
 All filesets for 5100-02_AIX_ML were found.
Updating the software to the latest level
01. Using smit
    # smit update_all
02. To update all filesets in a system using command line
    a. Create the list of filesets installed
       # lslpp -Lc | awk -F: ‘{print $2}’| tail -n +2 > /tmp/lslpp
    b. Update the softwares using installp command
       # installp -agxYd /dev/cd0 -e /tmp/<exclude_list> -f /tmp/lslpp
Another way of updating all the filesets
 # /usr/lib/instl/sm_inst installp_cmd  -acgNXY -d <localtion_of_updates> -f ‘_update_all’
For not committing and saving all replaced files
 # /usr/lib/instl/sm_inst installp_cmd  -agX -d <localtion_of_updates> -f ‘_update_all’
To list all the installed efixes on a system
 # emgr -l
To install a efix IY93496.070302.epkg.Z in /mnt directory
 # emgr -e /mnt/IY93496.070302.epkg.Z
inutoc
The inutoc command creates the .toc file in Directory. If a .toc file already exists, it is recreated with new information. The inutoc command adds table of contents entries in the .toc file for every installation image in Directory.
The installp command and the bffcreate command call this command automatically upon the creation or use of an installation image in a directory without a .toc file
To create a .toc file for the /tmp/images directory, enter:
 # inutoc /tmp/images
bffcreate
The bffcreate command creates an installation image file in backup file format (bff) to support software installation operations. It creates an installation image file from an installation image file on the specified installation media
To create an installation image file from the bos.net software package on the tape in the /dev/rmt0 tape drive and use /var/tmp as the working directory, type:
 # bffcreate  -d /dev/rmt0.1 -w /var/tmp bos.net
##security
/etc/security/environ
/etc/security/group
/etc/security/lastlog
/etc/security/limits
/etc/security/login.cfg
/usr/lib/security/mkuser.default
/etc/security/passwd
/etc/security/portlog
/etc/security/user
/etc/security/failedlogin
/etc/security/ldap/ldap.cnf
chsec -f /etc/security/limits -s joe -a cpu=3600 #change the CPU time limit of user joe to 1 hour
chuser rlogin=true smith #enable user smith to access this system remotely
pwdadm -c user1 # To reset the ADMCHG flag for the user user1<forces the user to change the password the next time a login command or an su command is given for the user>
who -a /etc/security/failedlogin # read failed login attempts
##LPAR and HMC<logical partition and hardware management console>
lsslot/hmcshutdown/chsysstate/lsrsrc/smtctl/vtmenu/mkvterm/rmvterm/lssysconff/lssysconn/ssysconn/lssyscfg/lsled/chled/lparstat -i/chhmc/mkvdev
##HACMP
HACMP Daemon
01. clstrmgr
02. clinfo
03. clmuxpd
04. cllockd
cllsgrp/clshowres/clRGmove/clfindres/clRGinfo/cldump/varyonvg/importvg/chvg/cllsappmon/clclear
##Storage
fcstat -D fcs0 | grep Attention #To find out the fiber channel link status
lsvpcfg #List all Vpath devices and their states
dpovgfix vg00 #fixe a DPO Vpath Volume group that has mixed vpath and hdisk volumes
###EMC powerpath
To configure all the emc hdisks, run emc_cfgmgr script. This script invokes the AIX cfgmgr tool to probe each adapter bus separately
To remove the Symettrix hdisks
 # lsdev -CtSYMM* -Fname | xargs -n1 rmdev -dl
To remove hdisks corresponding to CLARiiON devices
 # lsdev -CtCLAR* -Fname | xargs -n1 rmdev -dl
To probe all emc disks
 # inq
To set up multipathing to the root device
 # pprootdev on
To Remove all hdiskpower devices
 # lsdev -Ct power -c disk -F name | xargs -n1 rmdev -l
To find out which hdiskpower device contains hdsik132
 # powermt display dev=hdisk132
###HP Autopath
dlnkmgr view -drv
dlmrmdev #remove all the DLM drivers
dlmpr -a -c #To clear the SCSI reserves on the disks
/usr/DynamicLinkManager/drv/dlmfdrv.conf
###MPIO
To list all the paths which are in Enabled status
 # lspath -s ena -Fname -p fscsi0
 # chpath -s ena -l hdisk0
 paths Enabled
To list all available disks and their paths
  # lspath | sort +1
To list all disks which paths are in failed state
 # lspath -s failed
To list all disks which paths are in Defined state
 # lspath -s defined
To remove a path
 rmpath -dl <disk_name> -p <parent> -w <connection>
 rmpath -dl hdisk3 -p fscsi0 -w 5005076801105daf,1000000000000
Categories: tips, Unix Tags:

solaris tips

June 30th, 2013 No comments
pkginfo -l SUNWcsu
mdb & kmdb http://docs.oracle.com/cd/E19082-01/817-2543/index.html
echo ::memstat | mdb -k #memory usage profile
echo ::kmastat | mdb -k
echo “::threadlist -v” | mdb -k #collect the stack trace of all threads in mdb
truss -p <pid>/ndd/etc/system
pstack /var/core/core_doxerorg_vxconfigd_0_0_1343173375_140 #print a stack trace of running processes, or <pid>, /var/crash
/var/cron/log #solaris 10 cronlog,
###EFI SMI label http://www.chinaunix.net/old_jh/6/955384.html
###solaris proc tools
cd /proc/ ; for i in *; do echo — process $i —; pfiles $i | grep -i “port: 11961″; done
pldd 578 #dynamic libraries linked into the process
eeprom use-nvramrc? #whether nvramrc is enabled
eeprom nvramrc #aliases. halt will go to OK mode. type sync there then it will sync disks and then generate a static coredump<if faied, try savecore -L(live) in OK mode/or in OS>
eeprom auto-boot? #auto boot
mpathadm list mpath-support
/etc/dfs/dfstab #like /etc/mtab, also in HPUX
prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s – /dev/rdsk/c1t1d0s2 #solaris,copy disk head info from c1t0d0 to c1t1d0
isainfo -kv #64bits or 32bits, pagesize,
getconf PAGESIZE #linux
grep -i /etc/network path_to_inst #find out the network cards available on the system, cat /etc/path_to_inst | egrep -i ‘eri|ge|ce|qfe|hme’
sysdef #solaris, output system definition
mpstat 2 #solaris, per-processor,cpu
prtdiag -v #solaris hardware type, prtconf
dladm #solaris administer data links
dladm show-dev/show-link/show-aggr/ #two dev can be a link aggr<bonding, on one switch, increase link speed>
dladm show-dev -s e1000g1
ifconfig e1000g1 plumb
ifconfig vmxnet0 10.180.3.218 netmask 255.255.255.0 up #at last, config /etc/hostname.vmxnet0
ifconfig vmxnet0 down
dladm create-aggr -l passive -d e1000g2 -d e1000g3 1(first unplumb e1000g2/3)
ifconfig aggr1 plumb
ifconfig aggr1 10.180.3.220 netmask 255.255.255.0 up
/usr/sbin/cfgadm -c configure /dev/cfg/c2::5006048452a72687
/usr/sbin/fcinfo hba-port -l #solaris, hba, like Qlogic<chibrat5>, Emulex<upora06a>
fcinfo remote-port -l -p 210000e08b18024f #Lists the remote-port information
/{usr/,}opt/FJSV*
/opt/FJSVmadm/sbin/{madmin,hrdconf -l } #madmin is an menu-driven interactive utility that allows you to perform various hardware-related diagnostics and maintenance.
/opt/FJSVsnap/bin/fjsnap -a output #like sun explorer, add -C to include crashdump information
/opt/FJSVhwr/sbin/fjprtdiag -v
/opt/FJSVcsl/bin//mainmenu #hardware etc on SMC and partition poweron/poweroff(PrimePower System Management Controller)
/opt/FJSVcsl/bin/get_console -w -n <partition_name> #partition_name in /etc/hosts #maybe /opt/scripts/bin/console.sh 0 FORCE, if you know the number(through mainmenu)
##to get into OK prompt
a. ctrl+] to get the telnet prompt
b. From telnet prompt, type “send break”  to get OK prompt
ndd #change UDP parameters etc
solaris IPMP configuration solaris-IPMP.pdf

zpool create tank c4t0d0
zpool list
zpool list tank
zpool get autoexpand tank
zpool replace tank c4t0d0 c1t13d0
zpool list tank
zpool set autoexpand=on tank
zpool list tank
zfs userspace tank
zfs groupspace tank
zpool status
zpool status -x #all pools are healthy
zpool history
zpool history -l
zfs mount -a
zfs get mountpoint,compression tank
zfs create -o compression=gzip tank/home
zfs create -o compression=gzip tank/home/firsttry
zfs create -o compression=gzip -o mountpoint=/export/secondtry tank/home/secondtry
##the below is equal to:
##zfs create tank/home/secondtry
##zfs set mountpoint=/export/secondtry tank/home/secondtry
zfs get -s local all
zpool get all tank
zfs list
zfs mount zones/test #Mount the ZFS
zfs unmount zones/test/
zpool status -x
zpool status
zpool clear tank
/usr/lib/pool/poold #start poold manually
pgrep -l poold #1333 poold

/etc/svc/volatile #logs related to current services
fmadm faulty #fault management. fmd, fmdump, fmstat. FRU(Field replaceable unit)
fmadm #fault management configuration tool, fmadm faulty -a
fmadm config
fmadm faulty #show faults in fma
fmstat
fmstat -m zfs-retire 2 5
fmdump
fmdump -vv -u 177b4b48-8ed1-ea7a-e6f3-feed10dd4c38
fmdump -Vu 6252dd23-4397-cbda-8c72-8774fd175bc1
fmdump /var/fm/fmd/errlog
svcs -a | grep -i cron
svcs cron
svcs -l ipfilter #dependency, dependent
svcs -D ipfilter #dependent
svcs -d ipfilter #dependency
svcs -a|grep lrc #smf can monitor init.d scripts but can not manage them
svcadm enable -r ipfilter #boot cascade
svcadm enable -rt ipfilter #single user mode
svcadm restart cron
svcadm refresh #make snapshot working
##recover
svcs -p telnet#check relationship between services and processes, may need pkill -9
svcadm clear telnet #check /var/svc/log
#recover snapshot
svccfg ->select network/ipfilter:default -> listsnap ->help ->help revert ->revert start ->quit
svcadm refresh network/ipfilter:default #refresh /etc/svc/repository.db
svcadm restart network/ipfilter:default #restart
svcadm clear svc:/system/filesystem/local:default
svcs -xv FMRI#check for reason
svccfg -s network/ipfilter:default #unselect,quit
svccfg export pfil >/tmp/pfil.desc
pargs -e `pgrep -f cron`
svccfg -s system/cron setenv LD_PRELOAD libumem.so
svccfg -s system/cron setenv UMEM_DEBUG default
svcadm refresh system/cron
svcadm restart system/cron
pargs -e `pgrep -f cron`
inetadm -l telnet | grep tcp_trace
inetadm -m telnet tcp_trace=TRUE
inetdadm -l ftp|grep exec
inetadm -m ftp exec=”/usr/sbin/in.ftpd -a -l”
inetconv -e -i /etc/inet/inetd.conf
pkill -HUP inetd
/lib/svc/method/sshd start #man smf_method
svccfg import
/lib/svc/bin/restore_repository
/var/svc/profile
svcs |grep milestone
svcadm -v milestone -d multi-user-server:default #/var/svc/manifest/milestone/multi-user-server.xml
RBAC #man smf_security
svcadm enable apache2 # manipulate service instances
inetadm – bserve and configure services that are controlled by inetd. inetadm -?, inetconv
svcprop – retrieve values of service configuration properties
FMRI:Fault Management Resource Identifier
svccfg delete /network/http:apache2
ups auxww|grep fmd
-bash-3.00# svccfg -s smtp
svc:/network/smtp> list
:properties
sendmail
svc:/network/smtp> select sendmail
svc:/network/smtp:sendmail> list
:properties
svc:/network/smtp:sendmail> listprop *exec
start/exec astring “/lib/svc/method/smtp-sendmail start”
stop/exec astring “/lib/svc/method/smtp-sendmail stop %{restarter/contract}”
refresh/exec astring “/lib/svc/method/smtp-sendmail refresh”
More http://www.princeton.edu/~unix/Solaris/troubleshoot/smf.html
/lib/svc/method/fs-*

http://www.sunfreeware.com

http://www.opencsw.org #/opt/csw/bin:/opt/csw/sbin

http://wesunsolve.net/

http://www.unixpackages.com/

/usr/sbin/pkgchk -l -p /usr/sbin/fcinfo #SUNWfcprt
pkginfo #/var/sadm/pkg/pkgname/pkginfo
pkgmap #/var/sadm/install/contents
pkgtrans
pkginfo -d ./top-3.5-sol10-intel-local
pkgadd -d . topxxx
pkgadd -d ./xxx
pkgadd -d ./top-xxx -s /var/spool/pkg SMCtop
pkgtrans ./topxxx /var/spool/pkg
pkgadd -d . -s spool
pkginfo -d spool SMCtop
pkgrm -s spool SMCtop
grep showrev /var/sadm/install/contents
pkginfo|grep -i top
root@beisoltest02 ~ # pkgadd
pkgadd: ERROR: no packages were found in </var/spool/pkg>
less /etc/apache/README.Solaris
pkgrm SMCtop
pkgchk SMCtop
pkgchk -p /usr/local/doc/top/README
pkgparam SMCtop PATCHLIST
root:/usr/local/src# wget http://www.sunfreeware.com/BOLTpget.pkg
root:/usr/local/src# pkgadd -d BOLTpget.pkg all

zoneadm list -civ #-v,verbose,zfs list
zonecfg -z andyred #interactive configuration
zoneadm -z andyred boot #boot
zlogin andyred shutdown -i5 -g0 -y #shutdown
zoneadm -z andyred halt #halt,no shutdown scripts will be run
zoneadm -z andyred uninstall -F #delete
zoneadm -z test detach #detach the zone
zoneadm -z test attach -u
zlogin -C andyred #login zone from global zone
zonename #which zone am I in
netstat -p #non-global zone to get global zone name

Categories: tips, Unix Tags:

resolved – how to check nfs version in linux

September 11th, 2012 No comments

To know nfs version in linux/solaris:

  • On the nfs server side, you can run a nfsstat -s to check. The used version of nfs will have data summary other than 0% ones, as the following:

[email protected]# nfsstat -s
Server rpc stats:
calls badcalls badauth badclnt xdrcall
28 0 0 0 0

Server nfs v3:
null getattr setattr lookup access readlink
3 11% 4 14% 0 0% 1 3% 4 14% 0 0%
read write create mkdir symlink mknod
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
0 0% 0 0% 0 0% 0 0% 0 0% 2 7%
fsstat fsinfo pathconf commit
9 33% 4 14% 0 0% 0 0%

  • On the nfs server, we can also have a checking on what versions(2/3/4) and transport protocols(tcp/udp) the nfs supported with the command “rpcinfo -p localhost|grep nfs”:

root@doxer# rpcinfo -p localhost|grep nfs
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs

  • On the nfs client hosts, you can run a nfsstat -c to check the version the client is using. As always, the used version of nfs will have data summary other than 0% ones, as the following:

[email protected]# nfsstat -c

Client rpc:
Connection oriented:
calls badcalls badxids timeouts newcreds badverfs
1219760 322812 0 0 0 0
timers cantconn nomem interrupts
0 322808 0 0
Connectionless:
calls badcalls retrans badxids timeouts newcreds
0 0 0 0 0 0
badverfs timers nomem cantsend
0 0 0 0

Client nfs:
calls badcalls clgets cltoomany
753081 28 753081 0
Version 2: (0 calls)
null getattr setattr root lookup readlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
read wrcache write create remove rename
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir statfs
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
Version 3: (748700 calls)
null getattr setattr lookup access readlink
0 0% 140588 18% 61939 8% 184611 24% 150266 20% 8 0%
read write create mkdir symlink mknod
35415 4% 58540 7% 11703 1% 562 0% 248 0% 0 0%
remove rmdir rename link readdir readdirplus
3264 0% 0 0% 9 0% 0 0% 1165 0% 1219 0%
fsstat fsinfo pathconf commit
33435 4% 7160 0% 3309 0% 55259 7%

Client nfs_acl:
Version 2: (0 calls)
null getacl setacl getattr access
0 0% 0 0% 0 0% 0 0% 0 0%
Version 3: (4382 calls)
null getacl setacl
0 0% 4382 100% 0 0%

  • Also, you can run nfsstat -m on nfs client hosts to print information about each of the mounted NFS file systems(the output info has nfs version indicated also):

[email protected] # nfsstat -m
/apps/uriman/tmp from doxer:/export/was/trncsc_cell_urimantmp
Flags: vers=3,proto=tcp,sec=none,hard,intr,link,symlink,acl,rsize=32768,wsize=32768,retrans=5,timeo=600
Attr cache: acregmin=3,acregmax=60,acdirmin=30,acdirmax=60

PS:

  • Here’s more about output analytic of nfsstat:

The client- and server-side implementations of NFS compile per-call statistics of NFS service usage at both the RPC and application layers. nfsstat -c displays the client-side statistics while nfsstat -s shows the server tallies. With no arguments, nfsstat prints out both sets of statistics:

Code View: Scroll / Show All
% nfsstat -s 
Server rpc:
Connection oriented:
calls       badcalls    nullrecv    badlen      xdrcall     dupchecks   
10733943    0           0           0           0           1935861     
dupreqs     
0           
Connectionless:
calls       badcalls    nullrecv    badlen      xdrcall     dupchecks   
136499      0           0           0           0           0           
dupreqs     
0           

Server nfs:
calls       badcalls    
10870161    14          
Version 2: (1716 calls)
null        getattr     setattr     root        lookup      readlink    
48 2%       0 0%        0 0%        0 0%        1537 89%    13 0%       
read        wrcache     write       create      remove      rename      
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%        
link        symlink     mkdir       rmdir       readdir     statfs      
0 0%        0 0%        0 0%        0 0%        111 6%      7 0%        
Version 3: (10856042 calls)
null        getattr     setattr     lookup      access      readlink    
136447 1%   4245200 39% 95412 0%    1430880 13% 2436623 22% 74093 0%    
read        write       create      mkdir       symlink     mknod       
376522 3%   277812 2%   165838 1%   25497 0%    24480 0%    0 0%        
remove      rmdir       rename      link        readdir     readdirplus 
359460 3%   33293 0%    8211 0%     69484 0%    69898 0%    876367 8%   
fsstat      fsinfo      pathconf    commit      
1579 0%     7698 0%     4253 0%     136995 1%   
Server nfs_acl:
Version 2: (2357 calls)
null        getacl      setacl      getattr     access      
0 0%        5 0%        0 0%        2170 92%    182 7%      
Version 3: (10046 calls)
null        getacl      setacl      
0 0%        10039 99%   7 0%

 

The server-side RPC fields indicate if there are problems removing the packets from the NFS service end point. The kernel reports statistics on connection-oriented RPC and connectionless RPC separately. The fields detail each kind of problem:

calls

The NFS calls value represents the total number of NFS Version 2, NFS Version 3, NFS ACL Version 2 and NFS ACL Version 3 RPC calls made to this server from all clients. The RPC calls value represents the total number of NFS, NFS ACL, and NLM RPC calls made to this server from all clients. RPC calls made for other services, such as NIS, are not included in this count.

badcalls

These are RPC requests that were rejected out of hand by the server’s RPC mechanism, before the request was passed to the NFS service routines in the kernel. An RPC call will be rejected if there is an authentication failure, where the calling client does not present valid credentials.

nullrecv

Not used in Solaris. Its value is always 0.

badlen/xdrcall

The RPC request received by the server was too short (badlen) or the XDR headers in the packet are malformed (xdrcall ). Most likely this is due to a malfunctioning client. It is rare, but possible, that the packet could have been truncated or damaged by a network problem. On a local area network, it’s rare to have XDR headers damaged, but running NFS over a wide-area network could result in malformed requests. We’ll look at ways of detecting and correcting packet damage on wide-area networks in Section 18.4.

dupchecks/dupreqs

The dupchecksfield indicates the number of RPC calls that were looked up in the duplicate request cache. The dupreqs field indicates the number of RPC calls that were actually found to be duplicates. Duplicate requests occur as a result of client retransmissions. A large number of dupreqs usually indicates that the server is not replying fast enough to its clients. Idempotent requests can be replayed without ill effects, therefore not all RPCs have to be looked up on the duplicate request cache. This explains why the dupchecks field does not match the calls field.

The statistics for each NFS version are reported independently, showing the total number of NFS calls made to this server using each version of the protocol. A version-specific breakdown by procedure of the calls handled is also provided. Each of the call types corresponds to a procedure within the NFS RPC and NFS_ACL RPC services.

The null procedure is included in every RPC program for pinging the RPC server. The null procedure returns no value, but a successful return from a call to null ensures that the network is operational and that the server host is alive. rpcinfo calls the null procedure to check RPC server health. The automounter (see Chapter 9) calls the null procedure of all NFS servers in parallel when multiple machines are listed for a single mount point. The automounter and rpcinfo should account for the total null calls reported by nfsstat.

Client-side RPC statistics include the number of calls of each type made to all servers, while the client NFS statistics indicate how successful the client machine is in reaching NFS servers:

Code View: Scroll / Show All
% nfsstat -c 
Client rpc:
Connection oriented:
calls       badcalls    badxids     timeouts    newcreds    badverfs    
1753584     1412        18          64          0           0           
timers      cantconn    nomem       interrupts  
0           1317        0           18          
Connectionless:
calls       badcalls    retrans     badxids     timeouts    newcreds    
12443       41          334         80          166         0           
badverfs    timers      nomem       cantsend    
0           4321        0           206         

Client nfs:
calls       badcalls    clgets      cltoomany   
1661217     23          1661217     3521        
Version 2: (234258 calls)
null        getattr     setattr     root        lookup      readlink    
0 0%        37 0%       0 0%        0 0%        184504 78%  811 0%      
read        wrcache     write       create      remove      rename      
49 0%       0 0%        24301 10%   3 0%        2 0%        0 0%        
link        symlink     mkdir       rmdir       readdir     statfs      
0 0%        0 0%        12 0%       12 0%       24500 10%   27 0%       
Version 3: (1011525 calls)
null        getattr     setattr     lookup      access      readlink    
0 0%        417691 41%  14598 1%    223609 22%  47438 4%    695 0%      
read        write       create      mkdir       symlink     mknod       
56347 5%    221334 21%  1565 0%     106 0%      48 0%       0 0%        
remove      rmdir       rename      link        readdir     readdirplus 
807 0%      14 0%       676 0%      24 0%       475 0%      5204 0%     
fsstat      fsinfo      pathconf    commit      
8 0%        10612 1%    95 0%       10179 1%    

Client nfs_acl:
Version 2: (411477 calls)
null        getacl      setacl      getattr     access      
0 0%        181399 44%  0 0%        185858 45%  44220 10%   
Version 3: (3957 calls)
null        getacl      setacl      
0 0%        3957 100%   0 0%

 

In addition to the total number of NFS calls made and the number of rejected NFS calls (badcalls), the client-side statistics indicate if NFS calls are being delayed due to a lack of client RPC handles. Client RPC handles are opaque pointers used by the kernel to hold server connection information. In SunOS 4.x, the number of client handles was fixed, causing the NFS call to block until client handles became available. In Solaris, client handles are allocated dynamically. The kernel maintains a cache of up to 16 client handles, which are reused to speed up communication with the server. The clgets count indicates the number of times a client handle has been requested. If the NFS call cannot find an unused client handle in the cache, it will not block until one frees up. Instead, it will create a brand new client handle and proceed. This count is reflected by cltoomany. The client handle is destroyed when the reply to the NFS call arrives. This count is of little use to system administrators since nothing can be done to increase the cache size and reduce the number of misses.

Included in the client RPC statistics are counts for various failures experienced while trying to send NFS requests to a server:

calls

Total number of calls made to all NFS servers.

badcalls

Number of RPC calls that returned an error. The two most common RPC failures are timeouts and interruptions, both of which increment the badcalls counter. The connection-oriented RPC statistics also increment the interrupts counter. There is no equivalent counter for connectionless RPC statistics. If a server reply is not received within the RPC timeout period, an RPC error occurs. If the RPC call is interrupted, as it may be if a filesystem is mounted with the intr option, then an RPC interrupt code is returned to the caller. nfsstat also reports the badcalls count in the NFS statistics. NFS call failures do not include RPC timeouts or interruptions, but do include other RPC failures such as authentication errors (which will be counted in both the NFS and RPC level statistics).

badxids

The number of bad XIDs. The XID in an NFS request is a serial number that uniquely identifies the request. When a request is retransmitted, it retains the same XID through the entire timeout and retransmission cycle. With the Solaris multithreaded kernel, it is possible for the NFS client to have several RPC requests outstanding at any time, to any number of NFS servers. When a response is received from an NFS server, the client matches the XID in the response to an RPC call in progress. If an XID is seen for which there is no active RPC call — because the client already received a response for that XID — then the client increments badxid. A high badxid count, therefore, indicates that the server is receiving some retransmitted requests, but is taking a long time to reply to all NFS requests. This scenario is explored in Section 18.1.

timeouts

Number of calls that timed out waiting for a server’s response. For hard-mounted filesystems, calls that time out are retransmitted, with a new timeout period that may be longer than the previous one. However, calls made on soft-mounted filesystems may eventually fail if the retransmission count is exceeded, so that the call counts obey the relationship:

timeout + badcalls >= retrans

 

The final retransmission of a request on a soft-mounted filesystem increments badcalls (as previously explained). For example, if a filesystem is mounted with retrans=5, the client reissues the same request five times before noting an RPC failure. All five requests are counted in timeout, since no replies are received. Of the failed attempts, four are counted in the retrans statistic and the last shows up in badcalls.

newcreds

Number of times client authentication information had to be refreshed. This statistic only applies if a secure RPC mechanism has been integrated with the NFS service.

badverfs

Number of times server replies could not be authenticated. The number of times the client could not guarantee that the server was who it says it was. These are likely due to packet retransmissions more than security breaches, as explained later in this section.

timers

Number of times the starting RPC call timeout value was greater than or equal to the minimum specified timeout value for the call. Solaris attempts to dynamically tune the initial timeout based on the history of calls to the specific server. If the server has been sluggish in its reponse to this type of RPC call, the timeout will be greater than if the server had been replying normally. It makes sense to wait longer before retransmitting for the first time, since history indicates that this server is slow to reply. Most client implementations use an exponential back-off strategy that doubles or quadruples the timeout after each retransmission up to an implementation-specific limit.

cantconn

Number of times a connection-oriented RPC call failed due to a failure to establish a connection to the server. The reasons why connections cannot be created are varied; one example is the server may not be running the nfsd daemon.

nomem

Number of times a call failed due to lack of resources. The host is low in memory and cannot allocate enough temporary memory to handle the request.

interrupts

Number of times a connection-oriented RPC call was interrupted by a signal before completing. This counter applies to connection-oriented RPC calls only. Interrupted connection and connectionless RPC calls also increment badcalls.

retrans

Number of calls that were retransmitted because no response was received from the NFS server within the timeout period. This is only reported for RPC over connectionless transports. An NFS client that is experiencing poor server response will have a large number of retransmitted calls.

cantsend

Number of times a request could not be sent. This counter is incremented when network plumbing problems occur. This will mostly occur when no memory is available to allocate buffers in the various network layer modules, or the request is interrupted while the client is waiting to queue the request downstream. Thenomem and interrupts counters report statistics encountered in the RPC software layer, while the cantsend counter reports statistics gathered in the kernel TLI layer.

The statistics shown by nfsstat are cumulative from the time the machine was booted, or the last time they were zeroed using nfsstat -z:

nfsstat -z      Resets all counters.
nfsstat -sz     Zeros server-side RPC and NFS statistics.
nfsstat -cz     Zeros client-side RPC and NFS statistics. 
nfsstat -crz    Zeros client-side RPC statistics only.

 

Only the superuser can reset the counters.

nfsstat provides a very coarse look at NFS activity and is limited in its usefulness for resolving performance problems. Server statistics are collected for all clients, while in many cases it is important to know the distribution of calls from each client. Similarly, client-side statistics are aggregated for all NFS servers.

However, you can still glean useful information from nfsstat. Consider the case where a client reports a high number of bad verifiers. The high badverfs count is most likely an indication that the client is having to retransmit its secure RPC requests. As explained in Section 12.1, every secure RPC call has a unique credential and verifier with a unique timestamp (in the case of AUTH_DES) or a unique sequence number (in the case of RPCSEC_GSS). The client expects the server to include this verifier (or some form of it) in its reply, so that the client can verify that it is indeed obtaining the reply from the server it called.

Consider the scenario where the client makes a secure RPC call using AUTH_DES, using timestamp T1 to generate its verifier. If no reply is received within the timeout period, the client retransmits the request, using timestamp T1+delta to generate its verifier (bumping up the retrans count). In the meantime, the server replies to the original request using timestamp T1 to generate its verifier:

Code View: Scroll / Show All
RPC call (T1)               --->
                         ** time out **
RPC call (retry: T1+delta)  --->
                            <--- Server reply to first RPC call (T1 verifier)

 

The reply to the client’s original request will cause the verifier check to fail because the client now expects T1+delta in the verifier, not T1. This consequently bumps up thebadverf count. Fortunately, the Solaris client will wait for more replies to its retransmissions and, if the reply passes the verifier test, an NFS authentication error will be avoided. Bad verifiers are not a big problem, unless the count gets too high, especially when the system starts experiencing NFS authentication errors. Increasing the NFS timeoon the mount or automounter map may help alleviate this problem. Note also that this is less of a problem with TCP than UDP. Analysis of situations such as this will be the focus of Section 16.1Chapter 17, and Chapter 18.

For completeness, we should mention that verifier failures can also be caused when the security content expires before the response is received. This is rare but possible. It usually occurs when you have a network partition that is longer than the lifetime of the security context. Another cause might be a significant time skew between the client and server, as well as a router with a ghost packet stored, that fires after being delayed for a very long time. Note that this is not a problem with TCP.

Categories: Linux, Networking Security, Systems, Unix Tags:

solaris ipmp bonding experiment

August 17th, 2012 No comments

[email protected] ~ # cat /etc/hosts
#
# Internet host table
#
::1 localhost
127.0.0.1 localhost
10.240.3.221 host1-e1000g2
10.240.3.223 host1-e1000g3
10.240.3.222 host1

[email protected] ~ # cat /etc/hostname.e1000g2
host1-e1000g2 group bak deprecated -failover netmask + broadcast + up
addif host1 netmask + broadcast + up
[email protected] ~ #
[email protected] ~ # cat /etc/hostname.e1000g3
host1-e1000g3 group bak deprecated -failover standby netmask + broadcast + up
[email protected] ~ #
[email protected] ~ # cat /etc/default/mpathd
#
#pragma ident “@(#)mpathd.dfl 1.2 00/07/17 SMI”
#
# Time taken by mpathd to detect a NIC failure in ms. The minimum time
# that can be specified is 100 ms.
#
FAILURE_DETECTION_TIME=10000
#
# Failback is enabled by default. To disable failback turn off this option
#
FAILBACK=yes
#
# By default only interfaces configured as part of multipathing groups
# are tracked. Turn off this option to track all network interfaces
# on the system
#
TRACK_INTERFACES_ONLY_WITH_GROUPS=yes

 

After this, reboot host(ensure /usr/lib/inet/in.mpathd is running)

[email protected] ~ # ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 10.240.3.206 netmask ffffff00 broadcast 10.240.3.255
ether 0:c:29:d3:d1:68
e1000g2: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER > mtu 1500 index 3
inet 10.240.3.221 netmask ff000000 broadcast 10.255.255.255
groupname bak
ether 0:c:29:d3:d1:86
e1000g2:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 10.240.3.222 netmask ff000000 broadcast 10.255.255.255
e1000g3: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVE R,STANDBY,INACTIVE> mtu 1500 index 4
inet 10.240.3.223 netmask ff000000 broadcast 10.255.255.255
groupname bak
ether 0:c:29:d3:d1:90
[email protected] ~ # if_mpadm -d e1000g2 #(detach or offline an interface. a networking blip will occur here, but soon recover itself)
[email protected] ~ #
[email protected] ~ # ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 10.240.3.206 netmask ffffff00 broadcast 10.240.3.255
ether 0:c:29:d3:d1:68
e1000g2: flags=89040842<BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,OFFLINE> mtu 1500 index 3
inet 10.240.3.221 netmask ff000000 broadcast 10.255.255.255
groupname bak
ether 0:c:29:d3:d1:86
e1000g3: flags=29040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY> mtu 1500 index 4
inet 10.240.3.223 netmask ff000000 broadcast 10.255.255.255
groupname bak
ether 0:c:29:d3:d1:90
e1000g3:1: flags=21000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,STANDBY> mtu 1500 index 4
inet 10.240.3.222 netmask ff000000 broadcast 10.255.255.255
[email protected] ~ # if_mpadm -r e1000g2 #(reattach or online an interface that has been offlined with -d)
[email protected] ~ # tail /var/adm/messages
Aug 17 03:31:11 doxer.org at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:318)
Aug 17 03:31:11 doxer.org … 34 more
Aug 17 03:31:11 doxer.org root: [ID 702911 user.crit] => com.sun.patchpro.cli.PatchServices@910040 <=Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Aug 17 03:31:11 doxer.org at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)
Aug 17 03:31:11 doxer.org at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)
Aug 17 03:31:11 doxer.org at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:318)
Aug 17 03:31:11 doxer.org … 34 more
Aug 17 03:44:57 doxer.org snmpXdmid: [ID 290637 daemon.error] Unable to connect to snmpdx
Aug 17 04:17:19 doxer.org in.mpathd[188]: [ID 832587 daemon.error] Successfully failed over from NIC e1000g2 to NIC e1000g3
Aug 17 04:17:48 doxer.org in.mpathd[188]: [ID 620804 daemon.error] Successfully failed back to NIC e1000g2
[email protected] ~ # ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 10.240.3.206 netmask ffffff00 broadcast 10.240.3.255
ether 0:c:29:d3:d1:68
e1000g2: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 10.240.3.221 netmask ff000000 broadcast 10.255.255.255
groupname bak
ether 0:c:29:d3:d1:86
e1000g2:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 10.240.3.222 netmask ff000000 broadcast 10.255.255.255
e1000g3: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 4
inet 10.240.3.223 netmask ff000000 broadcast 10.255.255.255
groupname bak
ether 0:c:29:d3:d1:90
[email protected] ~ # ifconfig e1000g2 down
[email protected] ~ # ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 10.240.3.206 netmask ffffff00 broadcast 10.240.3.255
ether 0:c:29:d3:d1:68
e1000g2: flags=9040842<BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 10.240.3.221 netmask ff000000 broadcast 10.255.255.255
groupname bak
ether 0:c:29:d3:d1:86
e1000g2:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 10.240.3.222 netmask ff000000 broadcast 10.255.255.255
e1000g3: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 4
inet 10.240.3.223 netmask ff000000 broadcast 10.255.255.255
groupname bak
ether 0:c:29:d3:d1:90
[email protected] ~ # ping 10.240.3.221
^C
[email protected] ~ # ping 10.240.3.223
10.240.3.223 is alive
[email protected] ~ # ifconfig e1000g2 up
[email protected] ~ #
[email protected] ~ #
[email protected] ~ # tail /var/adm/messages
Aug 17 03:31:11 doxer.org … 34 more
Aug 17 03:31:11 doxer.org root: [ID 702911 user.crit] => com.sun.patchpro.cli.PatchServices@910040 <=Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Aug 17 03:31:11 doxer.org at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)
Aug 17 03:31:11 doxer.org at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)
Aug 17 03:31:11 doxer.org at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:318)
Aug 17 03:31:11 doxer.org … 34 more
Aug 17 03:44:57 doxer.org snmpXdmid: [ID 290637 daemon.error] Unable to connect to snmpdx
Aug 17 04:17:19 doxer.org in.mpathd[188]: [ID 832587 daemon.error] Successfully failed over from NIC e1000g2 to NIC e1000g3
Aug 17 04:17:48 doxer.org in.mpathd[188]: [ID 620804 daemon.error] Successfully failed back to NIC e1000g2
Aug 17 04:18:51 doxer.org in.mpathd[188]: [ID 975029 daemon.error] No test address configured on interface e1000g2; disabling probe-based failure detection on it

PS:

1.IPMP(bonding) and Link aggregation(LACP) are different things. Link aggregations(or trunk) provide high availability and higher throughput by aggregating multiple interfaces at the MAC layer. IP Multipathing (IPMP, or bonding) provides features such as higher availability at the IP layer. If you have 4 NICs, you can aggregate 2 nics and bonded them. This way you’ll have 2 gig throughput and protect switch and nic level failures. (ipmp or bonding works at IP layer. If you only want HA<failover>, you can use bonding/IPMP; but if you want HP<load balance> except for HA, then you should set up ether channel on switch.)

2.For more infomation about solaris IPMP, you may refer to the following pdf file solaris IPMP bonding.pdf

Categories: Networking Security, Unix Tags:

resolved – aix create and remove swap space

July 14th, 2012 No comments

To add a paging space “paging0″

  • Create a new LV for paging space

mklv -t paging -y paging0 rootvg 10

  • Add the entry in /etc/swapspaces to activate the paging space during next reboot

chps -a y paging0

  • Activate the paging space

swapon /dev/paging0

To remove an active paging space “paging00″

  • Deactivate the paging space using swapoff commnad

swapoff /dev/paging00

  • remove the paging space using rmps command

rmps paging00

  • Remove the entry from /etc/swapspaces so that it is not activated during next reboot

chps -a n paging00

Categories: Unix Tags:

re-ip on solaris server howto – change ip netmask defaultrouter gateway

May 18th, 2012 No comments

To change ip/netmask/defaultrouter/gateway on solaris 10 or solaris 9 server permanently, you need care for files below:

/etc/hosts -> /etc/inet/hosts
/etc/hostname.<tags of your interface>
/etc/inet/netmasks
/etc/defaultrouter

Let’s assume that the new ip address is 101.139.1.151, new netmask is 255.255.254.0, new gateway is 101.139.1.254, new broadcast address is 101.139.1.255, here goes the steps:
1)change /etc/hosts(or /etc/inet/hosts which of them are the same file)
101.139.1.151 <tag for your server’s ip address>

2)change defaultrouter in /etc/defaultrouter:
101.139.1.254 /etc/defaultrouter

3)change /etc/hostname.<tags of your interface>(this step may not needed):

4)change netmask in /etc/inet/netmasks:
You’ll need first calculate network address from the given ipaddress(101.139.1.151) and netmask address(255.255.254.0). You can calculate it by hand(refer to this article http://www.doxer.org/learn-linux/basic-knowledge-for-netmask-hexadecimal-decimal-binary-netmask-cidr-calculator/), but I would prefer to use ipcalc:
[root@doxer~]# ipcalc -pnbm 101.139.1.151 255.255.254.0
NETMASK=255.255.254.0
PREFIX=23
BROADCAST=101.139.1.255
NETWORK=101.139.0.0

So from the output, you’d know that the network address is 101.139.0.0. Then add a line to /etc/inet/netmasks with format <network address> <netmask address>:
101.139.0.0 255.255.254.0

PS:
If you need change ip/netmask using ifconfig temporarily on solaris, use the following command:
ifconfig qfe1 101.139.1.151 netmask 255.255.254.0 broadcast + up

5)Now reboot your server and then use ifconfig -a and netstat -rnv to confirm everything is working as expected.

PS:
If you encounter errors below when booting solaris, then there may be some problem with network configuration on your host. Consider going to single user mode and change networking configuration detailed in this article.

Setting /dev/arp arp_cleanup_interval to 60000
Setting /dev/ip ip_forward_directed_broadcasts to 0
Setting /dev/ip ip_forward_src_routed to 0
Setting /dev/ip ip_ignore_redirect to 1
Setting /dev/ip ip_respond_to_address_mask_broadcast to 0
Setting /dev/ip ip_respond_to_echo_broadcast to 0
Setting /dev/ip ip_respond_to_timestamp to 0
Setting /dev/ip ip_respond_to_timestamp_broadcast to 0
Setting /dev/ip ip_send_redirects to 0
Setting /dev/ip ip_strict_dst_multihoming to 1
Setting /dev/ip ip_def_ttl to 255
Setting /dev/tcp tcp_conn_req_max_q0 to 4096
Setting /dev/tcp tcp_conn_req_max_q to 1024
Setting /dev/tcp tcp_smallest_anon_port to 32768
Setting /dev/tcp tcp_largest_anon_port to 65535
Setting /dev/udp udp_smallest_anon_port to 32768
Setting /dev/udp udp_largest_anon_port to 65535
Setting /dev/tcp tcp_smallest_nonpriv_port to 1024
Setting /dev/udp udp_smallest_nonpriv_port to 1024
Setting /dev/ip ip_ire_arp_interval to 60000
Setting /dev/tcp tcp_extra_priv_ports_add to 6112
Setting /dev/tcp tcp_rev_src_routes to 0

Categories: Networking Security, Unix Tags:

ilom or alom ip address reassignment howto

May 3rd, 2012 No comments

Here’s steps to reassign ip address for ilom or alom system console(out of band access):

  • log on destination host’s system console through the system’s console port address or jump from KVM which connects the host
  • after log on system console, run showsc to confirm before starting, for example on my host:

doxer_con> showsc
Advanced Lights Out Manager CMT v1.1.8

parameter value
——— —–
if_network true
if_modem false
if_emailalerts true
netsc_dhcp false
netsc_ipaddr 192,168.52.164
netsc_ipnetmask 255.255.255.0
netsc_ipgateway 192,168.52.254
mgt_mailhost 172.20.2.231
mgt_mailalert(1) [email protected] 2
sc_customerinfo doxer
sc_escapechars #.
sc_powerondelay true
sc_powerstatememory false
sc_clipasswdecho true
sc_cliprompt doxer_con
sc_clitimeout 0
sc_clieventlevel 3
sc_backupuserdata true
diag_trigger power-on-reset error-reset
diag_verbosity normal
diag_level min
diag_mode normal
sys_autorunonerror false
ser_baudrate 9600
ser_parity none
ser_stopbits 1
ser_data 8
netsc_enetaddr 00:14:4f:7e:24:59
sys_enetaddr 00:14:4f:7e:24:50
doxer_con>

  • Now do the actual setting according to your need:

setsc netsc_ipaddr
setsc netsc_ipnetmask
setsc netsc_ipgateway
setsc if_connection ssh

  • confirm everything is what you want with showsc
  • Now reset the system controller with resetsc -y to make it take effect
  • Once the ILO has rebooted check that you can ssh to it and login as usual

NB:

For more info about alom/ilom/openboot prom commands, please read here alom/ilom/openboot prom commands help

Categories: Hardware, Servers, Unix Tags: , ,

method to start stop SUNWwbsvr webservd Sun webserver

April 28th, 2012 No comments

Here’s steps to start Sun webserver:

cd /apps/SUNWwbsvr/<https-tag-of-your-hostname>

./start

Here’s steps to stop Sun webserver:

cd /apps/SUNWwbsvr/<https-tag-of-your-hostname>

./stop

To check whether start/stop/restart completes:

ps -ef | grep SUNWwbsvr

Categories: IT Architecture, Unix Tags: ,

resolved pca 403 forbidden server error on solaris

April 28th, 2012 No comments

Today when I was patching a solaris 5.9 host, error occurred with error message as follows after entering MOS(my oracle support) user/password:

122300 56 < 63 RS- 22 SunOS 5.9: Kernel Patch
Looking for 122300-63 (2/52)
Trying Oracle
Please enter My Oracle Support Account User: [email protected]
Please enter My Oracle Support Account Password:
Trying https://getupdates.oracle.com/ (zip) (1/1)
Failed (Error 403: Forbidden)
Failed (patch not found)

Then I went to http://support.oracle.com and searched patch 122300-63. The patching info page says I’ll need “Vintage Solaris download access/privilege” to download this patch, but obviously none of my CSI had this Vintage Solaris download access/privilege.

As this account issue may take some time to resolve, so I choose cluster patch or you may say patchset method to do the patching on solaris 9. Here’s the steps we need to do cluster patching on solaris 5.9:

  • 1.download latest cluster patching package that satisfies your host here http://wesunsolve.net/bundles
  • 2.unzip the package and have a read of Recommended.README file comes with the package
  • 3.ensure there’s enough free space on /, /var(better >4Gb)
  • 4. Now run ./install_patchset or ./install_cluster(you can add -nosave parameter if  you have limited free space on /, /var, but you will not be able to backout individual patches if the need arises)
  • 5.For more installation messages refer to the installation logfile:    /var/sadm/install_data/<patchset-name>_log
  • 6.reboot your machine to make all patches applied to your host.

NB:

If you have raid 1(mirror) on your solaris system, you can try first patch submirror and then apply to all system if server runs well after booting up. You can refer to the following for more infomation:

http://www.doxer.org/learn-linux/solaris-patching-trick-%E2%80%93-first-patch-submirror-then-sync-between-mirrors/

 

solaris svm breaking need boot from mirror

April 11th, 2012 No comments

If solaris’s svm has broken, and that broken one is for rootdisk, then the system will fail to boot up. We can now try boot from mirror disk rather than SVM. If the mirror is in good condition, then your system will boot up and after it’s up, we can do something to repair the broken solaris svm.

Here goes the steps to boot solaris from mirror disk without svm:

1.Prepare a cd/dvd with solaris of your host’s version.

2.goto ok mode

3.ok> boot cdrom -s ( Or boot net -s)

4.mount the root slice on /a

5.Take backup of /a/etc/vfstab and /a/etc/system files.

6.Modify the entries of the vfstab files and system files of /etc

7.Edit the /a/etc/system file, and remove the “rootdev” line shown below:

# vi /a/etc/system
*rootdev:/pseudo/md@0:0,0,blk #yours may be different
——> Do not comment the line. Remove it.

8.In the /etc/vfstab file, replace the lines for the system file system
metadevices with their underlying partitions.

For example, change lines from:

/dev/md/dsk/d0 /dev/md/rdsk/d0 / ufs 1 no -

to:

/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no -

ONLY change the lines for root (/) and the file systems which were affected. All other metadevices, may stay ‘as is’, in this file.

9.Unmount and check the root file system.

# cd /
# umount /a
# fsck /dev/rdsk/c0t0d0s0

10.#/usr/sbin/installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/cXtXdXs0

[

If you are using a cd/dvd or net which is having advance version of the Solaris OS to
the Solaris OS on the disk to boot to single user, then install the bootblk using the following command.

#/a/usr/sbin/installboot /a/usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/cXtXdXs0

]

11.init 0

12.Boot from the mirror disk.
Ok boot disk0

PS:

You can find more info if you search for “Unable to boot from a DiskSuite-controlled system disk” in google.

Categories: Storage, Unix Tags: ,

solaris kernel bug – ack replied before sync/ack valid outbound packets dropped

April 9th, 2012 No comments

If you intermittently getting the following error “ldapserver.test.com:389; socket closed.” , and after some tcpdumping you may find the following:

From the network traffic analysing you may find the following incorrect package exchange chain exists:

testhost1 — > testhost2 (SYN)
testhost1 < — testhost2 (ACK) — on this point should be sent SYN ACK package
testhost1 — > testhost2 (RST) – respectively in case when it didn’t receive SYN/ACK – client initiate reset TCP connection

Actually this is a solaris kernel bug, more info you can refer to
The workaround is running this:
ndd -set /dev/ip ip_ire_arp_interval 999999999
After this, the packet drop to 1 per week per host.

More info about this kernel bug can be found here http://wesunsolve.net/bugid/id/6942436

Categories: Kernel, Unix Tags:

hostname is different between linux and solaris

February 21st, 2012 No comments

1. For linux, -a is a option for the command hostname:
-a, –alias
Display the alias name of the host (if used).
For example:
[root@linux ~]# hostname -a
linux localhost.localdomain localhost
[root@linux ~]# grep linux /etc/hosts
127.0.0.1 linux.doxer.org linux localhost.localdomain localhost

2.For solaris:

But for solaris, there’s no -a option, which means, if you run hostname -a on a solaris box, you’re actually setting the hostname to “-a”, which in turn will cause many problem especially ldap.

Categories: Kernel, Linux, Unix Tags:

Too many cron jobs and crond processes running

February 17th, 2012 No comments

I faced a problem that a ton of crond processes(cronjobs, or crontab) were running on the OS:

root@localhost# ps auxww|grep cron
vare 543 0.0 0.0 141148 5904 ? S 01:43 0:00 crond
root 4085 0.0 0.0 72944 976 ? Ss 2010 1:13 crond
vare 4522 0.0 0.0 141148 5904 ? S Feb16 0:00 crond
vare 5446 0.0 0.0 141148 5904 ? S 02:43 0:00 crond
vare 9202 0.0 0.0 141148 5904 ? S Feb16 0:00 crond
vare 10245 0.0 0.0 141148 5908 ? S 03:43 0:00 crond
vare 13989 0.0 0.0 141148 5904 ? S Feb16 0:00 crond
vare 15487 0.0 0.0 141148 5908 ? S 04:43 0:00 crond
vare 18796 0.0 0.0 141148 5904 ? S Feb16 0:00 crond
vare 20448 0.0 0.0 141148 5908 ? S 05:43 0:00 crond
root 23168 0.0 0.0 6024 596 pts/0 S+ 06:15 0:00 grep cron
vare 23474 0.0 0.0 141148 5904 ? S Feb16 0:00 crond
vare 27183 0.0 0.0 141148 5904 ? S Feb16 0:00 crond
vare 28358 0.0 0.0 141148 5904 ? S 00:43 0:00 crond
vare 32032 0.0 0.0 141148 5904 ? S Feb16 0:00 crond

…..(and more)

Now let’s see what cronjobs are running by user vare:
root@localhost# crontab -u vare -l
# run the VERA Deploy routine
43 * * * * cd /share/scripts > /dev/null 2>&1 ; sleep 5 ; /share/scripts/Application/VARE/Deploy > /dev/null 2>&1

After check the script /share/bbscripts/Application/VERA/Deploy, I can see that the script is changing directory to a NFS mount point<i.e. cd /share/scripts> and then do some checks<i.e. /share/scripts/Application/VARE/Deploy>. But as there’s problem during the process it’s changing to NFS mount point, so the script hung there and didn’t quit normally. As such, the number of crond was increasing.

Method to solve this specific problem(specific means you’ve to check your own script) is to first kill the hung processes of crond, then bounce autofs and then restart crond.

 

Categories: Kernel, Linux, Unix Tags:

luxadm forcelip/display on solaris 10

February 3rd, 2012 No comments

Now let’s talk luxadm forcelip/display on solaris. Pay attention to bold ones. This article will be a little long and all about cXtXdXsX, be patient. :D
testhost:root root # vxprint -ht|grep dm #check for the disks on OS’s view:
dm emc333263A c1t5006048452A70F7Cd231s2 auto 65536 212055808 -
dm emc3330DA8 c1t5006048452A70F7Cd232s2 auto 65536 17609728 -
dm emc3332646 c1t5006048452A70F7Cd230s2 auto 65536 70640128 -

testhost:root root # luxadm probe #this will probe for SAN disks and it’s multipath
No Network Array enclosures found in /dev/es
Found Fibre Channel device(s):
Node WWN:5006048452a70f7c Device Type:Disk device
Logical Path:/dev/rdsk/c1t5006048452A70F7Cd230s2 #the OS disk’s wwn
Node WWN:5006048452a70f7c Device Type:Disk device
Logical Path:/dev/rdsk/c1t5006048452A70F7Cd231s2
Node WWN:5006048452a70f7c Device Type:Disk device
Logical Path:/dev/rdsk/c1t5006048452A70F7Cd232s2
Node WWN:5006048452a70f43 Device Type:Disk device
Logical Path:/dev/rdsk/c3t5006048452A70F43d230s2
Node WWN:5006048452a70f43 Device Type:Disk device
Logical Path:/dev/rdsk/c3t5006048452A70F43d231s2
Node WWN:5006048452a70f43 Device Type:Disk device
Logical Path:/dev/rdsk/c3t5006048452A70F43d232s2

From output of luxadm probe, we’ll know that there’re c1 and c3. We can prove this from
bash-3.00# /usr/sbin/cfgadm -la|grep fabric
c1 fc-fabric connected configured unknown
c3 fc-fabric connected configured unknown

testhost:root root # cfgadm -la|grep fabric
c1 fc-fabric connected configured unknown
c3 fc-fabric connected configured unknown
testhost:root root # fcinfo hba-port -l
HBA Port WWN: 210000e08b18da4f #this is the wwn for hba
OS Device Name: /dev/cfg/c1 #device name for the hba
Manufacturer: QLogic Corp.
Model: 375-3102-xx
Firmware Version: 03.03.28
FCode/BIOS Version: fcode: 1.13;
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 200000e08b18da4f
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
HBA Port WWN: 210000e08b18024f
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3102-xx
Firmware Version: 03.03.28
FCode/BIOS Version: fcode: 1.13;
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 200000e08b18024f
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 1
Loss of Signal Count: 1
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0

To display information on remote targets(includes the storage manufacturer, the storage product type, WWPNs, and all of the SCSI targets that have been presented to the host):
testhost:root root # fcinfo remote-port -slp 210000e08b18024f #which luns are seen by hba 210000e08b18024f?
Remote Port WWN: 5006048452a70f43
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 5006048452a70f43
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 1
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 255
Invalid CRC Count: 0
LUN: 230
Vendor: EMC
Product: SYMMETRIX
OS Device Name: /dev/rdsk/c3t5006048452A70F43d230s2
LUN: 231
Vendor: EMC
Product: SYMMETRIX
OS Device Name: /dev/rdsk/c3t5006048452A70F43d231s2
LUN: 232
Vendor: EMC
Product: SYMMETRIX
OS Device Name: /dev/rdsk/c3t5006048452A70F43d232s2

To Display WWN data for a target device or host bus adapter on the specified fibre channel port
testhost:root root # luxadm -e port
/devices/pci@1e,600000/SUNW,qlc@3/fp@0,0:devctl CONNECTED
/devices/pci@1d,700000/SUNW,qlc@1/fp@0,0:devctl CONNECTED
testhost:root root # luxadm -e dump_map /devices/pci@1e,600000/SUNW,qlc@3/fp@0,0:devctl
Pos Port_ID Hard_Addr Port WWN Node WWN Type
0 10300 0 5006048452a70f7c 5006048452a70f7c 0×0 (Disk device)
1 15500 0 210000e08b18da4f 200000e08b18da4f 0x1f (Unknown Type,Host Bus Adapter)
Here’s the multipath info:
testhost:root root # vxdmpadm getctlr all
LNAME PNAME VENDOR CTLR-ID
========================================================================================================
c1 /pci@1e,600000/SUNW,qlc@3/fp@0,0 QLogic Corp. 21:00:00:e0:8b:18:da:4f
c3 /pci@1d,700000/SUNW,qlc@1/fp@0,0 QLogic Corp. 21:00:00:e0:8b:18:02:4f
c0 /pci@1c,600000/scsi@2 – -
Here’s the multipath info for a specific disk(c1t5006048452A70F7Cd231s2):

testhost:root root # vxdisk list c1t5006048452A70F7Cd231s2

Device: c1t5006048452A70F7Cd231s2
devicetag: c1t5006048452A70F7Cd231
type: auto
hostid: testhost
disk: name=emc333263A id=1277720253.8.testhost
group: name=tpdbrdbd01root-dg id=1277720279.10.testhost
info: format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags: online ready private autoconfig autoimport imported
pubpaths: block=/dev/vx/dmp/c1t5006048452A70F7Cd231s2 char=/dev/vx/rdmp/c1t5006048452A70F7Cd231s2
guid: {5da11fa8-1dd2-11b2-ab51-0003ba89d76a}
udid: EMC%5FSYMMETRIX%5F000290102333%5F33!G+000
site: -
version: 3.1
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=2 offset=65792 len=212055808 disk_offset=0
private: slice=2 offset=256 len=65536 disk_offset=0
update: time=1277829173 seqno=0.11
ssb: actual_seqno=0.0
headers: 0 240
configs: count=1 len=48144
logs: count=1 len=7296
Defined regions:
config priv 000048-000239[000192]: copy=01 offset=000000 enabled
config priv 000256-048207[047952]: copy=01 offset=000192 enabled
log priv 048208-055503[007296]: copy=01 offset=000000 enabled
lockrgn priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths: 2
c1t5006048452A70F7Cd231s2 state=enabled
c3t5006048452A70F43d231s2 state=enabled

To read more info:
1.Add and Configure LUNs in Solaris 
2.man page for luxadm 
3.man page for fcinfo 
4./usr/sbin/cfgadm -la |grep fabric#solaris, check Fibre Channel controller status
fcinfo hba-port -l #check hba infomation, like Qlogic, Emulex
/usr/sbin/lpfc/lputil #Emulex HBAs are not seen in cfgadm -al output. Emulex uses “lpfc” driver. You can manipulate them via /usr/sbin/lpfc/lputil
luxadm -e port #check whether hba cards are connected, this will show physical path
luxadm –e forcelip c2 #forcelip of one entire controller
cfgadm –c configure c2::5006048452a72687 #configure lun
cfgadm –c configure c2 #configure the whole controller, it does not effect previously configured LUNs
devfsadm -c disk #scan disks in solaris
symcfg disco #update sym db on this host.
luxadm probe #check FC disks allocated to this host

using timex to check whether performance degradation caused by OS or VxVM

February 1st, 2012 No comments

To check for differences between operating system times to access disks and Volume Manager times to access disks, we can know whether to check for differences between operating system times to access disks and Volume Manager times to access disks. This is because they should both be about the same since both commands force a read of disk header information. If one of those is markedly greater then it indicates a problem in that area.

#echo | timex /usr/sbin/format #to avoid prompt for user input. Use time instead of timex for linux
real          13.03

user           0.10

sys            1.49
#timex vxdisk –o alldgs list
real           2.65

user           0.00

sys            0.00

Categories: Kernel, Linux, Unix Tags:

start/stop syslogd on solaris 10 or solaris 8/9

January 21st, 2012 No comments

Here’s the configuration:

On Solaris 5.8 and 5.9, at the command prompt, enter /etc/init.d/syslog stop, followed by /etc/init.d/syslog start.

On Solaris 5.10, at the command prompt, enter svcadm disable svc:/system/system-log && svcadm enable svc:/system/system-log.

Now you can check ps -ef|grep syslogd, and to configure syslog, go to /etc/syslogd.conf. Log files for syslogd is under /var/log/syslog.

Categories: Unix Tags:

replace a broken disk under solaris svm control

January 12th, 2012 No comments

Firstly, you need detach the submirror that need replaced(save a copy of metastat -i/-p, metadb -i, df -k before doing these steps):

metadetach d0 d10 #d10 is c0t0d0 in this context

If you met error like:

attempt an operation on a submirror that has erred components

Then you’ll need do a -f with metadetach:

metadetach -f d0 d10

Now do a check that all SVM objects have been removed from the failing disk:

metastat -p | grep c0t0d0
metadb | grep c0t0d0

Insert the new disk now.

Now configure the new disk:(this step may not needed if the disk is there in the output of metastat -i)

cfgadm -c configure c1::dsk/c0t0d0

Verify the disk has a “configured”.

copy disk head info from c1t0d0(the good one) to c1t1d0(the replaced one):(this step may not needed if you use format and can see the new disk has partitions expected already)

root on testserver:/var/tmp # prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s – /dev/rdsk/c1t1d0s2

And use format -> partition to check the partitioning.

You can check device alias through eeprom:
root on testserver:/var/tmp # eeprom | grep devalias
nvramrc=devalias rootdisk /ssm@0,0/pci@18,700000/scsi@2/disk@0,0
devalias rootmirror /ssm@0,0/pci@18,700000/scsi@2/disk@1,0

To see mapping between physical device path and device name, use command format:
root on testserver:/var/tmp # format
0. c1t0d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/ssm@0,0/pci@18,700000/scsi@2/sd@0,0
1. c1t1d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/ssm@0,0/pci@18,700000/scsi@2/sd@1,0

At last, to clear corrupted submirror->reinit submirror->attach submirror:
metaclear d10 #may not needed if d10 is there after running metastat -i
metainit -f d10 1 1 c0t0d0s0 #may not needed if d10 is there after running metastat -i
metattach d0 d10 #to see the resync progress, run metastat -i|grep progress
metastat d0
metastat -p
metastat -i
metadb -i #if metadb is not on at least two physical disks, you may need create metadb on the new disk using  metadb -a -c 3 c0t0d0s7

Categories: Hardware, Storage, Unix Tags:

Linux hostname domainname dnsdomainname nisdomainname ypdomainname

December 20th, 2011 No comments

Here’s just an excerpt from online man page of “domainname”:

NAME
hostname – show or set the system’s host name
domainname – show or set the system’s NIS/YP domain name
dnsdomainname – show the system’s DNS domain name
nisdomainname – show or set system’s NIS/YP domain name
ypdomainname – show or set the system’s NIS/YP domain name
hostname will print the name of the system as returned by the gethost-
name(2) function.

domainname, nisdomainname, ypdomainname will print the name of the sys-
tem as returned by the getdomainname(2) function. This is also known as
the YP/NIS domain name of the system.

dnsdomainname will print the domain part of the FQDN (Fully Qualified
Domain Name). The complete FQDN of the system is returned with hostname
–fqdn.

Sometime you may find a weird thing that you can use ldap verification to log on a client, but you can not sudo to root. Now you should consider run domainname to check whether it’s set to (none). If it does, you should consider set the domainname just using domainname command.

Categories: Kernel, Linux, Unix Tags:

Extending tmpfs’ed /tmp on Solaris 10(and linux) without reboot

November 3rd, 2011 No comments

Thanks to Eugene.

If you need to extend /tmp that is using tmpfs on Solaris 10 global zone (works with zones too but needs adjustments) and don’t want to undertake a reboot, here’s a tried working solution.

PLEASE BE CAREFUL, ONE ERROR HERE WILL KILL THE LIVE KERNEL!

echo “$(echo $(echo ::fsinfo | mdb -k | grep /tmp | head -1 | awk ‘{print $1}’)::print vfs_t vfs_data \| ::print -ta struct tmount tm_anonmax | mdb -k | awk ‘{print $1}’)/Z 0×20000″ | mdb -kw

Note the 0×20000. This number means new size will be 1GB. It is calculated like this: as an example, 0×10000 in hex is 65535, or 64k. The size is set in pages, each page is 8k, so resulting allocation size is 64k * 8k = 512m. 0×20000 is 1GB, 0×40000 is 2GB etc.

If the server has zones, you will see more then one entry in ::fsinfo, and you need to feed exact struct address to mdb. This way you can change /tmp size for individual zones, but this can only be done from global zone.

Same approach can probably be applied to older Solaris releases but will definitely need adjustments. Oh, and in case you care, on Linux it’s as simple as “mount -o remount,size=1G /tmp” :)

 

Categories: Kernel, Unix Tags:

Want your ldap password never expired? Here goes the howto

September 6th, 2011 2 comments

First, let’s check when your password will expire using ldapsearch:
root on testserver:/tmp # ldapsearch -D cn=”Directory Manager” -h ldap.testserver.com -b ou=people,dc=testserver,dc=com uid=liandy passwordexpirationtime
Enter bind password:
version: 1
dn: uid=liandy,ou=people,dc=testserver,dc=com
passwordexpirationtime: 20111005230540Z

ldapsearch -D “cn=’Directory Manager’” -h ldap.testserver.com -b ou=people,dc=testserver,dc=com uid=liandy passwordexpirationtime #this should work also
Now, let’s create a file named passwd.dn with content:
dn:uid=liandy,ou=people,dc=testserver,dc=com
changetype:modify
replace:passwordexpirationtime
passwordexpirationtime:20120612135450Z
And the last step is to change the expiration time using ldapmodify:
root on testserver:/tmp # ldapmodify -D cn=”Directory Manager” -h ldap.testserver.com -f passwd.dn
Enter bind password:
modifying entry uid=liandy,ou=people,dc=testserver,dc=com
That’s all the steps you need to change ldap password expiration time. To verify this has taken effect, fire ldapsearch to show expiration time just as the first step:
root on testserver:/tmp # ldapsearch -D cn=”Directory Manager” -h ldap.testserver.com -b ou=people,dc=testserver,dc=com uid=liandy passwordexpirationtime
Enter bind password:
version: 1
dn: uid=liandy,ou=people,dc=testserver,dc=com
passwordexpirationtime: 20120612135450Z
So you can see the ldap expiration time has been extended to 20120612.
NB:
If you want to change password for liandy on ldap.testserver.com, do the following:
Create a file named passwd2.dn with content:
In passwd2.dn:
dn:uid=liandy,ou=people,dc=testserver,dc=com
changetype:modify
replace:userPassword
userPassword:EnterYourPassword

Then run ldapmodify to modify the password:
ldapmodify -D cn=”Directory Manager” -h ldap.testserver.com -f passwd2.dn

If you want to modify directory manager’s password, here goes the step:
Create a file passwd3.dn with content:
dn: cn=config
changetype: modify
replace: nsslapd-rootpw
nsslapd-rootpw: EnterYourPassword

Then run ldapmodify to change the password:
ldapmodify -D “cn=directory manager” -h ldap.testserver.com -f passwd3.dn

If you forget the password for directory manager, you then need firstly find dse.ldap under ldap/slapd-Portal1/config, then encrypt your password, then modify nsslapd-rootpw.

Categories: Linux, Unix Tags:

check solaris 10/09 version info – update 6/7/8/9

August 4th, 2011 No comments

Using uname -a you can get the basic information currently available from the solaris system. For example, on my server:
root@beisoltest02 / # uname -a
SunOS beisoltest02 5.10 Generic_144489-12 i86pc i386 i86pc
However, sometimes you need check the “update version” of solaris. For example, in oracle documentation, your machine should has solaris 10 update 6 or higher if you want to install oracle 11 on your solaris host. So how can we check the “update version” of solaris?

Step 1:

#cat /etc/release
Solaris 10 10/08 s10x_u6wos_07b X86 #it’s update 6!
Copyright 2008 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 27 October 2008
Step 2:(more detailed info)

Compare solaris version history below:

Notable features of Solaris currently include DTraceDoorsService Management FacilitySolaris ContainersSolaris Multiplexed I/OSolaris Volume ManagerZFS, andSolaris Trusted Extensions.

Updates to Solaris versions are periodically released, such as Solaris 10 10/09.

In ascending order, the following versions of Solaris have been released:

Colour Meaning
Red Release no longer supported
Green Release still supported
Blue Future release
Solaris version SunOS version Release date End of support[39] Major new features
SPARC x86
1.x 4.1.x 1991–1994 - September 2003 SunOS 4 rebranded as Solaris 1 for marketing purposes. See SunOS article for more information.
2.0 5.0 June 1992 - January 1999 Preliminary release (primarily available to developers only), support for only the sun4c architecture. First appearance of NIS+.[40]
2.1 5.1 December 1992 May 1993 April 1999 Support for sun4 and sun4m architectures added; first Solaris x86 release. First Solaris 2 release to support SMP.
2.2 5.2 May 1993 - May 1999 SPARC-only release. First to support sun4d architecture. First to support multithreading libraries (UI threads API in libthread).[41]
2.3 5.3 November 1993 - June 2002 SPARC-only release. OpenWindows 3.3 switches from NeWS to Display PostScript and drops SunView support. Support added for autofs and CacheFSfilesystems.
2.4 5.4 November 1994 September 2003 First unified SPARC/x86 release. Includes OSF/Motif runtime support.
2.5 5.5 November 1995 December 2003 First to support UltraSPARC and include CDE, NFSv3 and NFS/TCP. Dropped sun4 (VMEbus) support. POSIX.1c-1995 pthreads added. Doors added but undocumented.[42]
2.5.1 5.5.1 May 1996 September 2005 Only release to support PowerPC platform; Ultra Enterprise support added; user and group IDs (uid_t, gid_t) expanded to 32 bits,[43] also included processor sets[44] and early resource management technologies.
2.6 5.6 July 1997 July 2006 Includes Kerberos 5, PAMTrueType fonts, WebNFS, large file support, enhanced procfs. SPARCserver 600MP series support dropped.[45]
7 5.7 November 1998 August 2008 The first 64-bit UltraSPARC release. Added native support for file system meta-data logging (UFS logging). Dropped MCA support on x86 platform. Last update was Solaris 7 11/99.[46]
8 5.8 February 2000 March 2012 Includes Multipath I/OSolaris Volume Manager,[47] IPMP, first support for IPv6 and IPsec (manual keying only), mdb modular debugger. Introduced Role-Based Access Control (RBAC); sun4c support removed. Last update is Solaris 8 2/04.[48]
9 5.9 May 28, 2002 January 10, 2003 October 2014 iPlanet Directory Server, Resource Manager, extended file attributesIKE IPsec keying, and Linux compatibility added; OpenWindows dropped, sun4d support removed. Most current update is Solaris 9 9/05.
10 5.10 January 31, 2005 - Includes x86-64 (AMD64/Intel 64) support, DTrace (Dynamic Tracing), Solaris ContainersService Management Facility (SMF) which replaces init.d scripts,NFSv4Least privilege security model. Support for sun4m and UltraSPARC I processors removed. Support for EISA-based PCs removed. Adds Java Desktop System (based on GNOME) as default desktop.[49]

  • Solaris 10 1/06 (known internally as “U1″) added the GRUB bootloader for x86 systems, iSCSI Initiator support and fcinfo command-line tool.
  • Solaris 10 6/06 (“U2″) added the ZFS filesystem.
  • Solaris 10 11/06 (“U3″) added Solaris Trusted Extensions and Logical Domains.
  • Solaris 10 8/07 (“U4″) added Samba Active Directory support,[50] IP Instances (part of the OpenSolaris Network Virtualization and Resource Control project),iSCSI Target support and Solaris Containers for Linux Applications (based on branded zones), enhanced version of the Resource Capping Daemon (rcapd).
  • Solaris 10 5/08 (“U5″) added CPU capping for Solaris Containers, performance improvements, SpeedStep support for Intel processors and PowerNow!support for AMD processors [51][52]
  • Solaris 10 10/08 (“U6″) added boot from ZFS and can use ZFS as its root file system. Solaris 10 10/08 also includes virtualization enhancements including the ability for a Solaris Container to automatically update its environment when moved from one system to another, Logical Domains support for dynamically reconfigurable disk and network I/O, and paravirtualization support when Solaris 10 is used as a guest OS in Xen-based environments such as Sun xVM Server.[53]
  • Solaris 10 5/09 (“U7″) added performance and power management support for Intel Nehalem processors, container cloning using ZFS cloned file systems, and performance enhancements for ZFS on solid-state drives.
  • Solaris 10 10/09 (“U8″) added user and group level ZFS quotas, ZFS cache devices and nss_ldap shadowAccount Support, improvements to patching performance.[54]
  • Solaris 10 9/10 (“U9″) added physical to zone migration, ZFS triple parity RAID-Z and Oracle Solaris Auto Registration.[55]
11 Express 2010.11 5.11 November 15, 2010 - Adds new packaging system (IPS=Image Packaging System) and associated tools, Solaris 10 Containers, network virtualization and QoS, virtual consoles, ZFS encryption and deduplication, updated GNOME. Removes Xsun, CDE.[56]

A more comprehensive summary of some Solaris versions is also available.[58] Solaris releases are also described in the Solaris 2 FAQ.[59]

correctable ecc event detected by cpu0/1/3/4

July 20th, 2011 No comments

If you receive these kinds of alerts in Solaris, it means your server has Memory Dimm issue. Please check with hrdconf/madmin(HP-UX) or prtconf(SUN solaris) to see the error message.

For more information about ECC memory, you can refer to the following article: http://en.wikipedia.org/wiki/ECC_memory

Categories: Kernel, Unix Tags:

Analysis of output by solaris format -> verify

May 21st, 2011 No comments

Here’s the output of format -> verify command in my solaris10:

format> verify

Primary label contents:

Volume name = < >
ascii name =
pcyl = 2609
ncyl = 2607
acyl = 2
bcyl = 0
nhead = 255
nsect = 63
Part Tag Flag Cylinders Size Blocks
0 root wm 1 – 1306 10.00GB (1306/0/0) 20980890
1 var wm 1307 – 2351 8.01GB (1045/0/0) 16787925
2 backup wu 0 – 2606 19.97GB (2607/0/0) 41881455
3 stand wm 2352 – 2606 1.95GB (255/0/0) 4096575
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
8 boot wu 0 – 0 7.84MB (1/0/0) 16065
9 unassigned wm 0 0 (0/0/0) 0

Now, let’s give it an analysis:

  • Part

Solaris x86 has 9 slices for a disk, and for 8th and 9th, they’re preserved by solaris.

  • Tag

This is used to indicate the purpose of the slice. Possible values are:

unassigned, boot, root, swap, usr, backup, stand, home, and public, private(The latter two are used by Sun StorEdge).

  • Flag

wm – this slice is writable and mountable.

wu - this slice is writable and unmountable.

rm - this slice is readable and mountable.

ru – this slice is readable and unmountable.

  • Cylinders

This part shows the start and end cylinder number of the slice.

  • Size

The size of the slice.

  • Blocks

This shows the number of cylinders and sectors of the slice.

Now, let’s create a slice and mount the filesystem:

root@test / # format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@0,0/pci15ad,1976@10/sd@0,0
1. c1t1d0
/pci@0,0/pci15ad,1976@10/sd@1,0
Specify disk (enter its number): 1 #select this disk
selecting c1t1d0
[disk formatted]

FORMAT MENU:
disk – select a disk
type – select (define) a disk type
partition – select (define) a partition table
current – describe the current disk
format – format and analyze the disk
fdisk – run the fdisk program
repair – repair a defective sector
label – write label to the disk
analyze – surface analysis
defect – defect list management
backup – search for backup labels
verify – read and display labels
save – save new disk/partition definitions
inquiry – show vendor, product and revision
volname – set 8-character volume name
! – execute , then return
quit
format> partition #select partition to check and create new slice

PARTITION MENU:
0 – change `0′ partition
1 – change `1′ partition
2 – change `2′ partition
3 – change `3′ partition
4 – change `4′ partition
5 – change `5′ partition
6 – change `6′ partition
7 – change `7′ partition
select – select a predefined table
modify – modify a predefined partition table
name – name the current table
print – display the current table
label – write partition map and label to the disk
! – execute , then return
quit
partition> print #check slice topology
Current partition table (original):
Total disk cylinders available: 2607 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 1 – 1306 10.00GB (1306/0/0) 20980890
1 var wm 1307 – 2351 8.01GB (1045/0/0) 16787925
2 backup wu 0 – 2606 19.97GB (2607/0/0) 41881455
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
8 boot wu 0 – 0 7.84MB (1/0/0) 16065
9 unassigned wm 0 0 (0/0/0) 0

partition> 3 #select an unassigned slice. It will be /dev/rdsk/c1t1d0s3 after saving to format.dat
Part Tag Flag Cylinders Size Blocks
3 unassigned wm 0 0 (0/0/0) 0

Enter partition id tag[unassigned]: stand
Enter partition permission flags[wm]:
Enter new starting cyl[1]: 2352
Enter partition size[0b, 0c, 2352e, 0.00mb, 0.00gb]: $
partition> label #write label to disk
Ready to label disk, continue? y

partition> name #name the current table
Enter table name (remember quotes): hah

partition> quit

FORMAT MENU:
disk – select a disk
type – select (define) a disk type
partition – select (define) a partition table
current – describe the current disk
format – format and analyze the disk
fdisk – run the fdisk program
repair – repair a defective sector
label – write label to the disk
analyze – surface analysis
defect – defect list management
backup – search for backup labels
verify – read and display labels
save – save new disk/partition definitions
inquiry – show vendor, product and revision
volname – set 8-character volume name
! – execute , then return
quit
format> save #save new disk/partition definitions
Saving new disk and partition definitions
Enter file name["./format.dat"]:
format> quit

root@test / # newfs /dev/rdsk/c1t1d0s3 #create filesystem on newly created slice
newfs: construct a new file system /dev/rdsk/c1t1d0s3: (y/n)? y
Warning: 1474 sector(s) in last cylinder unallocated
/dev/rdsk/c1t1d0s3: 4096574 sectors in 667 cylinders of 48 tracks, 128 sectors
2000.3MB in 42 cyl groups (16 c/g, 48.00MB/g, 11648 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920,
3149856, 3248288, 3346720, 3445152, 3543584, 3642016, 3740448, 3838880,
3937312, 4035744
root@test / # fstyp /dev/dsk/c1t1d0s3 #check the filesystem type
ufs
root@test / # mkdir /hah
root@test / # mount /dev/dsk/c1t1d0s3 /hah #mount filesystem
root@test / # cd /hah/
root@test hah # touch aa #create a file to have a test
root@test hah # ls #finished, congratulations!
aa lost+found

Categories: Unix Tags:

resolved: zoneadm list segmentation fault core dumped

May 12th, 2011 1 comment

You are stuck in “zoneadm list segmentation fault core dumped” problem right? Now let me help you. :D
Here I can reproduce the scene:
root@test01~# uname -a
SunOS test01 5.10 Generic_142901-13 i86pc i386 i86pc

root@test01~# zoneadm list
Segmentation Fault (core dumped)

Then, we can see a coredump file under /var/core:
root@test01~# ls -l /var/core/core_test01*|tail
-rw——- 1 root root 5741655 May 11 02:22 /var/core/core_test01_zoneadm_0_0_1305076970_25407
-rw——- 1 root root 5741655 May 11 02:26 /var/core/core_test01_zoneadm_0_0_1305077182_25546
-rw——- 1 root root 5741655 May 12 02:30 /var/core/core_test01_zoneadm_0_0_1305163819_199

Now, truss it:
root@test01/var/core# truss zoneadm list
execve(“/usr/sbin/zoneadm”, 0x0804782C, 0×08047838) argc = 2
mmap(0×00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFF0000
resolvepath(“/usr/lib/ld.so.1″, “/lib/ld.so.1″, 1023) = 12
resolvepath(“/usr/sbin/zoneadm”, “/usr/sbin/zoneadm”, 1023) = 17
stat64(“/usr/sbin/zoneadm”, 0x080475C0) = 0
open(“/var/ld/ld.config”, O_RDONLY) Err#2 ENOENT
stat64(“./libzonecfg.so.1″, 0x08046E60) Err#2 ENOENT
stat64(“/usr/local/lib/libzonecfg.so.1″, 0x08046E60) Err#2 ENOENT
stat64(“/usr/openwin/lib/libzonecfg.so.1″, 0x08046E60) Err#2 ENOENT
stat64(“/lib/libzonecfg.so.1″, 0x08046E60) Err#2 ENOENT
stat64(“/usr/lib/libzonecfg.so.1″, 0x08046E60) = 0
resolvepath(“/usr/lib/libzonecfg.so.1″, “/usr/lib/libzonecfg.so.1″, 1023) = 24
open(“/usr/lib/libzonecfg.so.1″, O_RDONLY) = 3
mmap(0×00010000, 32768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_ALIGN, 3, 0) = 0xFEFB0000

Then, let’s try ldd:
# ldd /usr/sbin/zoneadm

libz.so.1 => /usr/local/lib/libz.so.1
libz.so.1 (SUNW_1.1) => (version not found)

That`s the lib problem with the SUNWzlib package, that is used for the gzip. So, googling around and praising unix.com forum, I was able to find another instance of the library that was expected, but somehow was mis-placed by another version:

# grep libz.so.1 /var/sadm/install/contents | grep " f "
/usr/lib/libz.so.1 f none 0755 root bin 80200 28295 1170178503 SUNWzlib
/usr/lib/sparcv9/libz.so.1 f none 0755 root bin 98656 6323 1170178503 SUNWzlib
/usr/local/lib/libz.so.1.2.5 f none 0755 bin bin 87892 20672 1273268080 SMCzlib

Now, I`m preceding the LD_LIBRARY_PATH with the path to the first one lib:

export LD_LIBRARY_PATH=/usr/lib/:/usr/local/lib:/usr/openwin/lib:/usr/local/lib:/usr/openwin/lib

Ok, then you can try zoneadm list -civ again. Good luck!

Thanks for Nik.

http://nikmaslov.com/blog/blog1.php/2010/08/03/segfault-due-to-the-libs-missing#c4165

 

Categories: Unix Tags:

Clear error logs from fma before obu firmware patching

May 12th, 2011 No comments

1. Clear logs from fma:
To clear the FMA faults and error logs from Solaris:
a) Show faults in FMA
# fmadm faulty
NOTE: Do not use ‘fmadm faulty -a’ in this step. When you specify the -a optionall resource information cached by the Fault Manager is listed, includingfaults, which have already been corrected or where no recovery action is needed(see ‘fmadm’ man page). The listing also include information for resources,that may no longer be present in the system
b) For each fault UUID listed in the ‘fmadm faulty’ run
# fmadm repair <uuid># fmadm faulty (to make sure the output is clean after repair)
c) Clear ereports and resource cache
# cd /var/fm/fmd# rm e* f* c*/eft/* r*/*
d) Reset the fmd serd modules
# fmadm reset cpumem-diagnosis# fmadm reset cpumem-retire# fmadm reset eft# fmadm reset io-retire
e) Reboot the system
To clear the FMA faults and error logs from Solaris without rebooting the system:
a) Stop the fmd:
# svcadm disable -s svc:/system/fmd:default
b) Remove all files from the FMA log directories. This is very specific to the files found in the FMS directories, all directories must be left intact.
# cd /var/fm/fmd
# find /var/fm/fmd -type f -exec ls {} \;
c) Check that only files within the /var/fm/fmd directory are identified then replace the ls with rm to remove them.
# find /var/fm/fmd -type f -exec rm {} \;
d) Restart fmd after the files are removed
# svcadm enable svc:/system/fmd:default
For more information about clearing logs please see Doc ID 1004229.1
2. Upgrade the SP and OBP firmware by installing patch 145673-02.

About how to do firmware patch, please refer to http://www.doxer.org/learn-linux/obu-firmware-patching/

Categories: Servers, Unix Tags: