Archive

Archive for July, 2011

Emcgrab download url

July 27th, 2011 1 comment

Just go to ftp://ftp.emc.com/pub/emcgrab/Unix/ to select yours and download(solaris, linux, aix etc.). The latest version is emcgrab-4.4.4.

PS:

emcgrab is also known as Remediation HEAT report. EMC grab report generates file that EMC or storage team might use for diagnostics and monitoring of storage usage by the client and possible issues; it also dumps logs/config into this file. This procedure proved to be safe, but need to be done out of hours if that`s possible.

Categories: Servers Tags:

correctable ecc event detected by cpu0/1/3/4

July 20th, 2011 No comments

If you receive these kinds of alerts in Solaris, it means your server has Memory Dimm issue. Please check with hrdconf/madmin(HP-UX) or prtconf(SUN solaris) to see the error message.

For more information about ECC memory, you can refer to the following article: http://en.wikipedia.org/wiki/ECC_memory

Categories: Kernel, Unix Tags:

convert squid access.log time to human-readable format

July 13th, 2011 No comments

If you’re looking at squid access.log, you’ll find the first column of it is the time, for example:
1310313302.640 829 58.215.75.51 TCP_MISS/200 104 CONNECT smtp.126.com:25 – DIRECT/123.125.50.112 -
1310313303.484 1845 58.215.75.51 TCP_MISS/200 104 CONNECT smtp.126.com:25 – DIRECT/123.125.50.111 -
This is not very human readable. Now we can use perl to convert squid access.log time to human-readable format, just one line command:

cat /var/log/squid/access.log |perl -nwe’s/^(\d+)/localtime($1)/e; print’

After this, it’ll look like this:
Sun Jul 10 09:55:03 2011.484 1845 58.215.75.51 TCP_MISS/200 104 CONNECT smtp.126.com:25 – DIRECT/123.125.50.111 -
Sun Jul 10 09:55:07 2011.146 355 88.80.10.1 TCP_MISS/200 1176 GET http://spam-chaos.com/pp/set-cookie.php – DIRECT/88.80.10.1 text/html
Hey, more human-readable, right?

Categories: Linux Tags:

Resolved – ld.so.1: httpd: fatal: libaprutil-0.so.0: open failed: No such file or directory

July 9th, 2011 No comments

When I tried to start up IBM http server on my Solaris, I met this error message:
ld.so.1: httpd: fatal: libaprutil-0.so.0: open failed: No such file or directory
Killed
This error was caused by library libaprutil-0.so.0 can not be found in the current library path environment variable. We should add /apps/IBMHTTPD/Installs/IHS61-01/lib to LD_LIBRARY_PATH to make libaprutil-0.so.0 be seen by ld.
Here goes the resolution:
#export LD_LIBRARY_PATH=/apps/IBMHTTPD/Installs/IHS61-01/lib:$LD_LIBRARY_PATH #set environment variable
#/apps/IBMHTTPD/Installs/IHS61-01/bin/httpd -d /apps/IBMHTTPD/Installs/IHS61-01 -d /apps/IBMHTTPD/Installs/IHS61-01 -f /apps/IBMHTTPD/Installs/IHS61-01/conf/ihs.conf -k restart #restart IHS
#/usr/ucb/ps auxww|grep -i ibmhttpd #check the result

PS:

Here’s more about ld.so: (from book <Optimizing Linux® Performance: A Hands-On Guide to Linux® Performance Tools>)

When a dynamically linked application is executed, the Linux loader, ld.so, runs first.ld.so loads all the application’s libraries and connects symbols that the application uses with the functions the libraries provide. Because different libraries were originally linked at different and possibly overlapping places in memory, the linker needs to sort through all the symbols and make sure that each lives at a different place in memory. When a symbol is moved from one virtual address to another, this is called a relocation. It takes time for the loader to do this, and it is much better if it does not need to be done at all. The prelink application aims to do that by rearranging the system libraries of the entire systems so that they do not overlap. An application with a high number of relocations may not have been prelinked.

The Linux loader usually runs without any intervention from the user, and by just executing a dynamic program, it is run automatically. Although the execution of the loader is hidden from the user, it still takes time to run and can potentially slow down an application’s startup time. When you ask for loader statistics, the loader shows the amount of work it is doing and enables you to figure out whether it is a bottleneck.

The ld command is invisibly run for every Linux application that uses shared libraries. By setting the appropriate environment variables, we can ask it to dump information about its execution. The following invocation influences ld execution:
env LD_DEBUG=statistics,help LD_DEBUG_OUTPUT=filename <command>

Backup by oracle on client xxx using policy ORACLE_PROD_DAILY_DB

July 6th, 2011 No comments

Sometimes you got error about netbackup hasn’t been done successfully and what’s weird was that no daily backup logs were created then. Actually, at that same time, the system log will show that xinetd is rejecting the connection:

xinetd[2926]: FAIL: vnetd per_source_limit from=1.2.3.4

Problem

GENERAL ERROR: Network errors during Oracle RMAN backups when using xinetd resulting in status 6, status 23, status 25, status 41, or status 58

Error

ORA-27028: skgfqcre: sbtbackup returned error

Solution

Overview:The xinetd process receives inbound socket connections from remote hosts and may reject the connections without starting the expected NetBackup process if there have recently been too many connections from the remote host.

Troubleshooting:

The problem may have many different NetBackup symptoms depending on the connecting process, target process, and connection method.  But in all cases, the destination process will not be started and will not create any debug log entries at the time of the expected connection.  At that same time, the system log will show that xinetd is rejecting the connection, e.g.

xinetd[2926]: FAIL: vnetd per_source_limit from=1.2.3.4

Log Files:

This bpbrm log shows a successful connection to bpcd followed by a forwarding socket request, both reach the client host and receive a file descriptor (fd).  However, xinetd closed the forwarding socket immediately as indicated by the errno 232.

08:20:38.890 [5871] <2> logconnections: BPCD CONNECT FROM 1.2.3.4.54105 TO 2.3.4.5.13724
...snip...
08:20:38.891 [5871] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.179: msg: VNETD CONNECT FROM 1.2.3.4.54106 TO 2.3.4.5.13724 fd = 10
08:20:38.892 [5871] <2> vnet_pop_byte: vnet.c.184: errno: 232 0x000000e8
08:20:38.892 [5871] <2> vnet_pop_byte: vnet.c.185: Function failed: 43 0x0000002b
08:20:38.893 [5871] <2> vnet_pop_string: vnet.c.266: Function failed: 43 0x0000002b
08:20:38.893 [5871] <2> vnet_pop_signed: vnet.c.310: Function failed: 43 0x0000002b
08:20:38.893 [5871] <2> version_connect: vnet_vnetd.c.1812: Function failed: 43 0x0000002b
...snipped three retries from bpbrm to vnetd, all of them closed by xinetd...
08:20:41.943 [5871] <2> bpcr_vnetd_connect_forward_socket_begin: nb_vnetd_connect(2.3.4.5) failed: 25
08:20:41.943 [5871] <2> local_bpcr_connect: bpcr_vnetd_connect_forward_socket_begin failed: 25
08:20:41.943 [5871] <2> ConnectToBPCD: bpcd_connect_and_verify(myhost, myhost) failed: 25
08:20:41.944 [5871] <16> bpbrm start_bpcd_stat: cannot connect to myhost, Connection reset by peer (232)

The bpcd log shows a failure waiting for bpbrm to send the forwarding socket information.

08:20:38.912 [24709] <2> logconnections: BPCD ACCEPT FROM 1.2.3.4.54105 TO 2.3.4.5.13724
...snip...
08:20:39.002 [24709] <2> bpcd main: output socket port number = 1
08:20:41.942 [24709] <2> get_long: (2) premature end of file (byte 1)
08:20:41.942 [24709] <2> get_vnetd_forward_socket: get_string ipc_string failed: 5
08:20:41.942 [24709] <16> bpcd main: get_vnetd_forward_socket failed: 23

The client vnetd log does not show any activity at the time of the forwarding socket requests.

08:20:38.889 [24709] <2> launch_command: vnetd.c.2125: path: /usr/openv/netbackup/bin/bpcd
...no entries during this time span...
08:20:42.227 [24672] <2> vnet_send_network_socket: vnet_vnetd.c.1535: hash_str2: 7f9786a02bd04ad3f585c06892bd74c1

The system messages log shows that xinetd rejected the 4 vnetd connections.

Jul 16 08:20:38 myhost xinetd[2926]: FAIL: vnetd per_source_limit from=1.2.3.4
Jul 16 08:20:39 myhost xinetd[2926]: FAIL: vnetd per_source_limit from=1.2.3.4
Jul 16 08:20:40 myhost xinetd[2926]: FAIL: vnetd per_source_limit from=1.2.3.4
Jul 16 08:20:41 myhost xinetd[2926]: FAIL: vnetd per_source_limit from=1.2.3.4

Another example bpbrm connection failure attempt to bpcd to perform a comm file update.

<16> bpbrm send_info_via_progress_file: cannot connect to <clientname>, Operation now in progress (150)

Resolution:

This situation can be resolved by adjusting the xinetd configuration to permit sufficient per_source connections for NetBackup operations.  On some operating system platforms, the default per_source value can be as low as 1-10.  A NetBackup master/media server may rapidly make more than this many connections to the client host, depending on the type of backup and connection method; up to 4 sockets per concurrent Automatic backup job and 9 sockets per concurrent Application backup job.

Review the xinetd configuration documentation and make appropriate changes for the platform and expected number of simultaneous NetBackup operations.  After making the changes, send a SIGHUP to the xinetd process to force a read of the updated configuration.

If appropriate, the default xinetd settings for all services can be changed in the /etc/xinitid.conf file, e.g.

defaults
{
per_source = <new connection limit or UNLIMITED>
}

Alternatively, the xinetd configuration settings for specific services can be changed in service specific files, e.g. /etc/xinetd.d/vnetdcould contain the following setting.

service vnetd
{
per_source = <new connection limit or UNLIMITED>
}

The NetBackup services that are most likely to be affected are vnetd and bpcd.

Plan to minimally set the number of per_source connections to:
The number of channels or concurrent backup streams * 9 (using the higher level required for application backups) = the per_source minimum setting.

Example: Given a client running Oracle RMAN Application type backups (requiring up to 9 concurrent connections), with 8 channels, the minimum setting would be 72 for per_source.

Legacy ID

300810

From:http://www.symantec.com/business/support/index?page=content&id=TECH58196

Categories: Databases Tags: