Sometimes you got error about netbackup hasn’t been done successfully and what’s weird was that no daily backup logs were created then. Actually, at that same time, the system log will show that xinetd is rejecting the connection:
Overview:The xinetd process receives inbound socket connections from remote hosts and may reject the connections without starting the expected NetBackup process if there have recently been too many connections from the remote host.
Troubleshooting:
The problem may have many different NetBackup symptoms depending on the connecting process, target process, and connection method. But in all cases, the destination process will not be started and will not create any debug log entries at the time of the expected connection. At that same time, the system log will show that xinetd is rejecting the connection, e.g.
xinetd[2926]: FAIL: vnetd per_source_limit from=1.2.3.4
Log Files:
This bpbrm log shows a successful connection to bpcd followed by a forwarding socket request, both reach the client host and receive a file descriptor (fd). However, xinetd closed the forwarding socket immediately as indicated by the errno 232.
08:20:38.890 [5871] <2> logconnections: BPCD CONNECT FROM 1.2.3.4.54105 TO 2.3.4.5.13724
...snip...
08:20:38.891 [5871] <2> vnet_connect_to_vnetd_extra: vnet_vnetd.c.179: msg: VNETD CONNECT FROM 1.2.3.4.54106 TO 2.3.4.5.13724 fd = 10
08:20:38.892 [5871] <2> vnet_pop_byte: vnet.c.184: errno: 232 0x000000e8
08:20:38.892 [5871] <2> vnet_pop_byte: vnet.c.185: Function failed: 43 0x0000002b
08:20:38.893 [5871] <2> vnet_pop_string: vnet.c.266: Function failed: 43 0x0000002b
08:20:38.893 [5871] <2> vnet_pop_signed: vnet.c.310: Function failed: 43 0x0000002b
08:20:38.893 [5871] <2> version_connect: vnet_vnetd.c.1812: Function failed: 43 0x0000002b
...snipped three retries from bpbrm to vnetd, all of them closed by xinetd...
08:20:41.943 [5871] <2> bpcr_vnetd_connect_forward_socket_begin: nb_vnetd_connect(2.3.4.5) failed: 25
08:20:41.943 [5871] <2> local_bpcr_connect: bpcr_vnetd_connect_forward_socket_begin failed: 25
08:20:41.943 [5871] <2> ConnectToBPCD: bpcd_connect_and_verify(myhost, myhost) failed: 25
08:20:41.944 [5871] <16> bpbrm start_bpcd_stat: cannot connect to myhost, Connection reset by peer (232)
The bpcd log shows a failure waiting for bpbrm to send the forwarding socket information.
08:20:38.912 [24709] <2> logconnections: BPCD ACCEPT FROM 1.2.3.4.54105 TO 2.3.4.5.13724
...snip...
08:20:39.002 [24709] <2> bpcd main: output socket port number = 1
08:20:41.942 [24709] <2> get_long: (2) premature end of file (byte 1)
08:20:41.942 [24709] <2> get_vnetd_forward_socket: get_string ipc_string failed: 5
08:20:41.942 [24709] <16> bpcd main: get_vnetd_forward_socket failed: 23
The client vnetd log does not show any activity at the time of the forwarding socket requests.
08:20:38.889 [24709] <2> launch_command: vnetd.c.2125: path: /usr/openv/netbackup/bin/bpcd
...no entries during this time span...
08:20:42.227 [24672] <2> vnet_send_network_socket: vnet_vnetd.c.1535: hash_str2: 7f9786a02bd04ad3f585c06892bd74c1
The system messages log shows that xinetd rejected the 4 vnetd connections.
Jul 16 08:20:38 myhost xinetd[2926]: FAIL: vnetd per_source_limit from=1.2.3.4
Jul 16 08:20:39 myhost xinetd[2926]: FAIL: vnetd per_source_limit from=1.2.3.4
Jul 16 08:20:40 myhost xinetd[2926]: FAIL: vnetd per_source_limit from=1.2.3.4
Jul 16 08:20:41 myhost xinetd[2926]: FAIL: vnetd per_source_limit from=1.2.3.4
Another example bpbrm connection failure attempt to bpcd to perform a comm file update.
<16> bpbrm send_info_via_progress_file: cannot connect to <clientname>, Operation now in progress (150)
Resolution:
This situation can be resolved by adjusting the xinetd configuration to permit sufficient per_source connections for NetBackup operations. On some operating system platforms, the default per_source value can be as low as 1-10. A NetBackup master/media server may rapidly make more than this many connections to the client host, depending on the type of backup and connection method; up to 4 sockets per concurrent Automatic backup job and 9 sockets per concurrent Application backup job.
Review the xinetd configuration documentation and make appropriate changes for the platform and expected number of simultaneous NetBackup operations. After making the changes, send a SIGHUP to the xinetd process to force a read of the updated configuration.
If appropriate, the default xinetd settings for all services can be changed in the /etc/xinitid.conf file, e.g.
defaults
{
per_source = <new connection limit or UNLIMITED>
}
Alternatively, the xinetd configuration settings for specific services can be changed in service specific files, e.g. /etc/xinetd.d/vnetdcould contain the following setting.
service vnetd
{
per_source = <new connection limit or UNLIMITED>
}
The NetBackup services that are most likely to be affected are vnetd and bpcd.
Plan to minimally set the number of per_source connections to:
The number of channels or concurrent backup streams * 9 (using the higher level required for application backups) = the per_source minimum setting.
Example: Given a client running Oracle RMAN Application type backups (requiring up to 9 concurrent connections), with 8 channels, the minimum setting would be 72 for per_source.
