Here's the outline for resolution:
1.check whether ntpd is running on the problematic host;
2.stop ntpd;
3.use ntpdate to manually sync with ntp server
4.start up ntpd.
Here's the detailed commands:
root@testhost# service ntpd stop
Shutting down ntpd: [ OK ]
root@testhost# ps -ef|grep ntp
root 9805 9542 0 01:53 pts/0 00:00:00 grep ntp
root@testhost# cat /etc/ntp.conf
tinker panic 0
server timehost1 prefer
server timehost2
server timehost3
driftfile /var/lib/ntp/drift
# Prohibit general access to this service.
#restrict default ignore #this is to allow ntpd to use the new ntpd server
# Permit the cluster node listed to synchronise with this time service.
# Do not permit those systems to modify the configuration of this service.
# Allow this host to be used as a timesource
# Permit all loopback interface access.
restrict 127.0.0.1
root@testhost# ntpdate -u timehost1
2 Feb 01:55:20 ntpdate[9824]: step time server 172.20.220.27 offset 59.998407 sec
root@testhost# service ntpd start
Starting ntpd: [ OK ]
root@testhost# ps -ef|grep ntp
ntp 9999 1 0 01:55 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root 10017 9542 0 01:55 pts/0 00:00:00 grep ntp
PS:
- If your host is a Virtual Machine running on Xen hypervisor, then you may find that ntpdate or ntpd will fail to synchronize time with the time server. That's because the VM will sync with hypervisor by default. So if you want to sync your VM's time with the time server, there're two methonds:
1. Synchronize time on XEN hypervisor and the VM will then sync with it automatically;
2. If you just want to sync VM's time without change XEN hypervisor's time setting, then on the VM, do the following (This setting does not apply to hardware virtualized guests):
echo 1 > /proc/sys/xen/independent_wallclock #or in /etc/sysctl.conf, xen.independent_wallclock=1
For VM, you should also note below(Wallclock Time Skew Problems):
Additional parameters may be needed in the boot loader (grub.conf) configuration file for certain operating system variants after the guest is installed. Specifically, for optimal clock accuracy, Linux guest boot parameters should be specified to ensure that the pit clock source is utilized. Adding clock=pit nohpet nopmtimer for most guests will result in the selection of pit as the clock source for the guest. Published templates for Oracle VM include these additional parameters.
Proper maintenance of virtual time can be tricky. The various parameters provide tuning for virtual time management and supplement, but do not replace, the need for an ntp time service running within guest. Ensure that the ntpd service is running and that the /etc/ntpd.conf configuration file is pointing to valid time servers.
After this step, you can now sync time for the VM without impacting others.
- You can use ntpq -p to check the status of remote time servers.
[root@test-host ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*LOCAL(0) .LOCL. 5 l 66 64 377 0.000 0.000 0.001
adcq7-rtr-9.us. .INIT. 16 u - 512 0 0.000 0.000 0.000
adcq7-rtr-10.us .INIT. 16 u - 512 0 0.000 0.000 0.000
On column "remote", we can see that it's blank space before adcq7-rtr-9.us and adcq7-rtr-10.us, this means that the two sources are discarded, failed sanity check and has never been synced to.
For more details about output of ntpq -p, you can read the explanation here.
- For ntpd firewall issue, if you want to use ntpd, then you need to fix your network/firewall/NAT so that ntpd can have full unrestricted access to UDP port 123 in both directions. Also, you may setup one cronjob to bounce ntpd every hour so that ntp will be forced to sync with time server. echo "`tr -cd 1-5 </dev/urandom | head -c 2` */1 * * * /sbin/service ntpd restart" >> /var/spool/cron/root #you can of course manully add to cronjob via crontab -e -u root. or echo "`tr -cd 1-5 </dev/urandom | head -c 2` */1 * * * root /sbin/service ntpd restart" >> /etc/crontab
- Many people have difficulties with using RESTRICT. They want to set themselves up to be as secure as possible, so they create an extremely limited default RESTRICT line in their /etc/ntp.conf file, and then they find that they can't talk to anyone. If you're having problems with your server, in order to do proper debugging, you should turn off all RESTRICT lines in your /etc/ntp.conf file, and otherwise simplify the configuration as much as possible, so that you can make sure that the basic functions are working correctly. Once you get the basics working, try turning back on various features, one-by-one. Here some tips for ntp restrict keyword controlling ntpd access.
- To get start with ntp, read this guide(explained server & peer & stratum. Pool is a list of servers). And here is the full document about ntp.
- You can run ntpstat to check ntp status too:
# ntpstat
synchronised to NTP server (10.240.152.1) at stratum 3
time correct to within 12 ms
polling server every 128 s
- linux 'leap-second' bug (link here) - if the system has been up since June 30th and is running ntp. One way to fix it is (as root from the command line):
date -s "`date`"
1. Observe usage through "top"
2. Cleanup orphan java/FF processes.
3. run this command as root (sudo su -) : date -s "`date`"
4. Observe usage through "top" again. It should fall drastically low (near zero)