I’ve an IOT application distributed over multiple systems all on a local network and all synchronized with some flavor of ntp running.
These all stay syncronized to within a second over long periods:
Raspberry Pi2
Raspberry Pi3
Raspberry PiZero-W
Ubuntu 16.04 desktop
Commercial Lorex security DVR
These two have drifted about three seconds slow already, although they seem well synced with each other:
Beaglebone Green running Debian 8.9 (rebooted this morning)
Beaglebone White running Debian 9.2 (booted yesterday morning)
All the Raspberries are running Raspbian, verson 8 for the Pi2 and version 9 for the rest
My initial assumption is somehow they are using different time servers that have a systematic error. Problem is I can’t find where the systemd-timesync stores its servers to see if Beaglebones ntp server settings matches the others.
The Lorex box would drift until I changed it sync interval from 6 hours to 1 hour. All the systems that now stay synced appear to be using pool.ntp.org, I can’t seem to find where the Beaglbones store the server they use.
Like with most things systemd I’m finding lots of misleading and confusing information that is generally not matching what I’m finding on my systems, but the Beaglebone solution may be as simple as it was for the Lorex device – shorten the sync interval. But how do I find what the interval is and then how do I shorten it?
This system has been running and evolving since 2015, and I’ve noticed the Beaglebones generally would be “off” + or - a few seconds from the beginning of development, but I’m just now getting to adding the feature that needs time sync accurate within one second.
I have been chasing something similar.
On Debian 8.8 and prior Jessie versions, systemd-timesyncd worked very well and issued a lot of (level debug) messages that told you what was going on, and the level of correction being applied.
When it is running the time would stay synced within 20 milliseconds or so.
On Stretch, I have been unable to receive any of these update messages.
Normally every 30 minutes, after initial correction convergence.
rsyslog is correctly configured to capture them.
All I see is a single syslog message at the time of initial sync, then nothing.
I am unable to convince myself that systemd-timesyncd is actually running, even though
Thanks, glad to see I’m not the only one with the issue. Did some more searching and it seems it only logs the sync that happens at daemon start up, but is supposed to resync at some compiled in interval. Check the most recent sync time with the times from stat /var/lib/systemd/clock
On my Debian 8.9 Beaglebone Green I changed /etc/systemd/timesyncd.conf to have a line: NTP=pool.ntp.org
This is the server that I know is being used by the Lorex DVR device which is the most important wrt the time sync with my Beaglebone Green.
I then did:
sudo systemctl stop systemd-timesyncd
sudo systemctl start systemd-timesyncd
and sudo systemctl status systemd-timesyncd showed I was apparently now using the pool.ntp.org server.
Using: stat /var/lib/systemd/clock
Mine seems to be updating every 35 seconds, although apparently the interval will change as sync gets better/worse.
After doing this about 45 minutes ago my Beaglebone is now within 1 second of the Lorex, where it was about 5 seconds off before. Is it possible that extra jitter in the time functions is caused by a fairly heavy and variable workload? tops shows load averages: 0.20 .010. 0.07
OTOH my BBW Debian 9.2 system, which has been up about a day longer than the BBG (did a test reboot yesterday) now seems well synced, but it is very lightly loaded basically just running my ssh session to make these tests.
The systemd-timesyncd update interval seems to be about 34 minutes.
So I’m not sure what is really going on here, but as long as the BBG and the Lorex stay synced to +/- 1 second I don’t need to understand it
If you're serious about keeping time. Then get an accurate real time clock,
and sync your system clock off that. Then update the real time clock once
every month, or however often you need in order to keep time as close to
dead on, as you need it.
I require sync to 1 second, ntp has up to now “just worked” to be that good everywhere I’ve ever used it.
I realize there can be systematic errors if all systems are not using the same time server, that may have been my problem, or something with systemd-timesyncd is FuBar on Debian 8.9 and the BBG, time will tell. The BBW with Debian 9.2 seems fine now that its been running for about two days. maybe it just takes that lonf when booting from a fresh install image…
I assume this answer is really meant for Graham who wants 20 mS sync.
My BBG timesyncd seems to be trying to sync every 35 seconds or so, whereas the BBW seems to be checking every 34 minutes or so.
On Sat, 9 Dec 2017 12:16:30 -0700, William Hermans
<yyrkoon@gmail.com> declaimed the following:
If you're serious about keeping time. Then get an accurate real time clock,
and sync your system clock off that. Then update the real time clock once
every month, or however often you need in order to keep time as close to
dead on, as you need it.
A GPS-based NTP server can be had for around $300, and being on the LAN
would bypass the latency of the Internet when syncing time...
If the devices only need to be close relative to each other, it might
suffice to dedicate one as a local NTP server used by all the others (and
which itself periodically syncs with global NTP server). That too could
remove Internet latency from the situation.
Well, I loaded Debian 8.2 on one of my BBB's, and then did a cold start and
you
can see the following in syslog. timesyncd starts off at a 32 second sntp
resync with the server, then moves up in powers of two to a maximum sync
time of about 34 minutes.
You can see it is making only single digit millisecond corrections to the
system clock, once it converges.
This BBB has had the caps across the master crystal changed to get the
master clock error down into the single digit ppm. Most of the factory
BBB's are in the 50 to 70 ppm error range.
Dec 9 18:28:49 beaglebone systemd-timesyncd[220]: Using NTP server
66.135.44.92:123 (2.debian.pool.ntp.org).
Dec 9 18:28:49 beaglebone systemd-timesyncd[220]:
interval/delta/delay/jitter/drift 32s/-0.182s/0.030s/0.000s/+0ppm
Dec 9 18:29:21 beaglebone systemd-timesyncd[220]:
interval/delta/delay/jitter/drift 64s/-0.001s/0.031s/0.000s/-7ppm
Dec 9 18:30:25 beaglebone systemd-timesyncd[220]:
interval/delta/delay/jitter/drift 128s/+0.002s/0.033s/0.001s/+0ppm
Dec 9 18:32:33 beaglebone systemd-timesyncd[220]:
interval/delta/delay/jitter/drift 256s/-0.001s/0.032s/0.001s/-1ppm
Dec 9 18:36:49 beaglebone systemd-timesyncd[220]:
interval/delta/delay/jitter/drift 512s/-0.000s/0.030s/0.001s/-1ppm
Dec 9 18:45:22 beaglebone systemd-timesyncd[220]:
interval/delta/delay/jitter/drift 1024s/-0.004s/0.036s/0.002s/-1ppm
(ignored)
Dec 9 19:02:26 beaglebone systemd-timesyncd[220]:
interval/delta/delay/jitter/drift 2048s/+0.001s/0.026s/0.002s/-1ppm
(ignored)
Dec 9 19:36:34 beaglebone systemd-timesyncd[220]:
interval/delta/delay/jitter/drift 2048s/+0.007s/0.037s/0.003s/+0ppm
Dec 9 20:10:42 beaglebone systemd-timesyncd[220]:
interval/delta/delay/jitter/drift 2048s/-0.000s/0.038s/0.003s/+0ppm
Dec 9 20:44:51 beaglebone systemd-timesyncd[220]:
interval/delta/delay/jitter/drift 2048s/-0.003s/0.042s/0.003s/+0ppm
The major source of time error is any asymetry in the inbound and outbound
path to the ntp server. (Common issue with cable modems.)
Reporting of this activity disappeared sometime between Debian 8 and Debian
9.
Reading the source code for timesyncd suggests it should still be there, as
cron.debug messages, but so far, no joy.
I'll watch stat /var/lib/systemd/clock to make sure it is running.
If anyone knows how to get the detailed timesyncd messages back, I would
like to hear about it.
It is a front end service for a pool of ntp servers, that can be anywhere in the world.
There are hundreds, if not thousands of them in the pool.
To level the loading on the ntp servers within the pool, you will get assigned to a different server for each call to pool.ntp.org.
To see which server you are actually using, you need to look at the IP address of the server, then do a whois lookup.
It is likely that each of your BBB’s is running off a different ntp server, even though they all called to pool.ntp.org.
But, to get admitted to that pool, you need to be running a full implementation of NTP, which is cross synched with at least five other servers, and each will likely be less than 10 ms in error from absolute time.
Many these days have their own GPS receiver, so will have time errors less than 0.1 ms.
If you want all of your units to use the same server, you will need to specify the IP address of the specific server, in that NTP= statement.
Preferably one close to you, with low latency and a symmetric path.
So apparently it tries to run both and ntp wins
I don’t recall doing anything about its time keeping other than setting the locale during installation in 2016.
My BBG seems to be either right on or two seconds slow compared to the others It still seems to be updating every 35 seconds.
Still puzzled. I’ll give it more time, and maybe try getting time from a single server if it doesn’t stablize.
