System clock ticking too fast on BeagleBones

GoogleGroups
1

Hello,

We are using tens of BeagleBones running the same 3.2.30-psp23 kernel and all powered using a 5V/2A power adaptor. On almost one third of nodes system clock is ticking too fast, approximatelly 4 times faster (i.e. Fri Nov 30 16:15:19 UTC 2012 which is correct time versus Mon Dec 3 17:02:37 UTC 2012). We are also running ntpd on all nodes, but larger time difference causes it to panic and exit (-g flag helps, but only for a very short time).

After some investigation I’ve noticed that some of the nodes boot with 32 KHz clock and some with 24MHz clock, but there is no rule that a particular clock is causing a time skew:

[node1]

$ grep -i clock /var/log/dmesg
[ 0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz
[ 0.000000] omap_dm_timer_switch_src: Switching to HW default clocksource(sys_clkin_ck) for timer1, this may impact timekeeping in low power state
[ 0.000000] OMAP clocksource: GPTIMER1 at 24000000 Hz
[ 0.000000] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 178956ms
[ 0.173176] Switching to clocksource gp timer
[ 2.273396] omap_rtc omap_rtc: setting system clock to 2000-01-01 00:00:00 UTC (946684800)

[node2]

$ grep -i clock /var/log/dmesg
[ 0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz
[ 0.000000] OMAP clocksource: GPTIMER1 at 32768 Hz
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms
[ 0.174926] Switching to clocksource gp timer
[ 2.253112] omap_rtc omap_rtc: setting system clock to 2012-10-25 19:56:51 UTC (1351195011)

I tried to compare /sys/kernel/debug/clock/summary files among nodes (after having mounted debugfs as per http://processors.wiki.ti.com/index.php/AM335x_PSP_User’s_Guide#Clock_Management_details) and I noticed that timer1_fck has different parent clock:

dziugas@evalg-data1:/tmp/clock$ diff -u1 node1.summary node2.summary
— node1.summary 2012-11-09 10:40:01.660509296 +0100
+++ node2.summary 2012-11-09 10:42:02.932847298 +0100
@@ -73,3 +73,3 @@
timer2_fck sys_clkin_ck 24000000 1
-timer1_fck sys_clkin_ck 24000000 1
+timer1_fck clk_32768_ck 32768 1
timer0_fck clk_rc32k_ck 32000 0
@@ -170,3 +170,3 @@
tclkin_ck none 12000000 0
-sys_clkin_ck virt_24m_ck 24000000 8
+sys_clkin_ck virt_24m_ck 24000000 7
virt_26m_ck none 26000000 0
@@ -177,2 +177,2 @@
clk_32khz_ck clkdiv32k_ick 32768 0
-clk_32768_ck none 32768 1
+clk_32768_ck none 32768 2

So I suspect that for some reason parent clock is changed after the boot, but system clock is still calculating jiffies according to the frequency that was available during the boot and it causes a time drift.

Does anybody else have the same problem and could assist finding a solution for it? Recent clock changes by TI mainly deal with suspend feature (http://e2e.ti.com/support/dsp/sitara_arm174_microprocessors/f/791/p/180128/654964.aspx) which we do not use, so there should be something else causing this clock problem. Our nodes are placed in different physical locations, so unreliable power source could hardly be the cause. For now we plan to run ntpdate every minute instead of ntpd, but this is a workaround rather than solution.

Thanks,
Džiugas

2

I’ve not noted clock skews on mines but I have seen that sometimes the clock source is 32Khz XTAL and other the 24Mhz XTAL.

We run Debian wheezy witch calls ntpdate when Internet connection is up.

root@bb:/home/juanjo# date
Sat Dec 1 11:31:32 CLST 2012

root@bb:/home/juanjo# hwclock
Sat 01 Dec 2012 11:31:36 AM CLST -0.910919 seconds

root@bb:/home/juanjo# w
11:31:48 up 3 days, 19:50, 1 user, load average: 0.01, 0.02, 0.05
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
juanjo pts/0 10.10.100.20 11:28 0.00s 0.90s 0.73s sshd: juanjo [priv]

Kernel logs:

[ 0.000000] IRQ: Found an INTC at 0xfa200000 (revision 5.0) with 128 interrupts
[ 0.000000] Total of 128 interrupts on 1 active controller
[ 0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz
[ 0.000000] OMAP clocksource: GPTIMER1 at 32768 Hz
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms

One detail, in my Kernel I have RTC as a module waiting to implement my own. So some things RTC related are not called at the same time and I boot with fixrtc:

[ 0.000000] Kernel command line: console=ttyO0,115200n8 fixrtc root=/dev/mmcblk0p2 ro rootfstype=ext4 rootwait ip=none

As I have the RTC as a module it does get load after rootfs is mounted:

[ 1.822662] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[ 1.830566] Waiting for root device /dev/mmcblk0p2…
[ 1.844207] mmc0: host does not support reading read-only switch. assuming write-enable.
[ 1.854644] mmc0: new high speed SD card at address b368
[ 1.860931] mmcblk0: mmc0:b368 AF UD 1.86 GiB
[ 1.867523] mmcblk0: p1 p2
[ 1.955352] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 1.963989] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
[ 1.981475] devtmpfs: mounted
[ 1.985015] Freeing init memory: 240K
[ 3.236602] udevd[147]: starting version 175
[ 4.549011] omap_rtc omap_rtc: rtc core: registered omap_rtc as rtc0

3

Hello,

We are using tens of BeagleBones running the same 3.2.30-psp23 kernel
and all powered using a 5V/2A power adaptor. On almost one third of
nodes system clock is ticking too fast, approximatelly 4 times faster
(i.e. Fri Nov 30 16:15:19 UTC 2012 which is correct time versus Mon Dec
3 17:02:37 UTC 2012). We are also running ntpd on all nodes, but larger
time difference causes it to panic and exit (-g flag helps, but only for
a very short time).

After some investigation I've noticed that some of the nodes boot with
32 KHz clock and some with 24MHz clock, but there is no rule that a
particular clock is causing a time skew:

There is specific reason behind configuring them.

Timer 1 is used for system timers (clocksource) and we need it tobe
ticking across suspend/resume cycle, so we need persistent clock.
In case of AM335x, RTC 32k clock is the only clock available which is
persistent.

[node1]

$ grep -i clock /var/log/dmesg
[ 0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz
[ 0.000000] omap_dm_timer_switch_src: Switching to HW default
clocksource(sys_clkin_ck) for timer1, this may impact timekeeping in low
power state
[ 0.000000] OMAP clocksource: GPTIMER1 at 24000000 Hz
[ 0.000000] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps
every 178956ms
[ 0.173176] Switching to clocksource gp timer
[ 2.273396] omap_rtc omap_rtc: setting system clock to 2000-01-01
00:00:00 UTC (946684800)

[node2]

$ grep -i clock /var/log/dmesg
[ 0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz
[ 0.000000] OMAP clocksource: GPTIMER1 at 32768 Hz
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps
every 131071999ms
[ 0.174926] Switching to clocksource gp timer
[ 2.253112] omap_rtc omap_rtc: setting system clock to 2012-10-25
19:56:51 UTC (1351195011)

I tried to compare /sys/kernel/debug/clock/summary files among nodes
(after having mounted debugfs as
per http://processors.wiki.ti.com/index.php/AM335x_PSP_User’s_Guide#Clock_Management_details)
and I noticed that timer1_fck has different parent clock:

dziugas@evalg-data1:/tmp/clock$ diff -u1 node1.summary node2.summary
--- node1.summary2012-11-09 10:40:01.660509296 +0100
+++ node2.summary2012-11-09 10:42:02.932847298 +0100
@@ -73,3 +73,3 @@
timer2_fck sys_clkin_ck 24000000 1
-timer1_fck sys_clkin_ck 24000000 1
+timer1_fck clk_32768_ck 32768 1
timer0_fck clk_rc32k_ck 32000 0
@@ -170,3 +170,3 @@
tclkin_ck none 12000000 0
-sys_clkin_ck virt_24m_ck 24000000 8
+sys_clkin_ck virt_24m_ck 24000000 7
virt_26m_ck none 26000000 0
@@ -177,2 +177,2 @@
clk_32khz_ck clkdiv32k_ick 32768 0
-clk_32768_ck none 32768 1
+clk_32768_ck none 32768 2

So I suspect that for some reason parent clock is changed after the
boot, but system clock is still calculating jiffies according to the
frequency that was available during the boot and it causes a time drift.

Does anybody else have the same problem and could assist finding a
solution for it? Recent clock changes by TI mainly deal with suspend
feature
(TMS570LS3137: Kelvin_GND pin 19 question - Arm-based microcontrollers forum - Arm-based microcontrollers - TI E2E support forums)
which we do not use, so there should be something else causing this
clock problem. Our nodes are placed in different physical locations, so
unreliable power source could hardly be the cause. For now we plan to
run ntpdate every minute instead of ntpd, but this is a workaround
rather than solution.

You can try to change the timer1 parent and see if it fixes this issue.
To change the parent of the timer by,

Modify "arch/arm/mach-omap2/timer.c" with,

diff --git a/arch/arm/mach-omap2/timer.c b/arch/arm/mach-omap2/timer.c
index 1af3871..91abfd3 100644
--- a/arch/arm/mach-omap2/timer.c
+++ b/arch/arm/mach-omap2/timer.c
@@ -420,7 +420,7 @@ OMAP_SYS_TIMER(3)
OMAP_SYS_TIMER_INIT(3_secure, OMAP3_SECURE_TIMER, OMAP3_CLKEV_SOURCE,
                        2, OMAP3_MPU_SOURCE)
OMAP_SYS_TIMER(3_secure)
-OMAP_SYS_TIMER_INIT(3_am33xx, 2, OMAP4_MPU_SOURCE, 1, AM33XX_RTC32K_SOURCE)
+OMAP_SYS_TIMER_INIT(3_am33xx, 2, OMAP4_MPU_SOURCE, 1, OMAP4_MPU_SOURCE)
OMAP_SYS_TIMER(3_am33xx)
#endif

Thanks,
Vaibhav

4

So basically getting a 24Mhz clock is a bug ?

[ 0.000000] OMAP clocksource: GPTIMER1 at 24000000 Hz
[ 0.000000] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 178956ms
[ 0.173176] Switching to clocksource gp timer

Because I also noted some of my Beagles take GPTIMER1 as 24Mhz and others as 32Khz

5

So basically getting a 24Mhz clock is a bug ?

No its not a bug.

If you have 32Khz crystal connected to RTC module, then timer will be
sourced from RTC-32Khz clock. And we have fall back mechanism for this, if
timer fails to get 32khz clock then it falls back to 24Mhz crystal clock.

[ 0.000000] OMAP clocksource: GPTIMER1 at 24000000 Hz
[ 0.000000] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps
every 178956ms
[ 0.173176] Switching to clocksource gp timer

Because I also noted some of my Beagles take GPTIMER1 as 24Mhz and
others as 32Khz

Most likely, either you don’t have 32k crystal OR you don’t have below patch
in u-boot
http://arago-project.org/git/projects/?p=u-boot-am33x.git;a=commit;h=5fdfe4d225485ac6a63b874143f1a784d5394f55

Thanks,
Vaibhav

6

Hello,

Thanks for the feedback. Indeed, none of the 3.2.x kernels from Robert Nelson’s tree (https://github.com/RobertCNelson/linux-dev/tree/am33x-v3.2/patches) do have the patch you’ve pointed (it does not even contain arch/arm/cpu/armv7 folder).

However, I’m also wondering if it is really the case that some cape-less BeagleBones of the same revision (we are using mostly Rev. 6) are equipped with 32 Khz crystal while others not?

I’ve just installed v3.2_AM335xPSP_04.06.00.09-rc2 and I can reproduce the following (feeding power from a power adaptor, no mini USB cable connected):

  1. If the board is booted when power plug was previously unplugged, clock is always 24 MHz:

grep -i clock /var/log/dmesg

[ 0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz
[ 0.000000] omap_dm_timer_switch_src: Switching to HW default clocksource(sys_clkin_ck) for timer1, this may impact timekeeping in low power state
[ 0.000000] OMAP clocksource: GPTIMER1 at 24000000 Hz
[ 0.000000] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 178956ms
[ 0.139246] PTP clock support registered
[ 0.141670] Switching to clocksource gp timer
[ 1.543804] clock: disabling unused clocks to save power
[ 1.564633] omap_rtc am33xx-rtc: setting system clock to 2000-01-01 00:00:00 UTC (946684800)

grep timer1 /sys/kernel/debug/clock/summary

timer1_ick l4_wkup_gclk 100000000 1
timer1_fck sys_clkin_ck 24000000 1

  1. If the reboot is triggered by pressing the reset button or issuing a ‘reboot’ command, clock is always 32 KHz:

grep -i clock /var/log/dmesg

[ 0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz
[ 0.000000] OMAP clocksource: GPTIMER1 at 32768 Hz
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms
[ 0.141387] PTP clock support registered
[ 0.143798] Switching to clocksource gp timer
[ 1.530700] clock: disabling unused clocks to save power
[ 1.551361] omap_rtc am33xx-rtc: setting system clock to 2012-12-10 16:33:40 UTC (1355157220)

grep timer1 /sys/kernel/debug/clock/summary

timer1_ick l4_wkup_gclk 100000000 1
timer1_fck clk_32768_ck 32768 1

And this makes to believe that the selection of the clocksource is somehow stateful, even though board is powered from the same source all the time. And since our nodes are not likely to be hit by any suspension mechanism, I still don’t get what makes a clocksource to change while the board is up and running.

Thanks,
Džiugas

7

Well, Vaibhav's patch does not touch the kernel tree.. It's for the
bootloader (u-boot), it also happens to be included in the u-boot
"v2013.01-rc1" release..

So... If (and only if) you are running my demo images, look at:

/boot/uboot/tools

is there an "update.sh" file?
if so run "sudo ./update.sh" then..

sudo ./bootloader_update.sh --use-beta-bootloader

Regards,

8

Hello,

Thanks for the feedback. Indeed, none of the 3.2.x kernels from Robert
Nelson's tree
(https://github.com/RobertCNelson/linux-dev/tree/am33x-v3.2/patches) do
have the patch you've pointed (it does not even contain
arch/arm/cpu/armv7 folder).

However, I'm also wondering if it is really the case that some cape-less
BeagleBones of the same revision (we are using mostly Rev. 6) are
equipped with 32 Khz crystal while others not?

I've just installed v3.2_AM335xPSP_04.06.00.09-rc2 and I can reproduce
the following (feeding power from a power adaptor, no mini USB cable
connected):

1. If the board is booted when power plug was previously unplugged,
clock is always 24 MHz:

# grep -i clock /var/log/dmesg
[ 0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz
[ 0.000000] omap_dm_timer_switch_src: Switching to HW default
clocksource(sys_clkin_ck) for timer1, this may impact timekeeping in low
power state
[ 0.000000] OMAP clocksource: GPTIMER1 at 24000000 Hz
[ 0.000000] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps
every 178956ms
[ 0.139246] PTP clock support registered
[ 0.141670] Switching to clocksource gp timer
[ 1.543804] clock: disabling unused clocks to save power
[ 1.564633] omap_rtc am33xx-rtc: setting system clock to 2000-01-01
00:00:00 UTC (946684800)

# grep timer1 /sys/kernel/debug/clock/summary
timer1_ick l4_wkup_gclk 100000000
1
timer1_fck sys_clkin_ck 24000000
1

2. If the reboot is triggered by pressing the reset button or issuing a
'reboot' command, clock is always 32 KHz:

# grep -i clock /var/log/dmesg
[ 0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz
[ 0.000000] OMAP clocksource: GPTIMER1 at 32768 Hz
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps
every 131071999ms
[ 0.141387] PTP clock support registered
[ 0.143798] Switching to clocksource gp timer
[ 1.530700] clock: disabling unused clocks to save power
[ 1.551361] omap_rtc am33xx-rtc: setting system clock to 2012-12-10
16:33:40 UTC (1355157220)

# grep timer1 /sys/kernel/debug/clock/summary
timer1_ick l4_wkup_gclk 100000000
1
timer1_fck clk_32768_ck 32768
1

And this makes to believe that the selection of the clocksource is
somehow stateful, even though board is powered from the same source all
the time. And since our nodes are not likely to be hit by any suspension
mechanism, I still don't get what makes a clocksource to change while
the board is up and running.

This is interesting data pointers,

I do not have BeagleBone in front of me right now, let me try this out
tomorrow once I get into office, will share my observation.

Thanks,
Vaibhav

9

Sorry, I haven’t noticed the patch was for u-boot and thanks for pointing this out.

I’ve retested the initial clock selection using u-boot from arago-project (so with the mentioned patch in place) and I can confirm that the clock source is always 32 MHz no matter how the board was booted:

grep -i clock /var/log/dmesg

[ 0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz
[ 0.000000] OMAP clocksource: GPTIMER1 at 32768 Hz
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms
[ 0.141113] PTP clock support registered
[ 0.143066] Switching to clocksource gp timer
[ 1.395019] clock: disabling unused clocks to save power
[ 1.414581] omap_rtc am33xx-rtc: setting system clock to 2000-01-01 00:00:00 UTC (946684800)

So I’m wondering to know whether the consistence of the initial clocksource will solve the original problem of the system clock start ticking to fast, i.e. what are the conditions for system clock not to change while the system is running?

Džiugas

10

The best solution to this problem is (if locally connected)
Install Local NTP server and put “service ntp restart” in an infinite loop with sleep time 30 seconds approx. by writing a code in “/etc/init.d/rc.local” file. Reboot system and time will be synchronized with the server computer.