I2C timeouts

Yesterday, at IRC there was some discussion about I2C timeouts [1]. Some observations, maybe they can help. I only talk about Beagle, *not* about EVM.

U-Boot v1

nishant_uboot_v2.c (16.8 KB)

Created issue #17

http://code.google.com/p/beagleboard/issues/detail?id=17

for this.

Dirk Behme wrote:

Dirk,

If your are running one of the angstrom based filesystems: ‘opkg
install http://www.angstrom-distribution.org/feeds/2008/ipk/glibc/armv6/base/i2c-tools_3.0.1-r0_armv6.ipk

regards,

Koen

Menon, Nishanth wrote:

Dirk,

From: beagleboard@googlegroups.com [mailto:beagleboard@googlegroups.com] On Behalf Of Dirk Behme
Sent: Sunday, June 15, 2008 9:10 AM
To: beagleboard@googlegroups.com
Subject: [beagleboard] I2C timeouts

Compiling recent OMAP git head beagle defconfig with CSL 2008q1-126
doesn't show any I2C errors. Most probable because no I2C is used? Any
easy kernel I2C test cases?

Do try this: http://www.lm-sensors.org/wiki/I2CTools
I2cdetect, i2cdump, set and get are good enough for testing i2c ops. Further, you will need to enable i2c_chardev to use this test cases.

Thanks to Nishant's help (thanks!), using i2cdetect with OMAP git defconfig results in:

# i2cdetect -a -y -r 1
      0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- UU UU UU UU -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
# i2cdetect -a -y -r 2
      0 1 2 3 4 5 6 7 8 9 a b c d e f
00: <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- ^C<3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out

# i2cdetect -a -y -r 3
      0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

Dirk,

Not really usefull for debugging, but recent kernel list all i2c
devices in /sys:

root@beagleboard:/sys/bus/i2c/devices# ls .
1-0048 1-004a 3-0050 3-0052 3-0054 3-0056
1-0049 1-004b 3-0051 3-0053 3-0055 3-0057
root@beagleboard:/sys/bus/i2c/devices# cat 3-0050/eeprom
...).x&#65533;&#65533;.&#65533;XL.&.PT/
&#65533;&#65533;@..a@aYE@EY1@1YH?
@0b&#65533;2@@&#65533;..2..&#65533;2UP.
      &#65533;AU5131
&#65533;
...).x&#65533;&#65533;.&#65533;XL.&.PT/
&#65533;&#65533;@..a@aYE@EY1@1YH?
@0b&#65533;2@@&#65533;..2..&#65533;2UP.
      &#65533;AU5131
&#65533;
            Q
root@beagleboard:/sys/bus/i2c/devices#

My monitor is an IIyama AU5131, so EDID readout works :slight_smile:

Menon, Nishanth wrote:

Dirk,

From: beagleboard@googlegroups.com [mailto:beagleboard@googlegroups.com] On Behalf Of Dirk Behme
Sent: Monday, June 16, 2008 1:28 PM
To: beagleboard@googlegroups.com
Subject: [beagleboard] Re: I2C timeouts

# i2cdetect -a -y -r 2
     0 1 2 3 4 5 6 7 8 9 a b c d e f
00: <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- ^C<3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out

Check pin mux of u-boot v2/kernel which ever. For some reason ( I have an hypothesis), without the proper muxing the timeout will happen. My hypothesis is that without terminating resistors, the i2c controller cannot detect NAK condition on bus, hence timesout.

The proof of concept on SDP3430 -> SDP3430 uses 2 busses ->i2c1 and i2c2. i missed moving my i2c pin mux out of a #define, and well.. I was seeing i2c timeouts in bus2 but not in bus1 no matter which ever speed I try. After ages of kicking myself, I decided to look at my pin mux, and bingo, bus2 started working.. as a test I decided to try i2c3, and it did not work.. so I am guessing I don't have terminating resistors out there/ I goofed up my pin mux for bus3 (yet again)..

But that does seem to show the issue.

Okay, yes your are right, it is pin mux. Thanks for this!

Checked with beagle:

At beagle, I2C2 is routed to expansion connector ( [1] page 81, table 18, lines 23 & 24). Alternatively, I2C SDA and SCL can be used as GPIO's 183 & 168. And this is what recent uboot pin mux configures:

http://www.sakoman.net/cgi-bin/gitweb.cgi?p=u-boot-omap3.git;a=blob;f=include/asm-arm/arch-omap3/mux.h;h=33947b9e8a078d36f43c95a720e1e6279e5e61a4;hb=refs/heads/test

line 637 & 638. I.e. uboot configures GPIO pins and not I2C2.

So we have two options to fix I2C2 timeout issues:

a) Disable I2C2 in uboot and kernel (how to do this?).

b) Configure (without anything at expansion connector: unused) I2C2 and not GPIOs in uboot's pin mux. For this we could use something like in attachment.

All: What do people think which option should we take?

Dirk

[1] http://www.beagleboard.org/uploads/Beagle_HW_Reference_Manual_A_5.pdf

uboot_v1_git_steve_i2c2_pin_mux_patch.txt (940 Bytes)

Okay, yes your are right, it is pin mux. Thanks for this!

Checked with beagle:

Good to know the issue…

At beagle, I2C2 is routed to expansion connector ( [1] page 81, table
18, lines 23 & 24). Alternatively, I2C SDA and SCL can be used as
GPIO’s 183 & 168. And this is what recent uboot pin mux configures:

http://www.sakoman.net/cgi-bin/gitweb.cgi?p=u-boot-omap3.git;a=blob;f=include/asm-arm/arch-omap3/mux.h;h=33947b9e8a078d36f43c95a720e1e6279e5e61a4;hb=refs/heads/test

line 637 & 638. I.e. uboot configures GPIO pins and not I2C2.

So we have two options to fix I2C2 timeout issues:

a) Disable I2C2 in uboot and kernel (how to do this?).

In Kernel: We can remove this from platform driver, the platform driver should register only two interfaces of I2C instances (one for T2 and other for EDID - DVI).
u-boot: shouldn’t crib about this as it doesn’t do any initialization for I2C during boot.

b) Configure (without anything at expansion connector: unused) I2C2
and not GPIOs in uboot’s pin mux. For this we could use something like
in attachment.

We wanted to test the Expansion connector signals for production, that was the only reason to configure them as GPIOs instead of I2C. May be option (a) is better.

I am getting similar queries on EVM for I2C2 from other folks, they say, sending large amount of data over I2C2 results in timeouts. I don’t know if this still holds good here.

Regards,
Khasim

Dirk Behme wrote:

Menon, Nishanth wrote:

Dirk,

From: beagleboard@googlegroups.com [mailto:beagleboard@googlegroups.com] On Behalf Of Dirk Behme
Sent: Monday, June 16, 2008 1:28 PM
To: beagleboard@googlegroups.com
Subject: [beagleboard] Re: I2C timeouts

# i2cdetect -a -y -r 2
     0 1 2 3 4 5 6 7 8 9 a b c d e f
00: <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- ^C<3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out

Check pin mux of u-boot v2/kernel which ever. For some reason ( I have an hypothesis), without the proper muxing the timeout will happen. My hypothesis is that without terminating resistors, the i2c controller cannot detect NAK condition on bus, hence timesout.

The proof of concept on SDP3430 -> SDP3430 uses 2 busses ->i2c1 and i2c2. i missed moving my i2c pin mux out of a #define, and well.. I was seeing i2c timeouts in bus2 but not in bus1 no matter which ever speed I try. After ages of kicking myself, I decided to look at my pin mux, and bingo, bus2 started working.. as a test I decided to try i2c3, and it did not work.. so I am guessing I don't have terminating resistors out there/ I goofed up my pin mux for bus3 (yet again)..

But that does seem to show the issue.

Okay, yes your are right, it is pin mux. Thanks for this!

Checked with beagle:

At beagle, I2C2 is routed to expansion connector ( [1] page 81, table 18, lines 23 & 24). Alternatively, I2C SDA and SCL can be used as GPIO's 183 & 168. And this is what recent uboot pin mux configures:

http://www.sakoman.net/cgi-bin/gitweb.cgi?p=u-boot-omap3.git;a=blob;f=include/asm-arm/arch-omap3/mux.h;h=33947b9e8a078d36f43c95a720e1e6279e5e61a4;hb=refs/heads/test

line 637 & 638. I.e. uboot configures GPIO pins and not I2C2.

Changing this with below uboot pin mux changes *doesn't* work:

# i2cdetect -a -y -r 2
      0 1 2 3 4 5 6 7 8 9 a b c d e f
00: <3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out
-- <3>i2c_omap i2c_omap.2: Arbitration lost
i2c_omap i2c_omap.2: Arbitration lost
-- ^C<3>i2c_omap i2c_omap.2: controller timed out
i2c_omap i2c_omap.2: controller timed out

Some additional results from discussion at IRC [1]:

- On EVM a team reported I2C timeout issues on I2C2 and there the I2C2 is connected to a camera module. So at beagle we are not alone with this issue

- If I2C2 pin mux (pull up) and clocks are correct, there should be *no* timeouts even if nothing is connected to beagle's expansion connector (I2C2 unused)

- Khasim proposed to comment the omap_i2c_idle in omap_i2c_xfer in drivers/i2c/busses/i2c-omap.c (sth like in attachment)

This makes things worse, system crashes already at boot:

-- cut --
...
<3>USB: No board-specific platform config found
<6>i2c_omap i2c_omap.1: bus 1 rev3.12 at 2600 kHz
<6>i2c_omap i2c_omap.2: bus 2 rev3.12 at 400 kHz
<6>i2c_omap i2c_omap.3: bus 3 rev3.12 at 400 kHz
<6>TWL4030: TRY attach Slave TWL4030-ID0 on Adapter OMAP I2C adapter [1]
<6>TWL4030: TRY attach Slave TWL4030-ID1 on Adapter OMAP I2C adapter [1]
<6>TWL4030: TRY attach Slave TWL4030-ID2 on Adapter OMAP I2C adapter [1]
<6>TWL4030: TRY attach Slave TWL4030-ID3 on Adapter OMAP I2C adapter [1]
<1>Unhandled fault: external abort on non-linefetch (0x1028) at 0xd8070008
Internal error: : 1028 [#1]
Modules linked in:
CPU: 0 Not tainted (2.6.26-rc6-omap1-04677-g5b97d3e-dirty #3)
PC is at omap_i2c_xfer+0x78/0x27c
LR is at msecs_to_jiffies+0x28/0x4c
pc : [<c02c7b08>] lr : [<c0197ed0>] psr: 80000013
sp : c7c1de58 ip : 00000000 fp : c7c1de94
r10: c7c1def6 r9 : 00000000 r8 : 00000001
r7 : c044a148 r6 : c7c90600 r5 : c0439f70 r4 : ffff6aa7
r3 : d8070000 r2 : 00000080 r1 : 00000200 r0 : 00000001
Flags: Nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 00c5387f Table: 80004018 DAC: 00000017
Process swapper (pid: 1, stack limit = 0xc7c1c2e0)
...
Backtrace:
[<c02c7a90>] (omap_i2c_xfer+0x0/0x27c) from [<c02c53c8>] (i2c_transfer+0x88/0x9c)
[<c02c5340>] (i2c_transfer+0x0/0x9c) from [<c02c8938>] (twl4030_i2c_write+0xc0/0xe8)
  r6:00000011 r5:c044a160 r4:c0449ff8
[<c02c8878>] (twl4030_i2c_write+0x0/0xe8) from [<c02c8988>] (twl4030_i2c_write_u8+0x28/0x30)
[<c02c8960>] (twl4030_i2c_write_u8+0x0/0x30) from [<c02c8aec>] (twl4030_attach_adapter+0x15c/0x698)
[<c02c8990>] (twl4030_attach_adapter+0x0/0x698) from [<c02c6228>] (i2c_register_driver+0xa8/0xec)
[<c02c6180>] (i2c_register_driver+0x0/0xec) from [<c001a334>] (twl4030_init+0x18/0x20)
  r7:c001a31c r6:00000000 r5:00000000 r4:c016c000
[<c001a31c>] (twl4030_init+0x0/0x20) from [<c0008934>] (kernel_init+0xa4/0x240)
[<c0008890>] (kernel_init+0x0/0x240) from [<c0196c10>] (do_exit+0x0/0x5f0)
Code: ea000057 ebfb531c e5963004 e3a00001 (e1d330b8)
<4>---[ end trace 1b75b31a2719ed1c ]---
<0>Kernel panic - not syncing: Attempted to kill init!
-- cut --

[1] http://www.beagleboard.org/irclogs/index.php?date=2008-06-20#T17:02:03

i2c_idle_patch.txt (562 Bytes)

I spent some time looking at this issue today.

Regarding the SCL and SDA lines, the TRM states "Open-drain output
buffer. Requires external pullup resistor."

The beagle design does not include external pull up resistors on the
I2C2 signals. My guess is that the trace capacitance of the signal
runs to the expansion connector are more than the internal pull-ups
can handle.

To test this theory I added pullups to both signals. I also made the
mux change described above to u-boot.

The result was that the timeout errors disappeared, tending to confirm
this theory.

Our choices seem to be:

1. Leave the u-boot mux setting for these signals as GPIOs and disable
the I2C2 channel in the linux board file for beagle. Adjust
expansion connector docs to reflect this.
2. Fix the u-boot mux settings, live with the timeout errors at boot
(they will disappear if/when expansion hw with pullups is connected)
3. Fix the u-boot mux setting and add pullup resistors. There is some
debate about whether the pullups should be to 1.8 or 3V

Steve

Response below...

Our choices seem to be:

1. Leave the u-boot mux setting for these signals as GPIOs and disable
the I2C2 channel in the linux board file for beagle. Adjust
expansion connector docs to reflect this.

I believe this is our best option because it can be applied in
software and will make all boards work. It further provides the most
flexibility to whomever creates an expansion device. Documentation
will need to be updated accordingly.

2. Fix the u-boot mux settings, live with the timeout errors at boot
(they will disappear if/when expansion hw with pullups is connected)

This will slow the boot and confuse the user.

3. Fix the u-boot mux setting and add pullup resistors. There is some
debate about whether the pullups should be to 1.8 or 3V

This won't work for people with boards today--they would have to
modify themselves.

The pullups are inside the level shifters. Please refer to the hardware
reference manual.

Gerald

You are correct. I was thinking the EDID channel. We could add them but
they could interfere with the operation of the lines if they were used
for other things on the expansion connector. As we have many lines on
the OMAP3 device that are not used in the BeagleBoard as they are on
other platforms, the SW needs to take into account pins that are not
used and set them accordingly. We cannot put pullups and pulldowns on
every pin to keep the SW compatible with all OMAP based boards.

Gerald

Jason Kridner wrote:

Response below...

Our choices seem to be:

1. Leave the u-boot mux setting for these signals as GPIOs and disable
the I2C2 channel in the linux board file for beagle. Adjust
expansion connector docs to reflect this.

I believe this is our best option because it can be applied in
software and will make all boards work.

I will try to do a kernel patch for this and update issue #17 accordingly.

Thanks to Steve for debugging!

Dirk

> 2. Fix the u-boot mux settings, live with the timeout errors at boot
> (they will disappear if/when expansion hw with pullups is connected)

This will slow the boot and confuse the user.

Actually, with u-boot mux settings fixed the boot is very much faster
and the timeout messages change to lost arbitration messages which
spit out quite snappily.

Steve

Is that an argument for leaving the pins defined as I2C? I'm still
partial to leaving them GPIO, only because of the lack of (strong-
enough internal?) pull-ups and pull-ups would interfere with pure GPIO
functions.

Jason,

It really is your call -- I was just adding information because one of
your decision factors previously was the slower boot waiting for
timeouts. That doesn't seem to be an issue anymore.

If you want 2 GPIO's we'll have to change git u-boot and the git
kernel. The git u-boot change will be simple since we still control
that process. The kernel change will need to go through the linux-
omap patch submission process.

Personally I would prefer i2c for those 2 pins since it is a whole lot
more useful than 2 GPIOs. But I can definitely see your point about
wanting to minimize confusion with boot messages, so I'm happy to
support whatever decision you make.

Steve

If the internal pull-ups work to prevent any boot delays, then I2C
seems better to me per our previous discussions (use if I2C GPIO
expanders, etc.). I only had doubts about the boot time and
reliability. Sounds like you've worked that out.

In conclusion, I2C.

Jason Kridner wrote:

Response below...

Our choices seem to be:

1. Leave the u-boot mux setting for these signals as GPIOs and disable
the I2C2 channel in the linux board file for beagle. Adjust
expansion connector docs to reflect this.

I believe this is our best option because it can be applied in
software and will make all boards work. It further provides the most
flexibility to whomever creates an expansion device. Documentation
will need to be updated accordingly.

Sent a patch to OMAP list for this:

http://marc.info/?l=linux-omap&m=121515287023862&w=2

Please have a look if you like it and comment there.

Thanks

Dirk