USB Networking via g_ether on Pocket Beagle 2 Industrial without configfs

General Discussion
1

Hello, I am trying to get USB networking on my PB2 Industrial variant via the USB gadget driver g_ether in the kernel to setup a NFS rootfs. It does not seem to work, but when I enable debug logs, it starts working - I have no idea what is going on. Any help would be appreciated. Thanks.

Kernel - 6.18.37

With no debug I get this on the host side with no network interface:

[21403.048184] usb 1-1: new high-speed USB device number 44 using xhci_hcd
[21403.175835] usb 1-1: New USB device found, idVendor=0525, idProduct=a4a2, bcdDevice= 6.18
[21403.175851] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[21403.175855] usb 1-1: Product: RNDIS/Ethernet Gadget
[21403.175858] usb 1-1: Manufacturer: Linux 6.18.37 with dwc3-gadget
[21408.389363] usb 1-1: can't set config #1, error -110

The related kernel logs are here:
nodebug-logs.txt (24.8 KB)

With loglevel=8 and initcall_debug enabled I get this:

[21489.137092] usb 1-1: new high-speed USB device number 47 using xhci_hcd
[21489.265547] usb 1-1: New USB device found, idVendor=0525, idProduct=a4a2, bcdDevice= 6.18
[21489.265563] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[21489.265568] usb 1-1: Product: RNDIS/Ethernet Gadget
[21489.265571] usb 1-1: Manufacturer: Linux 6.18.37 with dwc3-gadget
[21489.310631] cdc_subset 1-1:1.0: probe with driver cdc_subset failed with error -22
[21489.312453] cdc_ether 1-1:1.0 usb0: register 'cdc_ether' at usb-0000:03:00.3-1, CDC Ethernet Device, de:19:26:e4:3c:5a
[21489.360343] cdc_ether 1-1:1.0 enp3s0f3u1: renamed from usb0

10: enp3s0f3u1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether de:19:26:e4:3c:5a brd ff:ff:ff:ff:ff:ff
    altname enxde1926e43c5a

Related kernel logs:
debug-logs.txt (297.9 KB)

I tested this mutliple times and it seems the behaviour is consistent. But I don’t understand why is this happening.

The Kernel Image in the EMMC seems to us configfs to do the networking setup and it works everytime. I do not have a rootfs so I can not use that, I looked it up I guess I could use initramfs to do this setup but I want to keep it as my last resort.

I have more logs.

Kernel 7.1.0 and with loglevel=8 initcall_debug dyndbg="file drivers/base/dd.c +p" enabled:
kernel-driver-debug-logs.txt (640.0 KB)

EMMC logs:
pb2-emmc-boot-log.txt (61.0 KB)

2

Doesn’t the default distribution use libcomposite over g_ether? Let me check …

… ah the default distribution uses libcomposite, but calls the device g_multi - I call it usbg!

Anyway I’d check that /sys/kernel/config/usb_gadget/g_multi/ looks like you expect it to be

3

I don’t really understand what you are suggesting me to do here - Can you be a little more clear please :sweat_smile:

I get that the default distribution uses libcomposite but that is over configfs right? I am trying to avoid that and get a static driver solution like g_ether or something since I do not have a rootfs at this point - and I am reserving initramfs as the last resort.

4

this is what mine looks like

|-- UDC
|-- bDeviceClass
|-- bDeviceProtocol
|-- bDeviceSubClass
|-- bMaxPacketSize0
|-- bcdDevice
|-- bcdUSB
|-- configs
|   `-- c.1
|       |-- MaxPower
|       |-- acm.usb0 -> ../../../../usb_gadget/g_multi/functions/acm.usb0
|       |-- bmAttributes
|       |-- ncm.usb0 -> ../../../../usb_gadget/g_multi/functions/ncm.usb0
|       `-- strings
|           `-- 0x409
|               `-- configuration
|-- functions
|   |-- acm.usb0
|   |   |-- port_num
|   |   `-- protocol
|   `-- ncm.usb0
|       |-- dev_addr
|       |-- host_addr
|       |-- ifname
|       |-- max_segment_size
|       |-- os_desc
|       |   `-- interface.ncm
|       |       |-- compatible_id
|       |       `-- sub_compatible_id
|       `-- qmult
|-- idProduct
|-- idVendor
|-- max_speed
|-- os_desc
|   |-- b_vendor_code
|   |-- qw_sign
|   `-- use
|-- strings
|   `-- 0x409
|       |-- manufacturer
|       |-- product
|       `-- serialnumber
`-- webusb
    |-- bVendorCode
    |-- bcdVersion
    |-- landingPage
    `-- use
5

I think your setup still uses configfs no? That I am trying to avoid - I want to use a static driver. Also I see it is using g_ncm which I have not tried before so I gave it a try but got similar results as g_ether and g_mutli.

Every single one of works but only when loglevel=8 and initcall_debug are turned on. All of them fails when those params are not present. Weird behaviour, I don’t know if I am doing something wrong here tbh.

6

Ah - ok so not using libcomposite, but just using the drive directly so it brings up the usb with only a single device on it. I don’t remember anyone doing that for probably almost a decade. I’ll have to refresh my brain a bit, and see if I recall any of it …

… All I can think of is attach a serial interface to the PB2 (the Rpi Debug Lead), then log on over serial, and examine the various things that g_ether and g_ncm are meant to bring up.

7

Yes that is what I have been doing. I attached various kernel logs I captured via serial in my first post. I am new to kernel debugging so was not sure how to go about this. But with little help I think I got a solution.

Basically what I understood was the nodes f900000.dwc3-usb,f910000.dwc3-usb (parents of the usb controllers) were being deffered by power-controller because it was not getting initialized early enough.

[   34.058034] platform f900000.dwc3-usb: error -EPROBE_DEFER: wait for supplier /bus@f0000/system-controller@44043000/power-controller
[   34.076824] platform f900000.dwc3-usb: Added to deferred list
[   34.093409] platform f910000.dwc3-usb: error -EPROBE_DEFER: wait for supplier /bus@f0000/system-controller@44043000/power-controller
[   34.112192] platform f910000.dwc3-usb: Added to deferred list

Enabling debug logs with loglevel=8 initcall_debug mechanically delayed the boot and power-controller got enough time to initialize and thus the usb controllers were probed within time and we got usb networking working.

Same effect happened when I almost gave up on the idea and moved to initramfs + configfs based solution - I noticed that even with a bare init script having only a infinite while loop was making it work. So I tested with smaller usleep durations. The tests revealed that a delay between 0.125 seconds and 0.19 seconds is necessary for the power controller to initialize and the usb controller dependency to resolve correctly. For usleep(125000) it was failing but it was working for usleep(190000).

0.125s:

[    1.599897] g_ncm gadget.0: HOST MAC 62:73:01:1c:aa:76
[    1.605094] g_ncm gadget.0: MAC ca:48:19:6c:39:bc
[    1.609847] g_ncm gadget.0: NCM Gadget
[    1.613596] g_ncm gadget.0: g_ncm ready
[    1.628151] xhci-hcd xhci-hcd.3.auto: xHCI Host Controller
[    1.633711] xhci-hcd xhci-hcd.3.auto: new USB bus registered, assigned bus number 1
[    1.641580] xhci-hcd xhci-hcd.3.auto: USB3 root hub has no ports
[    1.647602] xhci-hcd xhci-hcd.3.auto: hcc params 0x0258fe6d hci version 0x110 quirks 0x0000808020000010
[    1.657066] xhci-hcd xhci-hcd.3.auto: irq 290, io mem 0x31100000
[    1.663929] hub 1-0:1.0: USB hub found
[    1.667727] hub 1-0:1.0: 1 port detected
[    1.674522] cpufreq: cpufreq_policy_online: CPU0: Running at unlisted initial frequency: 1400000 kHz, changing to: 1250000 kHz
[    1.709274] leds-gpio leds: Led green:indicator renamed to green:indicator_1 due to name collision
[    1.723273] clk: Disabling unused clocks
[    1.726059] sdhci-am654 fa00000.mmc: Got CD GPIO
[    1.740261] PM: genpd: Disabling unused power domains
[    1.740769] mmc1: CQHCI version 5.10
[    1.790820] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
[    1.800901] Freeing unused kernel memory: 3264K
[    1.805605] Run /init as init process
[    1.854192] mmc1: new UHS-I speed SDR104 SDHC card at address 59b4
[    1.861971] mmcblk1: mmc1:59b4 USD00 29.5 GiB
[    1.870016]  mmcblk1: p1 p2
[    1.936173] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000

Corresponding host side:

[38410.307328] usb 1-1: new high-speed USB device number 64 using xhci_hcd
[38410.434252] usb 1-1: New USB device found, idVendor=0525, idProduct=a4a1, bcdDevice= 6.18
[38410.434314] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[38410.434319] usb 1-1: Product: NCM Gadget
[38410.434323] usb 1-1: Manufacturer: Linux 6.18.37 with dwc3-gadget
[38415.992698] usb 1-1: can't set config #1, error -110

0.19s:

[    1.602638] g_ncm gadget.0: HOST MAC ea:d0:93:c9:fd:c8
[    1.607846] g_ncm gadget.0: MAC 26:a4:85:5e:24:f3
[    1.612609] g_ncm gadget.0: NCM Gadget
[    1.616360] g_ncm gadget.0: g_ncm ready
[    1.630860] xhci-hcd xhci-hcd.3.auto: xHCI Host Controller
[    1.636401] xhci-hcd xhci-hcd.3.auto: new USB bus registered, assigned bus number 1
[    1.644205] xhci-hcd xhci-hcd.3.auto: USB3 root hub has no ports
[    1.650216] xhci-hcd xhci-hcd.3.auto: hcc params 0x0258fe6d hci version 0x110 quirks 0x0000808020000010
[    1.659654] xhci-hcd xhci-hcd.3.auto: irq 290, io mem 0x31100000
[    1.666556] hub 1-0:1.0: USB hub found
[    1.670358] hub 1-0:1.0: 1 port detected
[    1.677255] cpufreq: cpufreq_policy_online: CPU0: Running at unlisted initial frequency: 1400000 kHz, changing to: 1250000 kHz
[    1.712032] leds-gpio leds: Led green:indicator renamed to green:indicator_1 due to name collision
[    1.726065] clk: Disabling unused clocks
[    1.728864] sdhci-am654 fa00000.mmc: Got CD GPIO
[    1.738645] PM: genpd: Disabling unused power domains
[    1.739149] mmc1: CQHCI version 5.10
[    1.789156] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
[    1.799189] Freeing unused kernel memory: 3264K
[    1.803917] Run /init as init process
[    1.851810] mmc1: new UHS-I speed SDR104 SDHC card at address 59b4
[    1.859760] mmcblk1: mmc1:59b4 USD00 29.5 GiB
[    1.867175]  mmcblk1: p1 p2
[    1.999303] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000

Corresponding host:

[37143.984499] usb 1-1: new high-speed USB device number 59 using xhci_hcd
[37144.111564] usb 1-1: New USB device found, idVendor=0525, idProduct=a4a1, bcdDevice= 6.18
[37144.111580] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[37144.111585] usb 1-1: Product: NCM Gadget
[37144.111589] usb 1-1: Manufacturer: Linux 6.18.37 with dwc3-gadget
[37144.176246] cdc_ncm 1-1:1.0: MAC-Address: ea:d0:93:c9:fd:c8
[37144.177001] cdc_ncm 1-1:1.0 usb0: register 'cdc_ncm' at usb-0000:03:00.3-1, CDC NCM (NO ZLP), ea:d0:93:c9:fd:c8
[37144.219084] cdc_ncm 1-1:1.0 enp3s0f3u1: renamed from usb0

So, I settled with adding a rootdelay=1 in the kernel params with no initramfs and no configfs. I seem to get an interface, now I will try getting a NFS root and I should be done with this.

16: enp3s0f3u1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether 4e:3e:e8:c5:35:22 brd ff:ff:ff:ff:ff:ff
    altname enx4e3ee8c53522
8

Bizarre, but yes you seem to have got to the source of the problem.

The question is how to stop that part of the kernel starting until after the power-controller has been configured, and that has me scratching my head. It a dependence that may not exist on all computers with that hardware - so its specific that on beagles the power controller brings up part of the USB. As its dependant on how the board has been wired, it can only live in the device tree. But I’m at a loss of what to change in the device tree - I think it means the usb specification in the device tree that has the power controller as an input, and so is needed to bring up the USB.

@RobertCNelson - you own the device trees, have you got any idea here?

OK - have got hold of the PB2 schematics. There were finalised in October 2022! So the board has been coming a long time. Anyway, I wondered if the USB goes via an external chip, needed to fit in with the usb specs. That though isn’t the case, the usb is wired direct to the AM6232 - so the connection should come up as soon as the CPU is powered. So this is confusing me - what power supply is needed for this.