[BB-X15] Dual Ethernet Speed Errata; Dual Ethernet Performance

Hi,

I think, I already asked this question, but can’t find it anymore…
There is an errata of the AM572x CPU saying that RGMII2 can only be clocked as fast that allows Fast-Ethernet (100BASE-T), no Gigabit Ethernet… Today I made some tests: Both Ethernet Ports of the BB-X15 are connected to a switch that supports Gigabit Ethernet. Auto negotiation selects 1000baseT-FD for both links. A test with iperf3 gives me ~940Mbps on both ports (sequentially). If I do tests on both links, things look very differently. I started the second test after ~5s and terminated it after ~10s. Performance on two links is far LESS than the performance on a single link. (The iperf3 server is connected to the network via 2*10Gbps and IS able to fill the link)

[me@test-vm ~]$ iperf3 -c 10.20.0.121 -t 600
Connecting to host 10.20.0.121, port 5201
[ 4] local 10.20.0.121 port 54362 connected to 46.234.46.30 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 115 MBytes 966 Mbits/sec 1054 311 KBytes
[ 4] 1.00-2.00 sec 111 MBytes 933 Mbits/sec 270 317 KBytes
[ 4] 2.00-3.00 sec 111 MBytes 933 Mbits/sec 286 314 KBytes
[ 4] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 156 266 KBytes
[ 4] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 495 KBytes
[ 4] 5.00-6.00 sec 62.5 MBytes 524 Mbits/sec 182 21.2 KBytes
[ 4] 6.00-7.00 sec 35.0 MBytes 293 Mbits/sec 190 21.2 KBytes
[ 4] 7.00-8.00 sec 37.5 MBytes 315 Mbits/sec 143 12.7 KBytes
[ 4] 8.00-9.00 sec 40.0 MBytes 336 Mbits/sec 210 26.9 KBytes
[ 4] 9.00-10.00 sec 38.8 MBytes 325 Mbits/sec 199 12.7 KBytes
[ 4] 10.00-11.00 sec 72.5 MBytes 608 Mbits/sec 120 236 KBytes
[ 4] 11.00-12.00 sec 111 MBytes 933 Mbits/sec 228 243 KBytes
[ 4] 12.00-13.00 sec 111 MBytes 933 Mbits/sec 312 276 KBytes
[ 4] 13.00-14.00 sec 112 MBytes 944 Mbits/sec 231 328 KBytes
[ 4] 14.00-15.00 sec 111 MBytes 933 Mbits/sec 309 341 KBytes
[ 4] 15.00-16.00 sec 111 MBytes 933 Mbits/sec 232 272 KBytes
[ 4] 16.00-17.00 sec 109 MBytes 912 Mbits/sec 334 165 KBytes
^C[ 4] 17.00-17.25 sec 22.5 MBytes 756 Mbits/sec 0 250 KBytes


[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-17.25 sec 1.50 GBytes 747 Mbits/sec 4456 sender
[ 4] 0.00-17.25 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated

The same results if I run two iperfs in different directions on the links. (iperf and iperf -R). Probably the connection of the internal switch to the A15-cores is the bottleneck.

Things look worse if I test UDP:

800Mbps from BB-X15 → BigHost

[me@test-vm ~]$ iperf3 -c 10.20.0.121 -t 10 -u -b 800M -R
Connecting to host 10.20.0.121, port 5201
Reverse mode, remote host 10.20.0.121 is sending
[ 4] local 10.20.0.121 port 40556 connected to 46.234.46.31 port 5201
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-1.00 sec 97.1 MBytes 814 Mbits/sec 0.059 ms 0/12425 (0%)
[ 4] 1.00-2.00 sec 96.8 MBytes 812 Mbits/sec 0.058 ms 0/12388 (0%)
[ 4] 2.00-3.00 sec 95.6 MBytes 802 Mbits/sec 0.062 ms 0/12234 (0%)
[ 4] 3.00-4.00 sec 92.6 MBytes 777 Mbits/sec 0.053 ms 0/11849 (0%)
[ 4] 4.00-5.00 sec 92.7 MBytes 778 Mbits/sec 0.059 ms 0/11869 (0%)
[ 4] 5.00-6.00 sec 97.9 MBytes 821 Mbits/sec 0.059 ms 0/12529 (0%)
[ 4] 6.00-7.00 sec 92.5 MBytes 776 Mbits/sec 0.053 ms 0/11839 (0%)
[ 4] 7.00-8.00 sec 96.7 MBytes 811 Mbits/sec 0.076 ms 0/12376 (0%)
[ 4] 8.00-9.00 sec 95.1 MBytes 798 Mbits/sec 0.056 ms 0/12173 (0%)
[ 4] 9.00-10.00 sec 98.3 MBytes 825 Mbits/sec 0.061 ms 0/12584 (0%)
[ 4] 10.00-11.00 sec 89.4 MBytes 750 Mbits/sec 0.088 ms 0/11447 (0%)
[ 4] 11.00-12.00 sec 56.8 MBytes 476 Mbits/sec 0.053 ms 0/7264 (0%)
[ 4] 12.00-13.00 sec 56.8 MBytes 476 Mbits/sec 0.049 ms 0/7269 (0%)
[ 4] 13.00-14.00 sec 56.1 MBytes 471 Mbits/sec 0.042 ms 0/7187 (0%)
[ 4] 14.00-15.00 sec 56.6 MBytes 474 Mbits/sec 0.060 ms 0/7239 (0%)
[ 4] 15.00-16.00 sec 54.4 MBytes 457 Mbits/sec 0.037 ms 0/6968 (0%)
[ 4] 16.00-17.00 sec 56.1 MBytes 471 Mbits/sec 0.061 ms 0/7183 (0%)
[ 4] 17.00-18.00 sec 56.7 MBytes 476 Mbits/sec 0.061 ms 0/7257 (0%)
[ 4] 18.00-19.00 sec 56.6 MBytes 475 Mbits/sec 0.050 ms 0/7242 (0%)
[ 4] 19.00-20.00 sec 55.8 MBytes 468 Mbits/sec 0.078 ms 0/7140 (0%)

(Results are similar on both links)
After ~10s I started the same test on the other link. Packet throughput was reduced by less than 50%. If I send 800Mbps on one link and receive 800Mbps on the other link, things work as expected (1.6Gbps total (rx+tx).
One very strange result is testing UDP reception on the BB-X15. I always get very high packet loss:

Hello,
You seems quite experienced with Ethernet. I would say you got the expertise.
I am interested by this board because it has two Ethernet ports. The idea is to have the first Ethernet port connected to the source (server) and the second one for cascading/daisy-chain…
Could you please tell me if it possible?
Thanks in advance.
Cheers,
Christophe