Additionally, the output had newline "\n" every time output was sent.
I am confused as to what your actual hardware configuration is, with respect to the “serial communications.”
The actual hardware used in the final project with the i2c connectors soldered on to a prototyping cape looks like this:
I have repeated the exact same fault on this:
Are you using a COM port in Windows to talk to a USB to serial cable, which is talking to a hardware serial port in the BBB?
I plugged the micro USB port on the BBB into my USB hub. The USB hub is currently attached to a windows PC but the same fault happens when the USB cable is plugged into my mac book running OSX.
Within a minute of the BBB being plugged in an extra COM port is added in windows. I connect to the com port with Termite (terminal program), ZOC7 (terminal program) or pyserial (the python serial library).
When I connect to the COM port I can login and get a shell. I ran some relevant commands in the shell so you can see the results.
root@beaglebone:/dev# w
20:57:21 up 6:52, 1 user, load average: 0.02, 0.06, 0.05
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root ttyGS0 14Nov15 1.00s 0.97s 0.02s w
root@beaglebone:/dev# who
root ttyGS0 2015-11-14 00:15
root@beaglebone:/dev# tty
/dev/ttyGS0
root@beaglebone:/dev# whoami
root
root@beaglebone:/dev# ls
alarm log_system ram11 tty16 tty44 ubi_ctrl
ashmem loop0 ram12 tty17 tty45 uinput
audio loop1 ram13 tty18 tty46 urandom
autofs loop2 ram14 tty19 tty47 usbmon0
binder loop3 ram15 tty2 tty48 usbmon1
block loop4 ram2 tty20 tty49 usbmon2
btrfs-control loop5 ram3 tty21 tty5 vcs
bus loop6 ram4 tty22 tty50 vcs1
char loop7 ram5 tty23 tty51 vcs2
console loop-control ram6 tty24 tty52 vcs3
core mapper ram7 tty25 tty53 vcs4
cpu_dma_latency mem ram8 tty26 tty54 vcs5
disk mixer ram9 tty27 tty55 vcs6
dri mmcblk0 random tty28 tty56 vcs7
dsp mmcblk0boot0 root tty29 tty57 vcsa
fb0 mmcblk0boot1 rtc0 tty3 tty58 vcsa1
fd mmcblk0p1 shm tty30 tty59 vcsa2
full mmcblk0p2 snd tty31 tty6 vcsa3
fuse mqueue sndstat tty32 tty60 vcsa4
hwrng net stderr tty33 tty61 vcsa5
i2c-0 network_latency stdin tty34 tty62 vcsa6
i2c-1 network_throughput stdout tty35 tty63 vcsa7
initctl null tty tty36 tty7 watchdog
input ppp tty0 tty37 tty8 watchdog0
kmem psaux tty1 tty38 tty9 xconsole
kmsg ptmx tty10 tty39 ttyGS0 zero
log ptp0 tty11 tty4 ttyO0
log_events pts tty12 tty40 ttyS0
logibone_mem ram0 tty13 tty41 ttyS1
log_main ram1 tty14 tty42 ttyS2
log_radio ram10 tty15 tty43 ttyS3
So, you’ve actually delved further into using the UARTs on this platform than I ever have.
I have also come to suspect that there are no hardware UARTS involved. I believe that the serial line is being emulated.
Anyway, my first instinct wants to say baud rate is involved somehow.
On the PC side I can set the baudrate to any number. The data rate and the error rate appear to be unaffected. For example:
x = serial.Serial(‘COM10’, baudrate=10)
x = serial.Serial(‘COM10’, baudrate=9600)
x = serial.Serial(‘COM10’, baudrate=100_000_000_000)
I have also set the baud rate with Termite 3.1 to demonstrate that it is not a bug in the python serial library.
On the BBB end the speed is always reported as 9600. Some commands to change the speed succeed and others fail but the reported speed is unchanged.
root@beaglebone:/dev# stty ispeed 7200
root@beaglebone:/dev# stty
speed 9600 baud; line = 0;
root@beaglebone:/dev# stty ispeed 14400
root@beaglebone:/dev# stty
speed 9600 baud; line = 0;
root@beaglebone:/dev# stty ispeed 2400
stty: standard input: unable to perform all requested operations
root@beaglebone:/dev# stty
speed 9600 baud; line = 0;
I will finish replying to everyone tomorrow. It is getting late where I am.
On Thu, 4 May 2017 15:10:07 -0700, William Hermans
<yyrkoon@gmail.com> declaimed the following:
I think it's important to note that the OP is not sending data from the
beaglebone, but receiving it. At least looking at the last code he pasted.
Usually you wont see "COM9" on a Linux system
The OP is running a program on Windows that is sending a command string
TO the BBB, and that command at the shell generates output (on stdout,
connected to the Windows COM port).
I am confused as to what your actual hardware configuration is, with respect to the “serial communications.”
The actual hardware used in the final project with the i2c connectors soldered on to a prototyping cape looks like this:
I have repeated the exact same fault on this hardware:
Are you using a COM port in Windows to talk to a USB to serial cable, which is talking to a hardware serial port in the BBB?
I plugged the micro USB port on the BBB into my USB hub. The USB hub is currently attached to a windows PC but the same fault happens when the USB cable is plugged into my mac book running OSX.
Within a minute of the BBB being plugged in an extra COM port is added in windows. I connect to the com port with Termite (terminal program), ZOC7 (terminal program) or pyserial (the python serial library).
When I connect to the COM port I can login and get a shell. I ran some relevant commands in the shell so you can see the results.
root@beaglebone:/dev# w
20:57:21 up 6:52, 1 user, load average: 0.02, 0.06, 0.05
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root ttyGS0 14Nov15 1.00s 0.97s 0.02s w
root@beaglebone:/dev# who
root ttyGS0 2015-11-14 00:15
root@beaglebone:/dev# tty
/dev/ttyGS0
root@beaglebone:/dev# whoami
root
root@beaglebone:/dev# ls
alarm log_system ram11 tty16 tty44 ubi_ctrl
ashmem loop0 ram12 tty17 tty45 uinput
audio loop1 ram13 tty18 tty46 urandom
autofs loop2 ram14 tty19 tty47 usbmon0
binder loop3 ram15 tty2 tty48 usbmon1
block loop4 ram2 tty20 tty49 usbmon2
btrfs-control loop5 ram3 tty21 tty5 vcs
bus loop6 ram4 tty22 tty50 vcs1
char loop7 ram5 tty23 tty51 vcs2
console loop-control ram6 tty24 tty52 vcs3
core mapper ram7 tty25 tty53 vcs4
cpu_dma_latency mem ram8 tty26 tty54 vcs5
disk mixer ram9 tty27 tty55 vcs6
dri mmcblk0 random tty28 tty56 vcs7
dsp mmcblk0boot0 root tty29 tty57 vcsa
fb0 mmcblk0boot1 rtc0 tty3 tty58 vcsa1
fd mmcblk0p1 shm tty30 tty59 vcsa2
full mmcblk0p2 snd tty31 tty6 vcsa3
fuse mqueue sndstat tty32 tty60 vcsa4
hwrng net stderr tty33 tty61 vcsa5
i2c-0 network_latency stdin tty34 tty62 vcsa6
i2c-1 network_throughput stdout tty35 tty63 vcsa7
initctl null tty tty36 tty7 watchdog
input ppp tty0 tty37 tty8 watchdog0
kmem psaux tty1 tty38 tty9 xconsole
kmsg ptmx tty10 tty39 ttyGS0 zero
log ptp0 tty11 tty4 ttyO0
log_events pts tty12 tty40 ttyS0
logibone_mem ram0 tty13 tty41 ttyS1
log_main ram1 tty14 tty42 ttyS2
log_radio ram10 tty15 tty43 ttyS3
So, you’ve actually delved further into using the UARTs on this platform than I ever have.
I have also come to suspect that there are no hardware UARTS involved. I believe that the serial line is being emulated.
Anyway, my first instinct wants to say baud rate is involved somehow.
On the PC side I can set the baudrate to any number. The data rate and the error rate appear to be unaffected. For example:
x = serial.Serial(‘COM10’, baudrate=10)
x = serial.Serial(‘COM10’, baudrate=9600)
x = serial.Serial(‘COM10’, baudrate=100_000_000_000)
I have also set the baud rate with Termite 3.1 to demonstrate that it is not a bug in the python serial library.
On the BBB end the speed is always reported as 9600. Some commands to change the speed succeed and others fail but the reported speed is unchanged.
root@beaglebone:/dev# stty ispeed 7200
root@beaglebone:/dev# stty
speed 9600 baud; line = 0;
root@beaglebone:/dev# stty ispeed 14400
root@beaglebone:/dev# stty
speed 9600 baud; line = 0;
root@beaglebone:/dev# stty ispeed 2400
stty: standard input: unable to perform all requested operations
root@beaglebone:/dev# stty
speed 9600 baud; line = 0;
I will finish replying to everyone tomorrow. It is getting late where I am.
So, other than repeating exactly what I said, what's your point ? Also
newline eliminates the need for flush.
On Thu, 4 May 2017 16:12:09 -0700, William Hermans
<yyrkoon@gmail.com> declaimed the following:
On Thu, 4 May 2017 15:39:43 -0700 (PDT), David Howlett
<david.howlett3@gmail.com> declaimed the
following:
I plugged the micro USB port on the BBB into my USB hub. The USB hub is
currently attached to a windows PC but the same fault happens when the USB
cable is plugged into my mac book running OSX.
Which, since there is no FTDI chip on the BBB, is effectively an
emulated serial port... The USB-serial driver responds to I/O operations as
if it were a serial port, but things like baud rate are mostly meaningless
-- the driver moves bytes from the client buffer to the USB buffer, and USB
transfers at (if USB2.0 full speed -- 480kbps; ignoring the breaks caused
by USB packet size AND that the host computer has to poll the USB device to
ask for each packet of data; effective throughput being closer to 250kbps).
And, the Windows (or Mac) side is also an emulated serial, but as the
host side, the USB driver controls the polling of the other end asking for
data or to send packets.
I have also come to suspect that there are no hardware UARTS involved. I
believe that the serial line is being emulated.
Unless you use wires on the UART pins of the BBB, likely true. Those
may be hardware UARTs, but as soon as something is connected that converts
to USB, the actual data transfer is USB and the other end emulated.
On the PC side I can set the baudrate to any number. The data rate and the
error rate appear to be unaffected. For example:x = serial.Serial('COM10', baudrate=10)
x = serial.Serial('COM10', baudrate=9600)
x = serial.Serial('COM10', baudrate=100_000_000_000)
As I suspect -- the baud rate is irrelevant when you get into the USB
protocol. At most, it may control a "filter" as to how rapidly the drivers
relay stuff to the USB protocol. That is -- at slow rates there may only be
a few bytes of data per USB packet, whereas at really high rates each USB
packet will contain more data (and hence be more efficient when taking the
polling overhead into account).
So, the COM port on Windows is talking to a virtual serial port in the BBB USB widget.
There is no hardware UART involved, and no hardware clocks anywhere to throttle or pace anything.
Then, you push this for transfer rate, while running it under a non realtime OS.
I guess I am not surprised that you are having problems.
The USB widget is nice for doing simple things, with fast, simple set up for beginners.
I suspect you are trying to run it in an application where it has not been well tested.
For a stable system, I would recommend considering doing the data transfer over a real hardware serial port, or if that is not fast enough, do the transfer using Ethernet. (And I don’t mean Ethernet over USB using the Widget.)
I find the true hardware serial ports, and the genuine FTDI serial to USB converters to be stable well above 100,000 bits per second. Some report success in the megabits per second range, I have just not had to personally go there, yet.
You never have been specific about how many bits per second of data that you have to move.
I think that is an important place to start.
— Graham
David,
As an example if I run the following program there is no data loss:
import serial.tools.list_ports
connect to the BBB serial port on Windows
x = serial.Serial(‘COM9’)
x.write(b’yes “123456”\n’)while True:
received = x.readline()
if received != b’123456\r\n’:
print(received)But there is data loss if I run this:
import serial.tools.list_ports
connect to the BBB serial port on Windows
x = serial.Serial(‘COM9’)
x.write(b’yes “1234567”\n’)
while True:
received = x.readline()
if received != b’1234567\r\n’:
print(received)
This seems to me that you are indeed overflowing your buffer. How or why I’m not sure, but if sending 6 characters all is fine, and 7 characters does not transmit properly then obviously something is wrong. I’d look into googling the driver name this device uses against any known reliability issues in Linux. One odd thing I did notice about your output you’ve shown us, is that everything in your received = x.readline() loop seems to be working fine.But all the failures are related to this, or a similar line you’ve not shown us yet: x.write(b’yes “1234567”\n’)
I’m suspecting you’re not showing us all your code however. Because the output you’ve shown us does not necessarily “jibe” with the code I’m seeing here.
So, the COM port on Windows is talking to a virtual serial port in the BBB
USB widget.
There is no hardware UART involved, and no hardware clocks anywhere to
throttle or pace anything.
Then, you push this for transfer rate, while running it under a non
realtime OS.
I guess I am not surprised that you are having problems.
This is kind of what I'm thinking, but just sayin' Graham, "gadget" not
"widgit" 
But yeah I do not know this hardware at all, but the conversion
from USB to RS485, there can not be any flow control there. So the gadget
side of the hardware has no flow control, where the RS485 side *may*. I'd
actually speed up the transmission rate, and perhaps put in a usleep()
between each send to fine tune things, and call it quits.
The USB widget is nice for doing simple things, with fast, simple set up
for beginners.
I suspect you are trying to run it in an application where it has not been
well tested.For a stable system, I would recommend considering doing the data transfer
over a real hardware serial port, or if that is not fast enough, do the
transfer using Ethernet. (And I don't mean Ethernet over USB using the
Widget.)I find the true hardware serial ports, and the genuine FTDI serial to USB
converters to be stable well above 100,000 bits per second. Some report
success in the megabits per second range, I have just not had to personally
go there, yet.You never have been specific about how many bits per second of data that
you have to move.
I think that is an important place to start.
So *maybe* if UART4 on the beaglebone was used in full RS485 mode, talking
to the USB to RS485 converter connected to the Windows machine through USB.
*Maybe* that could, or would help solve these issues ? Just a guess but hey
probably worth looking into.
Hi,
I plugged the micro USB port on the BBB into my USB hub. The USB hub is currently attached to a windows PC but the same fault happens when the USB cable is plugged into my mac book running OSX.Which, since there is no FTDI chip on the BBB, is effectively an emulated serial port... The USB-serial driver responds to I/O operations as if it were a serial port, but things like baud rate are mostly meaningless -- the driver moves bytes from the client buffer to the USB buffer, and USB transfers at (if USB2.0 full speed -- 480kbps; ignoring the breaks caused by USB packet size AND that the host computer has to poll the USB device to ask for each packet of data; effective throughput being closer to 250kbps).
I haven’t noticed that you are using USB serial gadged driver.What Dennis wrote is true.
Here you can see the source code of USB serial gadged driver:
Presented baudrate for tty is fixed to 9600:
/* 9600-8-N-1 ... matches defaults expected by "usbser.sys" on
* MS-Windows. Otherwise, most of these flags shouldn't affect
* anything unless we were to actually hook up to a serial line.
*/
gs_tty_driver->init_termios.c_cflag = B9600 | CS8 | CREAD | HUPCL | CLOCAL;
gs_tty_driver->init_termios.c_ispeed = 9600;
gs_tty_driver->init_termios.c_ospeed = 9600;
tty_set_operations(gs_tty_driver, &gs_tty_ops);
Change of baud rate is not handled anywhere, data are transferred at full USB speed.
The easiest solution is to do a simple handshake:
On PC:
import serial.tools.list_ports
# connect to the BBB serial port on Windows
x = serial.Serial('COM9')
while True:
x.write(b"\r\n")
x.flush()
received = x.readline()
print(received)
On BBB:
import serial.tools.list_ports
x = serial.Serial('/dev/ttyGS0')
while True:
received = x.readline()
x.write(b"DATA\r\n")
x.flush()
The BBB app must be run as first… I do not know much about python, maybe it’s ok 
The Second solution with baudrate emulation without RS232/USB converter:
On BBB enable ttyO1 and ttyO2. Connect Tx pins with Rx…
Install ser2net and configure it on port ttyO2.
Run your app on ttyO1
On windows install and connect it to ser2net port.
And you have “simple” baud rate emulator
BTW:
Why you want to use RS232, i would choose TCP ,UDP or WebServer…
if you want, can I send you, ready to use modular web server in java… jetty+vaadin+netbeans lookup
ArSi
Sorry for double posting last time, this is my first time using google groups.
I wonder if you run the application that failing for you there is you pipe that data to a file,
instead of to stdout, if you would be experiencing the same problem ?
The data loss problem also happens when the yes command is just repeating a string over and over to stdout so I don’t think the problem is python related.
Why don’t you humor me, and give that a try ?
I tried:
$ python3.6 ~/reusable/BBB_pressure_streamer.py > some_log_file.txt
When I view the log file I can’t see any errors
Have you tried reducing the receive FIFO level?
I had not thought of that. Today I tried setting the Tx and Rx buffers to 1, 8 and 14 bytes. Data continued to be lost on all settings.
Suspect I’d start with
stty ixon ixoff -ixany
and ensure the other end is set for equivalent.
I tried:
root@beaglebone:~# stty --file=/dev/ttyGS0 ixon ixoff -ixany
root@beaglebone:~# stty --all
speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^; erase = ^?; kill = ^U; eof = ^D; eol = ;
eol2 = ; swtch = ; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon ixoff
-iuclc -ixany imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt
echoctl echoke
and I also turned on flow control on the python side at the same time but I still got data loss.
Recommend you specify /which/ serial port
I was originally running stty while connected to the virtual serial port but I forgot to mention that detail.
I can’t duplicate your results as
debian@beaglebone:~$ python3.6 -c “while True: print(sum(i for i in range(10000)))”
I previously installed python3.6 on all my boards. I should have given an example that only depended on base Debian.
#flush input buffer (where did .read_all() come from?)
It is from pyserial/serial/serialutil.py at master · pyserial/pyserial · GitHub
I was using it to clear the serial buffer of all data. Calling .reset_input_buffer() is better
OKAY – THIS PRODUCED a drop-out 2574…2659
Thanks, it is nice to know that someone else can replicate the issue, it is not just me doing something stupid.
For a stable system, I would recommend considering doing the data transfer over a real hardware serial port
Today I purchased two connectors for the hardware serial pins from:
I will see if that helps.
You never have been specific about how many bits per second of data that you have to move.
Today I measured the average data rate of my application to be 69,218 bytes/second.
This seems to me that you are indeed overflowing your buffer.
How or why I’m not sure, but if sending 6 characters all is fine,
and 7 characters does not transmit properly then obviously something is wrong.
When I send:
x.write(b’yes “123456”\n’)
I receive 8 bytes per line not 6 due to the addition of /r/n. I don’t think this is terribly important though
I’m suspecting you’re not showing us all your code
My working repository is here:
A permanent link to the relevant commit is here:
There is not much to see in the repository but you are welcome to have a look.
The only thing worth noting is I use python 3.6 on the PC side too.
Here you can see the source code of USB serial gadged driver
Thankyou
The easiest solution is to do a simple handshake
That is a good idea for avoiding data loss, the troublesome buffer would never fill so there would be no data loss.
Why you want to use RS232, i would choose TCP ,UDP or WebServer
Personal preexisting skill set. I have been using serial connections for 4 years on multiple projects with multiple pieces of hardware. I wrongly believed that it would be easy to just use serial again.
In the future I plan to use the hardware serial connection.
$ python3.6 ~/reusable/BBB_pressure_streamer.py > some_log_file.txt
When I view the log file I can’t see any errors
So, actually, scripting languages in general can be far slower, that their native code counterparts. Firstly, because there is an interpreter, and secondly because of code bloat, through using libraries. Almost as a rule, larger executables will lead to poorer performing code. Python in particular is one of if not the slowest languages also when bench marked in some tests. And I do not mean by a second or two. I mean by a huge margin in seconds, for specific tests. Just as an example though, in a few tests I remember reading, Compared to Nodejs, Python was behind by a minute or more.
My point here is not to say Nodejs is better or not though. Nodejs is pretty terrible it’s self. This is meant demonstrate in general that scripting languages can be terrible when your app is I/O intensive. The reason why I suggested piping the output of your script to a file was to see if there was potentially any “normal” I/O bottlenecks involved. All languages can suffer from this however, because printing to screen( stdout ) is, or can be very I/O intensive. In C, you’re maybe going to max out around a couple hundred “samples” to stdout a second, before your application will noticeably slow down. With Nodejs, it’s actually far less. I do not know if Python is slower than Nodejs in this case. It’s been a while since I’ve read the benchmarks, but actually the benchmarks were not testing this capability anyhow. Maybe they should have been ?
All in all, I’m still not convinced this is the problem in whole. But I do feel that it could be a contributing factor. It still would not explain why you seem to be getting your full amount of transmissions. Just that some of those transmissions are “empty”. Especially when those empty transmissions are not consistent in timing / frequency. I would suggest that you do keep this in mind, and perhaps at least consider testing output to a file further. Instead of testing while output is to stdout. At minimum, this will allow you to rule out, output to stdout as a contributing factor in your code. With very little wasted time on your behalf.
Why you want to use RS232, i would choose TCP ,UDP or WebServer
Personal preexisting skill set. I have been using serial connections for 4 years on multiple projects with multiple pieces of hardware. I wrongly believed that it would be easy to just use serial again.
And I will finally learn a python
Test this: (on linux telnet ip_of_python_host 5000 on windows with putty telnet to port 5000 )
import select
import socket
from time import sleep
#telnet escape sequences
CLEAR_SCREEN = chr(27) + "[2J"
MOVE_TO_LINE_1 = chr(27) + "[1;1H"
MOVE_TO_LINE_2 = chr(27) + "[2;1H"
MOVE_TO_LINE_3 = chr(27) + "[3;1H"
MOVE_TO_LINE_4 = chr(27) + "[4;1H"
MOVE_TO_LINE_5 = chr(27) + "[5;1H"
MOVE_TO_LINE_6_ST = chr(27) + "[6;1H"
MOVE_TO_LINE_6 = chr(27) + "[6;15H"
CLEAR_LINE = chr(27) + "[2K"
CLEAR_LINE_TO_END = chr(27) + "[K"
SAVE_CURSOR = chr(27) + "7"
RESTORE_CURSOR = chr(27) + "8"
TXT_CMD = "Enter command:"On Sat, 6 May 2017 08:25:31 -0700 (PDT), David Howlett
<david.howlett3@gmail.com> declaimed the
following:
Thanks, it is nice to know that someone else can replicate the issue, it is
not just me doing something stupid.
Note though that I managed to get 14kB before seeing a drop-out
(caveat: I was scrolling the captured output data looking for mis-aligned
line lengths, so a drop out exactly matching a sequence of whole numbers
and the following space would not have been seen in the wrapped window)
Today I measured the average data rate of my application to be 69,218
bytes/second.
If that's a sustained rate, and not burst, it is likely extreme for
serial protocols.
On 8n1 (10-bits per byte) set up, that give an effective rate of 692180
bps, or 692kbps. That's probably pushing the capabilities of the BBB to
buffer and send via an emulation of a serial port, and even most hardware
UARTs may not manage that fast a rate -- it's 6 times the 115.2kbps of most
serial ports.
In the future I plan to use the hardware serial connection.
Recall that the debug serial header does NOT have hardware flow
control. The others, once pinmux is configured for UART, do support
RTS/CTS.
For contrast, Ethernet (using RJ54, not the USB gadget) starts at
10Mbps, so even with packet overhead should be capable. Though if you are
outputting just (say) 10 bytes at a time, the overhead could be significant
fraction <G> (~38 bytes worth [12 being a required gap between frames] at
the Ethernet level, and 16 bytes worth for IP packet header [no options
provided] and 16 for a TCP header [again, no options used] -- ~70 bytes of
overhead per sent frame).
Do you NEED every sent string, or are you okay with losses, as long as
it is not a drop-out within a data value?
TCP protocol ensures that data is received completely; if a packet is
lost/corrupted, the protocol should request a resend. TCP is a stream
protocol, and it is permissible for packets to be split and/or combined in
transit, so you do need to handle partial "lines" (so if the receiving end
is using <cr><lf> as a delimiter, if the end of the packet does not have
the delimiter, you need to prepend it to the next packet before searching
for delimiters).
In contrast, UDP will deliver the entire packet as sent -- no
fragmentation -- but it will just drop corrupted/lost packets. Unless the
data in the packets contains sequencing information, the application has no
way of knowing that a packet was lost. It's a "fire&forget" protocol.
Something else to consider: does your data need to be (ASCII) strings?
Using the struct module would let you build up binary data structures which
take less space in transit (though may be more difficult to verify). That
is, instead of sending 4294967295 (10 characters/bytes) you send the 4-byte
equivalent (internal 32-bit integer).
And I will finally learn a python
Thanks for using my language. I read and thought about your code but in the end I chose to use SSH rather then telnet. SSH support is already present in the debian distribution I am using so I could get it working without writing any more code on the beagle bone side.
If that’s a sustained rate, and not burst, it is likely extreme for serial protocols.
After rechecking my numbers, I agree it was a bad idea for me to be using serial for this data rate. I have switched to using SSH. I attach a minimal example below that I wrote to prove that the communication is faster and error free. To run this the paramiko library is required.
import paramiko
import time
setup logging
paramiko.util.log_to_file(‘BBB_paramiko.log’)
port = 22
hostname = ‘192.168.1.34’ # highest measured bytes per second over ethernet: 7,868,481
hostname = ‘192.168.7.2’ # highest measured bytes per second over usb: 5,941,004
username = ‘root’
password = ‘’
client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(hostname, port, username, password)
stdin, stdout, stderr = client.exec_command(‘yes 1234567890’, bufsize=2**24)
print(‘streaming started’)
startTime = time.perf_counter()
total_data = 0
for i in range(10_000):
line = stdout.readline()
if line == ‘1234567890\n’:
total_data += len(line)
else:
print(line)
x = stdout.read(10_000) # this is faster then readline
total_data += len(x)
print(f’bytes per second: {int(total_data/(time.perf_counter()-startTime))}')
client.close()
The above program transfers data with no losses at an average rate of 2,580,544 bytes per second. The speed changes according to how the data is processed on the PC side so that is probably the speed limitation now. Now that I have proven the communication works fine I will make my pressure data stream over SSH. At some point in the future I will probably remove the checksums on the pressure data once I can show there are no checksum failures in production.
I don’t need any more help, thanks everyone.
One last thing I forgot to reply to:
Something else to consider: does your data need to be (ASCII) strings?
Not really, they are just easy to read. If I hit further performance issues I will switch format.
On Tue, 16 May 2017 02:18:05 -0700 (PDT), David Howlett
<david.howlett3@gmail.com> declaimed the
following:
After rechecking my numbers, I agree it was a bad idea for me to be using
serial for this data rate. I have switched to using SSH. I attach a minimal
example below that I wrote to prove that the communication is faster and
error free. To run this the paramiko library is required.
You could probably have gone to just direct TCP/IP sockets, without
adding the overhead of SSH.
The above program transfers data with no losses at an average rate of
2,580,544 bytes per second. The speed changes according to how the data is
processed on the PC side so that is probably the speed limitation now. Now
that I have proven the communication works fine I will make my pressure
data stream over SSH. At some point in the future I will probably remove
the checksums on the pressure data once I can show there are no checksum
failures in production.
Since SSH nominally runs over TCP/IP, and TCP data transfer is
guaranteed (individual IP packets may be lost, but the TCP protocol is
designed to detect such losses and request resends from the other end --
note that this is at the protocol level, the application never sees a
resend request), you shouldn't really need your own checksum. Every packet
already has a checksum in the TCP header, besides having packet length
information as another check.