Writing large files to USB port

I would like to write some large files (about 7 gb each) to a 64 gb USB flash drive from the network port on my bbb in real time. The file data arrives on the network at about 18 mega bits per sec (tv mpg files). Does it matter which USB port I connect the flash drive to? Right now I'm using the large regular USB port on the bbb, but I get a lot (every 4-5 seconds) network errors. It runs fine at slower speeds such as 4 mega bits per sec. I also tried reading from the Ethernet port and writing back to a network NAS server, but it couldn't keep up with the traffic either, that's why I'm trying the USB port.
Thanks for your help.
Bruce

If your USB stick is formatted with VFAT (the "standard" for USB sticks)
then the file size is limited to ~2GB (IRRC). Try e.g. ext2!

I'm way past that point. I have formatted my 64gb flash drive and test it with ext2, ext3, ext4. Ext4 is by far the fastest and ext2 the slowest. I can access the entire flash drive (well at least 59 gb anyway), but I can't seem to write faster than 5 MB/s.

There is another recent thread in the BBB stream called “Video Camera Capture”. You might want to compare notes with Matthew there. He is seeing USB access top out at about 13.2MB/S.

Sorry, that would be “VGA Camera Capture”

Did you play around with the mount options as well?

but I get a lot (every 4-5 seconds) network errors.
This might indicate that your filesystem is syncing every 5s and this disturbs the network stream.
You might want to try

mount -o sync /dev/usbstick /mnt

Further, you might even write “raw” to disk without filesystem and mounting, e.g. something like
scp pathtofile | dd of=/dev/usbstick

Interesting ideas. I'll try the sync option first. The files recorded with these errors are still viewable, they just have some stuttering throughout. I'm just a little surprised that I can't get my write speeds above 5MB/s with a flash drive. But I still have several more flash drives to test. I ordered one that supposedly will write at 18MB/s with USB 2.0. We'll see. My last test was with a flash drive that would write at 10MB/s, but I could only get 5. Maybe I'll have to go to a solid state USB drive.
Thanks for your suggestions. I'll keep this updated with my results.

Bruce

Bruce, if that 5MB/s limit you’re seeing is on the USB host port. Then i am afraid it is most likely the flash drive you’re using. I have had a USB HDD connected to this same port, and am getting slightly better than 20MB/s write. I also suspect that the crappy USB <->ATA controller for this drive is holding speed back some too.

I'm beginning to think you are correct. Even though the mfgr specs say 10MB/s write, it always doesn't work that way. I''m trying to make this system a low energy system, that's why I'm trying the flash drives. I have a 18MB/s flash drive on the way from newegg, so if that doesn't work a hard drive is next. But ultimately I'll go to a solid state drive for lower energy if the flash drives don't work.Thank you for responding. I m encouraged by your 20MB success. I have a 250GB USB drive to try next.

Thanks,
Bruce

For sure, there are a few high performance USB thumb drives out there. I was told these new tiny drives from wallmart ( 16GB-32GB ) flash drives to around 30MB/s to a PC. BUt I have yet to get one and test this myself. Then there are those corsair drive( voyager? I dont remember ) that newegg sells that are also supposed to be pretty fast.

That's pretty much where I am. I'm trying to find a fast thumb drive, since I think that would be the best option if it is fast enough. I tried several from tiger direct and two of them wouldn't even support Linux. The third one is a centon 64gb that was advertised to write 10MB/s with USB 2.0, but so far I can't much better than 5MB out of it. I 've a transcended coming with 18MB/s specs, so I'll test that next. If no go there I'm going to spring for a ssd drive, looks to be about $100 or so. Other than the cost, I would be ok with that. I'm already over $ 400 for this ota antenna to dvr idea, but I am really learning a lot, especially about Linux on the bbb.
Thanks for your ideas and help. It is good to know what other people are doing with the hardware.
Bruce

Here is an update on my experiences with testing. I have tried over 60 combinations and I think I have found a pattern for what is happening.
I have tested with both qam digital cable (Bright House) and a digital antenna. I tried several small flat 12 x 12 antennas, but they just didn’t
work well at my home in Florida. Finally, my third antenna, a Wineguard Flatwave gives me five stations with signals quality above 80%. All the
Qam cable channels are 100%. I can record any channel, just fine, with my Win/XP to a local hard drive. But I’m trying to come up with a small, low energy system that I can run
24/7. Maybe I should have bought a channel master DVR or something like that for just my atsc recordings. But, again, I wanted to do something different.
Soooo, I bought a Beaglebone Black, a SiliconDust Dual, a Addonics NAS 4, and a 1TB Seagate external USB 2.0 drive. I tried both a 100mb and a 1Gb switch,
but there doesn’t seem to be any difference.
My first test using hdhomerun_config and recording to the NAS 4 drive would show lots (thousands) of network errors. I concluded that the Beaglebone wasn’t
fast enough to read and write to the ethernet port fast enough with the HD channels (10 to 18 Mbs). In all my tests the SD channels (2 to 7 Mbs) would always
record fine with no errors. So I tried writing to a 16GB Lexar flash drive (of unknow specs) on the BBB USB port. I tried ext2, ext3, ext4 FS with very similar results.
Overflow and network errors. The recordings would playback on my win/xp media player ok, but with fairly frequent stuttering. Since the Lexar was something I’ve had
for years and it would only record about 2 hours of HD I bought a 64GB Centon Rush flash drive with a write speed of 10MB/s. I had a lot of difficulty getting the drive to
format to any of the ext formats until I tried Magictool partition on my win/xp. It still seemed like I would have to try various formatting configurations to get things to work,
but when I did, it would run with many more errors than the Lexar. So I bought a 64GB Transcend drive with 18MB/s write speed. Same formatting difficulties, but even more
errors than before. I mean I would get over 20,000 overflow errors and gobs of network errors. The screen didn’t show that many as the HD_config program ran. They would occur
every fews seconds though. I found articles on the web about unusual formatting ideas for flash drives and tried some, but nothing changed. I did come across a very helpful
disk write test (dd cound=1000 bs=1M if=/dev/zero of=usb/test) that would show me the write speed for any flash drive and FS combination. At this point I thought the flash
drives were the problem, so I formatted and installed a 250GB USB 2.0 external Seagate drive. The write speed test showed 24MB/s! Wow that should do it. WRONG! I
just got more errors than ever before. I don’t know what is going on but I’m GUESSING (I’m not technical enough to know why) that the faster the drive the more resources the
BBB has to use to perform the writes and just can’t keep up to the ethernet input data from the HDHD dual. I had a class 4 SD 8GB memory card around, so I tried that and low and behold
it ran a lot better. Still some errors, but not bad. Now I thought that maybe the USB flash sticks were the problem and 32GB class 10 memory card might be the answer since the memory
cards seem to be much more regulated by speed according to the class, ie; 4=4MB, 6=6MB, 10=10GB. So I spent some more money (I’m over $400 at this point) for a 32GB class10.
But nope - more errors. So once more I went back and bought a 32GB class4 microSD card. Now here’s what happens. Formatted with ext2 the speed test returns 1.3MB/s. That is not
fast enough to record about 1.8MB/s HD video. Ext3 runs at 2.1MB/s. When I record with this config I get a few o’s on the screen, but 27,695 overflow displayed at the end of a 1/2 hour recording.
The video plays pretty good. Now I tried ext4 = 3.8MB/s. That records with very few o errors on the screen and only several hundred errors displayed at the end. The video plays fine.
So there it is. With a BBB I needed to slow the USB drive down to avoid overloading the BBB. I had a programmer write a C program, using the SD HDhomerun libraries, to record my
shows from a hand edited text schedule file. It now runs 24/7 on the BBB recording the evening NBC HD news and Craig Fergusen every night. Works great! It still gets a few
stutters here and there, but I’m not convinced it’s just the antenna signal. This has been a super learning experience, especially learning Linux from scratch. I still don’t get why the
HDhomerun_config program shows so few errors on the screen with the dots(.) and then displays so many at the end when I ctl/c to end it. But no matter the SD libraries do seem to
work and I grateful to have everything working. I’ll bet the new HDHR with h.264 encoding will solve my data volume overload problems, but now I’m waiting for the the new Simple TV.
Maybe that’s the way to go. Best of luck to all.
Bruce

Hi,

I’ve read your conclusion with interest and I’m curious if I understood it properly:

You’re saying, that when the USB-drive could answer quickly then the BBB has problems to keep the performance. So something like: BBB checks the maximum transfer rate, use this maximum and tries to send with this speed. Due to some architectural problems (?) BBB cannot keep the pace and therefore fails. If it would detect lower speed then no problem in writing. Is this right?

Q1: Do you mean the USB-interface and there is a SD-card-reader attached OR do you mean in the above test the internal SD-drive?
Q2: Is there a difference between a HDD and a SD-card and Ethernet? (Ethernet means: Computer attached to the switch and receiving the packages)
Q3: Did you observe a difference between USB2.0 and 3.0-drives? (Cause I would guess that 3.0 devices should be more likely to be limited by the 2.0-speed then 3.0)

Yes, contrary to what you would expect, the faster the drive, the more errors I get. It seems to me that if you get the device and the filesystem to write at about 3MB/s (that's mega bytes, not megabits) it works pretty well. For example, my 16gb Lexar USB flash thumb drive writes the following: ext2=2.0 MB/s, ext3=2.9 MB/s, and ext4=5.3 MB/s. I can record to this thumb drive with ext3 for long periods of time with zero errors. But this drive is limited to storing about 2 1/2 hours of HD tv. The USB thumb drive is connected to my bbb with a USB hub because I'm also using a usb wireless keyboard. So the wireless dongle is on the hub also. I use a 2amp power supply to the 5v plug on the bbb to insure the bbb and the hub have enough power.
I have tried 5 different USB 2 & 3 thumb drives. Some wouldn't format with Linux. I have tried several USB flash memory cards ( normally used for cameras, etc) and the class 4 (rated for up to 4 MB/s) 32GB Kingston works the best formatted to ext4 (3.8 MB/s with speed test). A 32GG class 10 card failed miserably. I tthought my 250GB USB 2.0 external Seagate hard drive would work great since the speed test ran at 24MB/s, way faster than anything I've tried, but it got excessive errors too.
Both tests to my NAS (speed test 11MB/s) gets excessive errors from the bbb and from my win/xp, both running hdhomerun_config, although one is the Linux version and one is windows version.
I can get things to work, but it's always to opposite of what you would expect to work.
Bruce