problems with webcams

GoogleGroups
61

I have confirmed that it is indeed possible to grab AND convert frames at 640x480 and 30 fps with the BBB and Logitech C920 operating in MJPEG mode. My solution was to basically install libjpeg-turbo from source with NEON enabled and then building and installing OpenCV from source, using libjpeg-turbo as the JPEG codec. I also built OpenCV with NEON enabled.

Here is sample output with every frame being processed. The ‘top’ utility shows about 50% CPU and only about 2% memory usage. Before executing the code, I set the CPU to operate at constant 1Ghz using “sudo cpufreq-set -g performance” so that there isn’t a “slow start” while the BBB starts grabbing frames at 300 Mhz before it finally kicks into 1Ghz under load.

time ./framegrabber -f mjpeg -H 480 -W 640 -c 1000 -I 30 -o
Startup took 0.000000 seconds
Captured 1000 frames and Processed 1000 in 13.940000 seconds
Shutdown took 0.010000 seconds

real 0m33.760s
user 0m11.566s
sys 0m2.518s

I ended up using distcc to do a distributed cross-build of OpenCV on my PC, but still had problems with running out of space trying to store all of the source and object files on the 2GB eMMC on the BBB. I ended up having to mount an ext2 formatted filesystem using a USB thumb drive hooked up to the BBB and building OpenCV from there. I probably ended up doing a lot of extraneous things that had little to no effect but I do have a working process for those who need to get 30 fps. I need to go back and clean up some notes I took and then I plan to post a step-by-step guide for anybody else who wants to replicate what I did.

Thanks,

-Mike

62

Mike,

I look forward to your write up. I am glad to hear distcc worked for you. A note about the files to put on the BBB, you should really only need the compiled libs and the header files. The build I did on my surface (Windows) created a directory called install that has a lib and an include directory with a total size of ~10 MB. I would imagine the build for Linux would be close to that.

63

Yeah. I did a “make install” step that saved all of the files to /usr/local/lib and /usr/local/include. The thumb drive was just for temporary storage while building.

64

Yeah. I did a “make install” step that saved all of the files to /usr/local/lib and /usr/local/include. The thumb drive was just for temporary storage while building.

65

Richard,

Please see the post http://blog.lemoneerlabs.com/post/BBB-webcams concerning the PS3Eye, C920, and frame rates. I believe it will answer your questions.

66

Hi Richard,

Here is a summary of what I ended up doing to get 30 fps out of the BBB. You are right – it simply came down to taking advantage of NEON hardware acceleration. My little “How-To” guide might be a bit verbose, but you can skip all of the background information and skip right to the steps I took if you like. However reading some of the background information might give you some more insight into what I have already tried with the PS3 Eye (along with many others like Matthew) and what we have learned. Its also chock-full of great references on the topic.

Right now, I am considering this a DRAFT as I have not gone back through to make sure that all the commands I pasted will work verbatim. If you’re pretty comfortable in Linux, I’m sure this is enough for you to replicate what I have done.

I wrote it up in LaTeX, as that was the easiest for me, so here it is in both PDF and HTML format.

Best of luck. If you end up taking a look at this, please let me know if you have any comments or suggestions for improvement.

-Mike

BBB_30fps.html (221 KB)

BBB_30fps.pdf (204 KB)

67

Mike,

Great write up. When I have some free time, I will replicate your steps on my BBB running Arch Linux. Not sure if you have a personal website, but would you mind me posting this to my blog once it makes it out of draft?

I look forward to seeing your aircraft in action; on to the CV problems!

68

Thanks! You’re absolutely welcome to distribute it freely. I don’t have my own website so that’s a great way to share the information.

Luckily I have most of my CV algorithm done. I have a bit of cleaning up to do but most of my work is implementing hardware from here on out.

Thanks again SO much for your help! I look forward to getting feedback on the write up.

-Mike

69

Mike,

Looking over the code in you document, I noticed some formatting was off and I needed to clean up the way I handled processing the subset of the frames. I am cleaning things up and will get you a new version shortly.

70

Mike,

Here is the cleaned up one. Here are the differences:

-o is not used to indicate which frames to convert to OpenCV Mats and requires an integer argument

-o 1 would convert every frame

-o 2 would convert every 2nd or ever other, etc

default is 1

-p is similar to -o in the original framegrabber. However, it doesn’t actually output anything. It just controls if any frames are to be converted.

Captured count and processed count variables have been renamed and moved to the top.

Formatting has been corrected.

Testing of your procedure to follow.

framegrabberCV.c (19.8 KB)

71

-o is not used to indicate which frames to convert to OpenCV Mats and requires an integer argument
should read
-o is now used to indicate which frames to convert to OpenCV Mats and requires an integer argument

72

I just wanted to say thanks for this awesome thread. With your procedure and code, I have VGA capture now working too. I would like to add the following comments:

  1. I am using a Logitech C230 camera (looks like an eyeball). So you can add this to the list of cameras that support MJPG capture mode with full JPEG output.
  2. I am not getting a full 30 fps, only about 24. But there are differences between my setup and yours. I am running Ubuntu 12.04 (Precise) + LXDE, and haven’t really optimized much except for building libjpeg-turbo and rebuilding OpenCV as you suggest.
  3. I built OpenCV from source on the BBB without using distcc or an external thumb drive. I am booting from a 4GB uSD card. It built in 3-4 hours.

Thanks again!
Jim

73

Jim,

I’m glad to hear that you found my write-up helpful. I really appreciate your kind words and will definitely add the C230 as a MJPEG-ready webcam. Out of my own curiosity, are you 1) doing any additional processing with OpenCV or 2) displaying the video in real-time (either of which could be cause for a slower frame rate)? If you have LXDE, my guess is that #2 is the culprit. At the very least, it sounds like you’re happy with the performance for your application and thats what matters! =)

Thanks again!

  • Mike
74

Hi Mike,

Yes, I am doing additional OpenCV processing, and have 4 video windows displayed, 2 being updated in “real-time” (hah!). I have capture running in a separate thread. My frame processing rate is well under 1 fps, so I’m ecstatic with 24 fps capture rate.

Thanks again!
Jim

75

Excellent, excellent thread! I just purchased my BBB yesterday for a personal project, and I’ve been reading for many many hours how and what would be the best way to stream video from the BBB to a PC over WiFi with the lowest latency as I could get. This thread, including all the links that users posted here, were really helpful in understand the steps I should take.

I’m just running to the store in a few minutes to buy a C920 and start trying stuff :slight_smile: I’m glad I’ve waited and read a lot before buying the PS3 Eye… Thanks guys. I will update any results I achieve.

76

Glad to hear it. Looking forward to your updates.

-Mike

77

Glad to hear our work is helping others. Let us know how your endeavors go.

-Matthew

78

I am working with the C920 on an ODroid, similar to the BBB. I am also having a problem capturing more than 15fps in OpenCV, however, I need to use OpenCV for doing processing on the video. Since I am not just grabbing frames, I can’t do something like framegrabber. Is there anyone who has done this and have a solution. I have tried changing settings on v4l before running my code, but have had no luck.

Thanks.

79

All you have to do is adapt the frame grabber code for your needs. I actually went ahead and rewrote the capture code to be object oriented so that I could conveniently use it with OpenCV’s C++ interface. You might find some of my code helpful as an example to work from. It is in a public git repo:
https://github.com/mdarling39/LinuxVision/blob/master/

I think you would primarily be interested in taking a look at CamObj.hpp, CamObj.cpp, and v4l2Cap.h, v4l2Cap.cpp, and/or v4l2_c.h and v4l2_c.c. As a word of warning, my code is sloppy and specific for my needs, but it does what I need it to do. Feel free to borrow from it and modify it to your heart’s content.

I actually don’t remember how all of this stuff works anymore, but you can look at main_threaded.cpp to see how I am using v4l2_c.h to get frames into a cv::Mat object. If I remember right, I had to abandon the object-oriented approach for some reason so that later so that I could implement multi-threading using pthreads. I think the magic you are looking for is in the function v4l2_process_image(cv::Mat, const void), where the frame buffer is actually decoded and saved to a cv::Mat object. Once you have that, you’re home free and can just deal with OpenCV.


static void v4l2_process_image(cv::Mat &img, const void *p)

{
        cv::Mat buff(img.cols, img.rows, CV_8UC3, (void*)p);

        img = cv::imdecode(buff,CV_LOAD_IMAGE_COLOR);

}

I hope that helps you to get started. If you need any help, just ask. I’ll do my best to help where I can.

Good luck,
Mike

80

To piggy back on Mike’s response, when we were testing the capture rate of the BBB, we used a modified version of framegrabber that is attached to this article http://blog.lemoneerlabs.com/post/bbb-optimized-opencv-mjpeg-stream

It allows you to set all the parameters the regular framegrabber accepts but converts each frame to an OpenCV Mat. It is C code and as another word of warning, I won’t say the code is any less sloppy than Mike’s :slight_smile: It was written for testing purposes.

The article itself details how to compile OpenCV to use NEON and libjpeg-turbo courtesy of Mike.

I am surprised to hear the ODroid has trouble capturing higher than 15 fps. I recently bought a Wanboard Quad, but the ODroid was on my short list.

A few things to consider:

Have you tested with framegrabber to see if you can capture the raw frames at a speed greater than 15 fps? It could be that OpenCV is the bottle neck and perhaps you should recompile it per Mike’s instructions.

Is auto exposure turned off? Even with the frame rate set to 30 fps, in low light conditions, the camera increases exposure time to compensate leading to about 15 fps.

Good luck, and as Mike said, if you need help, we will do out best.