VGA camera capture

My follow-up on this subject:

  1. In attachment, a table with experiments on the cameras I have. Entries are in the format “laptop / BBB”.

  2. To test the USB bandwidth, I experimented playing some video files on a USB pen connected to the BBB with mplayer and redirecting X to my laptop (connecting with ssh -Y). The movie files were:

a) RIFF (little-endian) data, AVI, 608 x 256, 23.98 fps, video: XviD, audio: MPEG-1 Layer 3 (stereo, 48000 Hz)
b) RIFF (little-endian) data, AVI, 720 x 404, ~30 fps, video: DivX 5, audio: MPEG-1 Layer 3 (stereo, 48000 Hz)
c) ISO Media, MPEG v4 system, version 1

Results are: a) played OK, b) played a little slow and c) played really slowly, almost stopped.

  1. I have a theory on why the Logitech camera works and the others don’t. The Logitech is the only one which is able of fps = 5, 10, 15, 20, 25 and 30. The others are 30 fps (YUYV) and one of them is 60 fps (MJPG). This goes along with what Matthew (?) said: fps must be reduced in order to be able to capture. This reduction is probably made automatically with the Logitech, while the others are not able to go below 30 or 60 fps.

  2. I’ve been experimenting with this v4l2 code:

fd_set fds;
FD_ZERO(&fds);
FD_SET(fd, &fds);
struct timeval tv = {0};
tv.tv_sec = 0;

if(-1 == xioctl(fd, VIDIOC_STREAMON, &buf.type))
{
perror(“Start Capture”);
return 1;
}

/* int r = select(fd+1, &fds, NULL, NULL, &tv);
if(-1 == r)
{
perror(“Waiting for Frame”);
return 1;
}*/

if(-1 == xioctl(fd, VIDIOC_DQBUF, &buf))
{
perror(“Retrieving Frame”);
return 1;
}

if(-1 == xioctl(fd, VIDIOC_STREAMOFF, &buf.type))
{
perror(“Stop Capture”);
return 1;
}

to see if I could reduce the interval during which the stream is open (and data is transferred through the bus). This code is more or less the same in Derek Molloy’s examples, v4l2grab, etc. but I’m using it from capturev4l2. I had to add the VIDIOC_STREAMOFF, otherwise my laptop would crash with kernel panic. I moved the FD_* code out of the interval and commented the call to select, but it does not change the result.

cameras.pdf (36.9 KB)

João,

Please see my post here http://blog.lemoneerlabs.com/post/BBB-webcams

There you will find some timing results for the PS3Eye and the Logitech C920 as well as an example application that will allow you to change the frame rate.

Yes I successfully used capture.c etc, then found I couldn’t find a suitable imageviewer program for angstrom. Gimp always crashes.
Btw Matthew, with the moustache placer code there’s a little script which is supossed to be able to tweet you your photo. That might be useful to look at.
It is certainly an interesting set of results. Basically opencv can be very limiting. But it’s excellent to see how much data is passed by the camera in one go.
I did look at the narcissus website yesterday which handles angstrom package building. However it looks like the stripped down version can provide less usb functionality. But I still feel its all in the software used for streaming that the cpu power is used, so I’m not sure if distributions can be managed. One potential for me would have been to lower the CPU MHz and reduce the power consumption, so I could run on solar/battery.
Keep up the could work. I only really need to be able to tell python to be able to find the opencv modules.
If I get a chance I might try and go for a step by step attempt with C++.
James

Matthew,

When you say:

I can tell you with the Logitech C920 it is possible to capture 1920x1080 frames with the FPS set to 30 in both MJPEG and YUYV format.

you mean in BBB?

Correct. On the BBB, you can still capture frames from the C920 with the FPS set to 30; however, you will only get about 12 FPS in performance. Because of the way the PS3Eye sends the data, you cannot receive any frames when the FPS is set to 30 and the resolution is 640x480.

Actually I was experiencing a thing with my camera where I think it was set to 640x480 but it gave me 3 streams fitted into the 640 window none of them being 640x480 but it had tried to fit stream data to the window.
A bit odd, but makes some sense, but interestingly it may have forced the stream into the window.

Regards,
James

Matthew,
If opencv is correctly in your pythonpath can you post what it says.
For me in the python console.

Import sys
Print sys.path

Tells me which folders are findable by python. I can get the opencv folders into the pythonpath within the python console, but back into the command line python will not find the folders.
If I recheck they are no longer listed. So it’s not making a permanent change. It may be just one file.

James,

Here is the output

[’/root’, ‘/usr/lib/python27.zip’, ‘/usr/lib/python2.7’, ‘/usr/lib/python2.7/plat-linux2’, ‘/usr/lib/python2.7/lib-tk’, ‘/usr/lib/python2.7/lib-old’, ‘/usr/lib/python2.7/lib-dynload’, ‘/usr/lib/python2.7/site-packages’]

In the screenshot, you can see OpenCV (cv) is in /usr/lib/python2.7/site-packages on my BBB.

opencv.JPG

Hi Matthew,
Thanks for the image as it shows exactly what I’m missing. Whatever I do with configuring python I’m not going to get a cv.py and cv.so without building it specifically. So adding opencv literally to the folder wasn’t going to work. Build instructions for opencv for python a probably a must then. I don’t think I have to worry about it too much.
I managed to spend the weekend reading up on neon optimizations. Writing assembly for neon proves to be a whole lot faster and I think that just using neon flags doesn’t do a lot without code optimization. http://computer-vision-talks.com/2012/07/opencv-tutorial-part-4/

Shamefully there is one book that deals with neon optimizations but its seriously written for iphone/ipad. So the code examples would probably need some debugging to work which you can be really stranded if you don’t know. Uncannycv reports some excellent optimizations.
The issue being then is if opencv can run that much faster if optimized, is it worth taking the neon coding route, but the way the guy in the example has written his tutorial it still works as an addendum to the c++ code.

Firstly i would like to say i came across your discussion looking for select timeout error solution but as i went through it i didnt notice if you found a solution, but if you have it now, i would appreciate it. The select timeout error still seems to be in existence even on BBB Rev C with kernel 3.8.13-bone79. I am working on an Iris recognition system that initially has to track eyes in a live video stream (from there captures cropped eye images that will be processed). I have installed different OpenCV versions, 2.4.9, 2.4.11 and 3.0.0 and in all of them i am getting the same error. I am working with a Logitech, Inc. QuickCam E 3500 webcam. I am accessing the BBB Desktop using Tightvnc client on my PC running Ubuntu 14.04 LTS.

What is surprising is that it is capable of video streaming with opencv installed by apt-get (apt-get install python-opencv) but the limitation with this version is that it has very old cv bindings and documentation on some functions for Histogram equalisation is not available online. I am stuck, i need your help.

Hi Tinashe,

Please see my article here http://blog.lemoneerlabs.com/post/BBB-webcams

There is a version of framegrabber.c linked to it that allows you to specify the frame rate with the command line parameter -I. If reducing the frame rate works for you, then the code for framegrabber should provide a starting point for accomplishing the same thing in your own program.

Hi Matthew,

I looked into the article as i went through your discussion but did not try the framegrabber.c, will test it though. I finally got a solution to the problem from links listed below. It kind of made sense to me (lack of memory in ARM systems) because running the same code on my PC worked perfectly well. Running this command sysctl vm.overcommit_memory=1 worked for me after a long struggle. What i’m not really sure are the implications (being it on Hardware or Software) if there comes a state when large size memory is really needed.

The issue causing the select time out detailed in my article has to do with how much data is being sent over the USB and how it is sent. Reducing the frame rate reduces the load on the USB. Looking at the articles you linked, one says it solved an issue but not the select timeout, and the other shows an error message that says it could not allocate memory. Neither one of these are the cause of the select timeout I addressed.

The select timeout occurs when the select times out. Looks like in the cases in your links a previous call to allocate memory failed followed by select failing. I have never had an issue with memory allocation. All my troubles had to do with too much data on the USB. You might want to confirm what your actual problem is.

Hi Matthew,

I’m a rookie in this linux/opencv area i wouldn’t really know what it means, only learning from you. I posted a question https://groups.google.com/forum/#!category-topic/beagleboard/debian/VFuvveM_8Gc looking for a solution because it always happened when i plugged in the webcam on BBB. Running the command i previously mentioned ended my woes.

The timeout error resurfaced again, i followed you blog post on http://blog.lemoneerlabs.com/3rdParty/Darling_BBB_30fps_DRAFT.html but got the following errors in compiling libjpeg-turbo ;

root@beaglebone:~/libjpeg-turbo-1.3.0/build# make
make all-recursive
make[1]: Entering directory /root/libjpeg-turbo-1.3.0/build' Making all in java make[2]: Entering directory /root/libjpeg-turbo-1.3.0/build/java’
make[2]: Nothing to be done for all'. make[2]: Leaving directory /root/libjpeg-turbo-1.3.0/build/java’
Making all in simd
make[2]: Entering directory /root/libjpeg-turbo-1.3.0/build/simd' make all-am make[3]: Entering directory /root/libjpeg-turbo-1.3.0/build/simd’
make[3]: Nothing to be done for all-am'. make[3]: Leaving directory /root/libjpeg-turbo-1.3.0/build/simd’
make[2]: Leaving directory /root/libjpeg-turbo-1.3.0/build/simd' Making all in md5 make[2]: Entering directory /root/libjpeg-turbo-1.3.0/build/md5’
make[2]: Nothing to be done for all'. make[2]: Leaving directory /root/libjpeg-turbo-1.3.0/build/md5’
make[2]: Entering directory /root/libjpeg-turbo-1.3.0/build' make[2]: Leaving directory /root/libjpeg-turbo-1.3.0/build’
make[1]: Leaving directory `/root/libjpeg-turbo-1.3.0/build’
root@beaglebone:~/libjpeg-turbo-1.3.0/build#

Before compiling libjpeg-turbo i tested my code to see what will be happening on the resolution and frame rate. I am working with a Logitech QuickCam E3500 which has support for both YUVY and MJPEG. I set the resolution to 320x240 with a 30fps frame rate but after running the code, got select timeout errors and the resolution had changed to 640x480 YUVY with a 15fps frame rate;

root@beaglebone:~/libjpeg-turbo-1.3.0/build# v4l2-ctl -V
Format Video Capture:
Width/Height : 640/480
Pixel Format : ‘YUYV’
Field : None
Bytes per Line: 1280
Size Image : 614400
Colorspace : SRGB
root@beaglebone:~/libjpeg-turbo-1.3.0/build#

Hi Tinashe,

My two cents:

root@beaglebone:~/libjpeg-turbo-1.3.0/build# make
make all-recursive
make[1]: Entering directory `/root/libjpeg-turbo-1.3.0/build'
Making all in java
make[2]: Entering directory `/root/libjpeg-turbo-1.3.0/build/java'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/libjpeg-turbo-1.3.0/build/java'
Making all in simd
make[2]: Entering directory `/root/libjpeg-turbo-1.3.0/build/simd'
make all-am
make[3]: Entering directory `/root/libjpeg-turbo-1.3.0/build/simd'
make[3]: Nothing to be done for `all-am'.
make[3]: Leaving directory `/root/libjpeg-turbo-1.3.0/build/simd'
make[2]: Leaving directory `/root/libjpeg-turbo-1.3.0/build/simd'
Making all in md5
make[2]: Entering directory `/root/libjpeg-turbo-1.3.0/build/md5'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/libjpeg-turbo-1.3.0/build/md5'
make[2]: Entering directory `/root/libjpeg-turbo-1.3.0/build'
make[2]: Leaving directory `/root/libjpeg-turbo-1.3.0/build'
make[1]: Leaving directory `/root/libjpeg-turbo-1.3.0/build'
root@beaglebone:~/libjpeg-turbo-1.3.0/build#

There is no error above.

Before compiling libjpeg-turbo i tested my code to see what will be
happening on the resolution and frame rate. I am working with a Logitech
QuickCam E3500 which has support for both YUVY and MJPEG. I set the
resolution to 320x240 with a 30fps frame rate but after running the
code, got select timeout errors and the resolution had changed to
640x480 YUVY with a 15fps frame rate;

Due to this, among other reasons, we gave up on the BeagleBone Black and are now using an A20-OLinuXino-LIME.

João M. S. Silva

There are no actual errors in the output you posted.

Do you have access to any other webcams? I have tested this with the Logitech C920, the Logitech C270, and the PS3Eye. Without access to your webcam, it is hard for me to look into what is going on.

Hi,

The same is happening on the other webcam i have (not sure if its openCV changing resolution). I bought it on AliExpress and has the following discription; HD 1.3 megapixel 960p 2.8-12mm varifocal lens Aptina AR0130 android external usb camera module with IR cut ELP-USB130W01MT-FV. The supplier can be found on http://www.elpcctv.com.

Since there are no errors, what could be the problem? Is there some dependencies i have to install?

I tested with your framebrabber.c code and was getting this (see below) with the Logitech webcam,

root@beaglebone:~# time ./framegrabber -f mjpeg -H 480 -W 640 -c 1000 -I 30
Startup took 0.010000 seconds
Captured 1000 frames in 0.220000 seconds
Shutdown took 0.000000 seconds

real 0m42.068s
user 0m0.025s
sys 0m0.223s

The no name webcam was giving the following results;
root@beaglebone:~# time ./framegrabber -f mjpeg -H 480 -W 640 -c 1000 -I 30
Startup took 0.020000 seconds
Captured 1000 frames in 0.260000 seconds
Shutdown took 0.000000 seconds

real 2m10.965s
user 0m0.033s
sys 0m0.262s
root@beaglebone:~#

Silva,

There has to be way. Dereck Molloy https://www.youtube.com/watch?v=8QouvYMfmQo was able to get a video stream then its possible to video stream with BBB. I don’t have much time to look for another development board because the Project is due next week.

If the camera has a good compression mechanism (e.g. MJPEG), it may be softer on the USB bus.

In our board, we are using OpenCV but we don't capture images with the OpenCV instruction. Instead, we use something like this:

https://linuxtv.org/downloads/v4l-dvb-apis/v4l2grab-example.html

For video capturing there is also an example:

https://linuxtv.org/downloads/v4l-dvb-apis/capture-example.html

Note that OpenCV uses v4l underneath.