Mike,
I have posted a working C example on my blog here http://blog.lemoneerlabs.com/post/opencv-mjpeg
I was just reading this thread of messages after being away for a few days. I saw mention of the delay when streaming over a network using rtp. The closest to realtime I’ve been able to achieve with my own project using the C920 on the BBB appears to be just under two seconds. (rough estimate by pointing the C920 at the analog clock on my computer)
Is under 0.8 seconds possible with this camera and hardware?
Does anyone have a suggestion of how I can accurately measure the lag that is being intruduced, and where it’s being introduced in the process?
Wim.
Well… I’ve modified Martin Fox’s capture code a bit to implement the OpenCV decodeImage() function you pointed me to (big thanks!) when capturing in MJPEG format (code attached). At 640x480 and 30 fps, it runs flawlessly on my laptop, giving nice and sharp real-time video at a measured framerate of just about 30 fps on-the-dot.
Next, I set up the code to save off one in every 100 frames as JPEGs so that I could evaluate the image quality when running on the BBB. With the settings at 640x480, 30fps, and capturing in MJPEG mode, I actually end up with ~10 frames per second. That might be enough for my application, but I have to add processing time on top of that as I am just acquiring video and throwing it away for now. Also, the few images that I saved off had a bit of motion blur when I jostled the camera around, which could be the biggest issue for me. So I went back to running the code on my desktop and saved off some frames for a comparison, but they appeared slightly blurred as well.
I ended up going back to the BBB again to try and measure the load on the CPU using the “top” utility, but this time recompiled my code with the -Ofast g++ option. At the beginning frames were capturing slowly again, but after a few seconds the framerate increased and settled to just about 20 fps with CPU usage of approx 73%. This might cut it for what I need to do with the BBB. I’m concerned that the motion blur will still be an issue, but that might be unavoidable with this kind of hardware.
Is there any other way that you can think of improving the frame rate or motion blur?
Thanks a ton for all of the help you have provided on this! I think (hope) this performance will be sufficient for me to move forward in my project. =)
-Mike
OCVCapture.h (3.54 KB)
OCVCapture.cpp (16.6 KB)
camera.cpp (1.38 KB)
Mike,
I briefly looked at the code, and one thing I noticed is the MJPEG frame is being converted to an OpenCV Mat and then the Mat is saved. If you want to save the images, you can write them out directly when they are captured as MJPEGs because they are essentially jpegs - there is no need to convert them.
Are you running the code that displays the images on the BBB? If so, you should be able to greatly reduce the cpu use by not displaying the image, but I do not know if that will result in any significant increase in frame rate. I can also tell you that if you forgo saving the images and just work with the images in memory, you should see better frame rates as writing files (in my experience) seems to take a while.
I know I can capture 30 FPS at 640x480 in MJPEG on the BBB without doing any processing on them. I will modify my code to load the image into an OpenCV Mat and see how it affects frame rate. I should have something to report this evening.
As far as motion blur, I do not think we will be able to eliminate it. The cameras available (outside of the PS3Eye) were intended for video conferencing, etc. and do not capture at a high enough frame rate. I read somewhere that the PS3Eye could produce jpegs as well as uncompressed images, but I have found no indication that it is possible with the driver under Linux. Instead, we have to find ways to work with the noisy images.
I am glad to hear this has helped clear the way for you to continue on your project. Maybe you’ll let me read your thesis when you are done?
William,
I do not have a good way as of yet to measure the lag, but the first place I would look is the application you use to view the stream. Depending on how it buffers the stream, it may buffer a second or so worth before displaying anything. For instance, in VLC, in Open Media → Network, click Show more options and there is a setting for how much to cache. Another place where latency is probably introduced is in the conversion to the rtp stream on the BBB and the decoding on the other side. In my case I am capturing from a usb webcam and streaming over a usb wifi dongle. Since the BBB has just one usb port, I am sure there is latency introduced there as well.
Matthew: Thanks for the suggestion on the network caching item. Do you have any idea how to do this in VLC using an SDP file? (I may have figured it out, but I’d like confirmation. In VLC on windows I’ve used the “Open Advanced” option, added the SDP file to the file selection, and reduced the caching to 0ms. )
A simple example of just pushing video from my webcam to my PC using a recent build of FFMPEG has the following command on my BBB:
ffmpeg -f v4l2 -video_size 1280x720 -framerate 30 -input_format h264 -i /dev/video0 -vcodec copy -f rtp rtp://239.8.8.8:8090
When running on the BBB, I commented out all of the namedWindow() and imshow() lines that display the image to save on cpu usage. I also only had it save 1 out of every 100 frames so that it didn’t have to store a lot of data to memory (approximately one image every 5 seconds). I found that the BBB could easily capture the raw image data at 30 fps using your capture code from earlier, but I think the extra overhead of decompressing the JPEGs might be the limiting factor now.
Let me know what you find. I have a little bit more tinkering that I want to do with disabling the autofocus, and investigating a little more into what the PS3 Eye driver does support.
Also, I would be happy to send you a copy of my thesis whenever I finally get it finished!
Thanks!
Mike
William,
I will have to look more into VLC, but what you did sounds close to memory. I actually may have some code to post shortly that I have been working on for streaming. I have been able to stream 640x480 at 30 FPS with no noticeable latency using a modified version of the framgrabber app running on the BBB and a C application running on and windows machine. The windows app uses opencv to display the stream. Transfer of the images from the BBB to the windows machine is done using raw data over a socket. I have more testing to do, but so far it looks promising. CPU use on the BBB is 6 - 12%. I hope to have it on my blog by the tomorrow evening.
William,
I have posted my streaming code and writeup here http://blog.lemoneerlabs.com/post/bbb-mjpeg-streaming
Mike,
Here are the numbers for the version of framegrabber I used to capture and convert to OpenCV images. I ran a run of 1000 and a run of 5000 captures. In both cases, every frame was converted to an OpenCV image.
[root@alarm ~]# time ./framegrabber -f mjpeg -H 480 -W 640 -c 1000 -I 30 -o
Startup took 0.000000 seconds
Captured 1000 frames in 45.890000 seconds
Shutdown took 0.000000 seconds
real 0m47.135s
user 0m43.242s
sys 0m2.955s
[root@alarm ~]# time ./framegrabber -f mjpeg -H 480 -W 640 -c 5000 -I 30 -o
Startup took 0.000000 seconds
Captured 5000 frames in 228.000000 seconds
Shutdown took 0.000000 seconds
real 3m54.043s
user 3m33.734s
sys 0m14.566s
[root@alarm ~]#
In short it runs at a little more than 21 FPS. I noted that memory use during this was only 2% but cpu use was almost 98%. I am not sure it will get much better than that as it appears most of the time is spent in converting to an OpenCV image.
And here are the numbers for only converting every 3rd frame.
[root@alarm ~]# time ./framegrabber -f mjpeg -H 480 -W 640 -c 1000 -I 30 -o Startup took 0.000000 seconds
Captured 1000 frames and Processed 334 in 14.830000 seconds
Shutdown took 0.010000 seconds
real 0m33.963s
user 0m14.105s
sys 0m1.031s
[root@alarm ~]# time ./framegrabber -f mjpeg -H 480 -W 640 -c 5000 -I 30 -o
Startup took 0.000000 seconds
Captured 5000 frames and Processed 1667 in 74.070000 seconds
Shutdown took 0.010000 seconds
real 2m47.270s
user 1m9.268s
sys 0m5.115s
Memory usage is still 2% but cpu drops to ~45%.
I’m making a final last-ditch effort at speeding things up just a little bit more. I am not sure what the build configuration is for the OpenCV package in the Ubuntu ‘apt-get’ repo, but I have found that you can optionally build OpenCV to use a different JPEG image codec. In particular, you can build it with ‘libjpeg-turbo’ (http://libjpeg-turbo.virtualgl.org/), which supposedly is 2-4 times faster than libjpeg. I am going to try installing libjpeg-turbo, and then building OpenCV from source using these build options:
cmake -DWITH_JPEG=ON -DBUILD_JPEG=OFF -DJPEG_INCLUDE_DIR=/path/to/libjepeg-turbo/include/ -DJPEG_LIBRARY=/path/to/libjpeg-turbo/lib/libjpeg.a /path/to/OpenCV
(http://stackoverflow.com/questions/10465209/how-to-compile-opencv-with-libjpeg-turbo).
I’m a little bit concerned that the BBB may not be able to build OpenCV, so if the build fails I’ll have to look into cross-compiling for the BBB. If I am able to get it built, I will recompile your framegrabber program and post the new results for comparison.
-Mike
Max,
Quick update: The BBB ran out of space while building OpenCV near the 40% mark (no surprise there). I will try and do my research on cross-compiling starting with the link you posted awhile back.
Let me know if you have any other tips/suggestions for getting OpenCV cross-compiled. It seems like there are a lot of ways to go about it and none of them seem very straightforward. I should have all of my dependencies for OpenCV pre-installed on the BBB, but I’m not sure that that does me any good now that I have to cross-compile, anyways.
-Mike
I have a 64 gb class 10 sd card. I can install Arch on it and see if I can compile OpenCV without running out of space. If so, it may be an option for you if you cannot get the cross compiler to work. The instructions I sent you are apparently how all packages for Arch Linux Arm are built. Let me know if you run into any issues, and I will see what I can do.
One thing you might also want to do is enable neon support. I am not sure what parts of OpenCV have neon optimizations, but you might be able to squeeze a little more performance out of it.
Mike,
Looks like Arch Linux Arm uses libjpeg-turbo as the system library. Send me your code and I will compile and confirm.
I actually found some really straightforward directions for distributed cross compilation for the raspberry pi. It seems like distributed cross compilation is by far the easiest way to go for compiling big projects with many dependencies like OpenCV. I’m adapting the instructions for BBB and going to give it a shot tonight. I’ve been able to cross compile a small ‘hello world’ program so I know I have the right tool chain and am on the right path.
I’ll let you know if I run into any troubles. If all goes smoothly, I’ll have some results soon.
Oh and also the BUILD.txt file in the libjpeg source directory it says that the package automatically installs NEON on arm systems, so I think that should all be taken care of.
I had to spend some time getting my local network working again so I’ll give distcc a shot tomorrow morning.
Matthew,
Thanks! That may be really useful for me. Am messing around with a PTZ camera at the moment, and lag in the video stream has been a bit of a problem so far…
Regards,
Jon
Matthew – would you be willing to upload the version of your framegrabber.c program that you used to time frame conversion to cv::Mat objects? It looks like OpenCV is building fine via distributed cross-compilation and I would like to be able to compare ‘apples-to-apples’ to see if libjpeg-turbo improves performance.
Thanks,
-Mike
Mike,
Please see the attached file. I left a comment in there for the lines to remove if you want to capture all frames instead of every 3rd.
framegrabberCV.c (18.7 KB)
Jon,
I hope you find it useful. I would appreciate any feedback you have to give.
Best of luck,
Matthew