Collecting RAW Images from IMX219 on BeaglePlay

General Discussion
1

Thought I’d drop a post here demonstrating how I went about reading single-frame Bayer-formatted images from Arducam IMX219 CSI-2 camera on BeaglePlay. Had seen a few posts on this topic that, while they are all several months old, apparently were unresolved. @RenierSTFR @damia

Module Driver

Applied IMX219 device-tree overlay supplied by @Nishanth_Menon here.

Neither the IMX219 nor AM625 SoC have image signal processing (ISP) blocks on their processors, meaning that using them to stream video was effectively impossible. However, seeing evidence of variously-successful attempts online, I took it upon myself to extract pixel data from the sensor via the V4L2 API.

Pipeline Configuration

Running media-ctl -d /dev/media-0 -p shows the IMX219 is being probed succesfully. All entities of the pipeline must, crucially, be configured in uniform before frames can be captured from the camera; configuring the sensor by itself results in an error…

# media-ctl -V '"imx219 4-0010":0[SRGGB8/640x480]'
# media-ctl -V '"cdns_csi2rx.30101000.csi-bridge":0[SRGGB8/640x480]'
# media-ctl -V '"30102000.ticsi2rx":0[SRGGB8/640x480]'

Extracting Pictures to File

Fortunately, the V4L2 API provides a method to obtain a single-frame image from the camera’s memory buffer:

v4l2-ctl --device /dev/video0 --set-fmt-video=width=640,height=480,pixelformat=RGGB --stream-mmap --stream-to=picture.raw --stream-count=1

Items worth considering:

  • If the camera isn’t probing, confirm the CSI cable is connected in the proper orientation …
  • The WXH dimensions can presumably be configured to different values than those above.
  • While obtaining images from the camera is gratifying, note that the pixel format is 8-bit RGGB (Bayer); converting this into RGB and other formats requires add’l processing.

Further Reference:

2

Thanks for this great writeup @egakaned !

Have you considered submitting this as a PR to Using CSI Cameras — BeagleBoard Documentation ?

Cc @lorforlinux , please take a look at this.

3

Do you know how (or is it possible) to access the I2C bus of the CSI, to control the sensor?

4

@Illia_Pikin I would assume it’s possible but it’s unfamiliar territory for me… Sorry I can’t be of greater help.

@dhruvag2000 Thanks a lot! Not sure why the Beagleplay Docs page is laid out this way but there is in fact a tutorial on using CSI under the “Expansion” heading. Also there’s a quick little tutorial in section 4.4 of the Beagleplay Manual on using CSI that was also helpful.

5

Coming back to this thread because the last step in the original write-up (“converting this into RGB and other formats requires add’l processing”) is where a lot of people get stuck, so here is a sketch of how the raw dump from v4l2-ctl --stream-to=picture.raw can be turned into a viewable image on the board itself.

With the pipeline set to SRGGB8 at 640x480, the file is simply 640*480 bytes, one byte per photosite, laid out as an RGGB mosaic. On a BeaglePlay running Debian, python3-numpy and python3-opencv from apt are enough:

import numpy as np, cv2
raw = np.fromfile("picture.raw", dtype=np.uint8).reshape(480, 640)
rgb = cv2.cvtColor(raw, cv2.COLOR_BayerRG2BGR)   # try BayerBG2BGR if colours look swapped
cv2.imwrite("picture.png", rgb)

A few things that are easy to miss:

  • If the colours come out with a strong magenta or green cast, the Bayer pattern constant is off by one row/column. OpenCV names the pattern differently from V4L2, so RGGB can map to COLOR_BayerBG2BGR. Trying the four variants on one frame settles it quickly.
  • Because the AM625 has no ISP, nothing is doing white balance, black-level subtraction, gamma or lens shading for you. A simple grey-world white balance (scale R and B so their means match G) plus a 2.2 gamma curve makes a huge difference before you judge the sensor.
  • The first frame after streaming starts is often under-exposed because the IMX219 exposure/gain controls are at defaults. Setting exposure and analogue_gain with v4l2-ctl -d /dev/v4l-subdevX --set-ctrl (on the sensor subdevice, not /dev/video0) gives repeatable captures. That also partly answers the I2C question above: the driver writes those controls over the camera connector’s I2C bus for you.
  • For 10-bit modes (SRGGB10) the data is unpacked into 16-bit words, so read with dtype=np.uint16 and shift right by 2 before demosaicing to 8 bit.

Once you have clean PNGs, the rest is ordinary image work. For classic processing on the board OpenCV is the obvious choice; for creative experiments on captured frames afterwards (restyling a shot, swapping a background, extending the frame) a browser-based image to image ai editor takes a photo plus a short prompt and keeps the original composition and lighting. That is outside what the BeaglePlay itself needs to do, but it is a quick way to see whether the debayered output has sensible colour before building a bigger pipeline around it.

Thanks again @egakaned for documenting the media-ctl configuration; the “configure every entity in the pipeline uniformly” point is the piece missing from most older threads.