GSoC 2014 - BARD project reports

Google Summer of Code
1

Week 1 report.

Accomplished:

  • A kernel module which can register a usb-driver for Android device.
  • Probing when device is connected or disconnected

Issues:

  • Had issues with printk’s after usb_register but fixed just a while ago.

Plans for next week:

  • Request Android device to change to ADK mode from the usb-driver
  • USB endpoints to facilitate data transmission.

Best,

Praveen Kumar Pendyala,
Senior Undergraduate,
Department of Electrical Engineering,
IIT Bombay.
http://home.iitb.ac.in/~praveendath92/

2

Week 2 report

Accomplished:

  • ADK mode from user space code now working.
  • Detection and saving of endpoints in the interface.
  • Partial work with issuing control messages.

Issues:

  • usb_control_msg returns error code -110

Plans for next week:

  • Fixing the above issue and complete adk mode setup.
  • Setting up the Android app.
3

Hello Praveen,

can you please write up week 3 and 4 as well.

Regards,

Vladimir

4

Praveen, please reply soon because mid-terms are next week and that could result in not passing.

5

Hi Jason, Vladimir,

Due to high fever I couldn't be there for the weekly meeting. I'm
posting the status updates below.

Week 3 report:

I was attending a conference abroad and not much progress has been made.

Week 4 report:

Accomplished:
- Fixed the usb_control_msg returns error code -110 issue
- The device is now set to ADK mode from the Kernel space.
- Probing and reprobing - On connection probing happens and issues
commands to put the device in ADK mode. Device disconnects by itself
and reattaches in ADK mode. Probing happens again but a different set
of tasks are done this time.
- Registering device on /dev/* for further communication with the device.
- Some reading on the kernel framebuffer and DisplayLink driver udlfb.
- Extracted a framebuffer from /dev/fb0 - more about this in issues
- Testing accessory setup in ADB+ADK mode. (So far all accessory setup
was in ADK mode only). More in issues.
- Initialized the Android repo. Testing is still done on a different
project (not yet pushed to github). Lack of ADB while in ADK mode is a
big pain as of now.

Issues:
- Not able to interpret the format of frame extracted from /dev/fb0.
Will be using DisplayLink instead.
- Although the device is now attached in ADB+ADK mode, eclipse logcat
shows device disconnected. No success on ADB while device in ADK mode
yet.
- When the accessory setup is done in kernel space, even though the
identifiers are right, the corresponding accessory app doesn't show by
itself (unlike in userspace ADK setup).

6

Plans for next week:

- Completing kernelspace-java communication. Should be able to send
data over bulk, receive it in the app and show the data on UI.

- Try to have ADB log running while in ADK mode.

7

Week 6 report:

Accomplished:

In short, the driver to device communication is now ready :smiley:

  • Fixed identifiers encoding issue: Accessory app now shows up on device automatically when it enters accessory mode.

  • Completed Kernel - Java communication: Tested with about 3 bytes. Will be extending after the cleanup.

  • Partially tested Java - Kernel communication: Data received but encoding has to be tested and there are some failures which are to be tested as well.

  • Android app from scratch: Developed the android app from scratch, available on git. Currently it can only be used for communication testing through adb logs. No UI messages but that’s an easy step anyway. It’s working.

  • ADB in ADK: Attempts have been made to read adb logs while in ADK mode. The following two were tried:

  • Userspace using libusb (in issues)

  • Kernel driver only on the ADB interface (in issues)

Issues:

  • Java - Kernel communication: This has to be tested well. There were a couple of write failures on Android end which resulted in indefinite wait on the kernel driver front. The driver read from device code also has to revised to correct for such indefinite wait issues.
  • ADB + ADK : Two attempts were made and the respective issues were:
  • libusb: NULL handle with libusb_open_device_with_vid_pid
  • kernel: Read function hangs up with error -110 (our earlier friend with control messages :/). Will read about adb and see if there should be an initial sequence that has to be sent before attempting read logs from BULK IN endpoint.

Plans for next week:

  • ADB + ADK : Check if this is possible. Some documentation on ADB. Just a couple of more attempts. This would be a great debugging tool! There is a slightly, not as good, alternative if this fails.
  • Device - Kernel tests: Test the java - kernel communication. Figure out the reasons and apply fix for random failures. A few pointers: The write handle on Android being nulled.
  • Kernel - Android for large data: Extend this for larger amounts of data given to the driver from outside, currently from userspace.
  • Start work on DisplayLink driver - udlfb.

PS: If you are wondering about the week numbering, week 4 report is of week 5. I wasn’t around for weeks 3 & 4 - sorry but I’m catching up :slight_smile:

8

Week 7 report:

Accomplished:
- ADK driver: ADB +ADK working: ADB over Wifi is now working for logs
while in ADK mode

- ADK driver: Testing with test data from sent from userspace - test
data is generated in kernel space earlier

- Android app: Fixed fc on opening w/o accessory attached.

- Android app: Added accessory connection status, read values and
write options in the app UI

- Framebuffer: Added android device as DisplayLink device for
Framebuffer. Probing works now, only on the interface of interest - No
duplicate probing issue.

- Framebuffer: Updated endpoint address for BULK writes to device. Now working.

- Framebuffer: Logged raw initialization sequence data sent by the
framebuffer to DisplayLink devices on the Android device. Will be
analyzing this for future work. Can be seen in the image attached.

Issues:
- FB data interpretation: The meaning of the initialization sequence
data from framebuffer is not clear. Also the encoding of the data
might be needed. Not sure if it is UTF8 encoded right now.

- FB Triggers: It is unclear as of yet if there is a need for another
module or userspace code which should be issuing the triggers for
sending updated frames from udlfb driver. Right now the driver isn't
doing any work after the probe function.

- Android app: A couple of FCs when the device is detached from ADK
mode. Should not be a big issue. Can be fixed easily.

Plans for next week:
- Interpretation of the data sent from the FB driver in the
initialization sequence only if that happens to affect the working of
a basic version.

- Finding out how the continuous frame updation part works in the
driver. May be a need for external module?

- If the above two goes well, try to interpret the frame data and see
if we can get an image on the Android device. udlfb uses run length
encoding on the frame data sent to a DisplayLink device.

This report is also available with updates through the week at
http://blog.praveenkumar.co.in/2014/05/beagle-gsoc-progress-tracking.html

Best,
Praveen.

Screenshot_2014-07-01-14-02-10.png

9

Week 8 report:

Accomplished:

  • Framebuffer: Attempted to start xserver with DisplayLink userspace driver and udlfb (issues)
  • Framebuffer: Attempted to start xserver with a couple of other methods listed below, (issues). Few more that I lost track of, they didn’t work though.
  • http://www.thinkplexx.com/learn/howto/linux/system/start-and-use-xfdb-enable-x-server-on-headless-system
  • https://github.com/notro/fbtft/wiki/Framebuffer-use- Framebuffer: Success with starting an xserver on the framebuffer added by udlfb. Details are updated on git README in udlfb repo. fbgrab or fbdump read the expected screen in the framebuffer. Shown in the fbgrab image attached.
  • Android: Changes to the way data received is displayed on Android. The changes are how the TextView data is built without resulting in a log sequence of Garbage collections.
  • Framebuffer: Attempted to use core keyboard as input device for framebuffer (issues)

Issues:

  • Framebuffer: Displaylink userspace driver with udlfb is giving segmentation faults. Fixed the EDID segmentation faults though. This has become hard to debug as the segmentation fault crashes my PC and the log couldn’t be saved. Besides, even with the log files, tracing the root cause for the fault is hard to detect as the logs only show Hexadecimal address in the trace and not the function call or the module that caused it. Alternative found. Can be ignored for now.
  • Framebuffer: Other methods either resulted in a crash or when the x server starts, it doesn’t start on the right framebuffer probably. Because there is data receiption at the device end, no new kernel logs on starting xserver on fb. Also, the framebuffer fbgrab/fbdump doesn’t show the expected frame (only a green screen). Alternative found. Can be ignored.
  • Framebuffer: Framebuffer data encoding has to be figured out. The raw dumps on the Android device show negative numbers also. Also format of the frame sent by the framebuffer has to found out. Mostly PPM.
  • Framebuffer: Framebuffer uses compression before sending the data to the Android device. The algorithm has to be understood and the respective counter part to decompress has be done on the Android side.
  • Inputdevice: When the core keyboard is added as inputdevice the xserver start command gave errors about the inputdevice entry. On commenting this entry, the xserver starts.

Plans for next week:

  • Framebuffer: Figure out the encoding of the data. Since negative values are also received in Java code.
  • Framebuffer: Understanding the compression used in framebuffer (things would be simple if it is RLE ) and writing the counterpart in Java for decompression. Alternatively compression may be bypassed for initial stages.
  • Android: Conversion of the framebuffer frame data into bitmap that can be showed in a ImageView on Android.
  • Android: Decompression part in Java if it is not bypassed right now.
  • Inputdevice: Will try to fix the issue with using core keyboard as an input device for framebuffer added by udlfb.

This report is also available with updates through the week at
http://blog.praveenkumar.co.in/2014/05/beagle-gsoc-progress-tracking.html

fb2.png

10

Week 9 report:

rgb24-bitmap.png

rgb565-bitmap.png

11

Week 10 report####

Accomplished:

  • Framebuffer: Raw frame data is now sent to Android. Successfully removed hline_trim and hline_compress functions without any issues. Removed udlfb urbs and replaced with a new usb transfer code. The new code can transfer upto 16384 Bytes at a time compared to the previous urb method which does 256 Bytes at a time. We are however, transferring only one page (4096 Bytes) at a time after setup. Will improve the size/transfer in future.

  • Android: Tested auto updating of setup frame and the xserver frames with a button click on Android.

  • Android: Implemented a Frame class to hold raw frame buffer data and UI updates are done with the data in this frame. New frame from PC will override the current frame data. This updated frame data can be pushed to the ImageView with a button click currently.

  • BBB: Installed the required packages on BBB - fbcon, X11, fbdev and added framebuffer support. Partially tested the driver. (in issues about a full testing)
    Issues:

  • Framebuffer: The random page fault is now more serious with more frequent occurrences now. Strange thing is that there is no code changes from the version in which the it’s frequency was very rare.

  • Framebuffer: Increasing the work load after each frame reception on Android seems to be triggering the above issue.

  • Android + Framebuffer: Need to encode the page information along with each page transferred for proper frame updates in future and also a decoding scheme on Android. Currently the new page data is filled in a Bytebuffer of framelength and on overflow, it is being filled from beginning. This won’t work when the page faults are in the middle of the frames.

  • BBB: The data sent per transfer when testing the driver on BBB is just 16 Bytes! Possibly the reason for xinit failure.

Plans for next week:

  • Fixing the paging issue and then we shall have a continuously updating frames on Android.
  • Tweaking the Bytebuffer size being used for a Frame. A higher or lower value is triggering an insufficient bytes to copy pixels error on Android.
  • Once the above two are done, will be encoding damaged pages information along with the page data.
  • Testing in BBB and resolving the write buffer as just 16 Bytes issue.
  • Once these issues are resolved, we shall be having a driver that can run a xserver and update frames on Android automatically. The next work would be on compression.
    This is a screenshot (attached) of the auto updated frame from the raw frame data sent from driver. I tried recording a video only to find out that the video was corrupted :confused: Will make one for next week report.

This report is also available with updates through the week at,
http://blog.praveenkumar.co.in/2014/05/beagle-gsoc-progress-tracking.html

Screenshot_2014-07-23-15-57-26.png

12

Week 11 report:####

Accomplished:

13

Note: My net connection so bad that I have to use basic html version
of gmail. For a well formatted version of the report, please refer:

http://blog.praveenkumar.co.in/2014/05/beagle-gsoc-progress-tracking/

Week 12 report:

Accomplished:

-- Checkout the following video demos below
-- Bone: Driver now fully working on bone
-- Bone: Tested with starting an xsession on framebuffer created by driver
-- Bone: Tested with various applications in the xsession
-- Framebuffer: Removed all redundant operations in render functions –
No more URB related operations in hline_render
-- Android: Now uses full resolution image to show the frames. Fixed
all force closes caused when using a full resolution image.
-- Android: Decreased workload on the USB reader thread by shifting
frame updating to an independent runner

Issues:
-- Bone: Even after disconnecting the device and stopping the
xsession, there are continuous calls to damage handle. This is
preventing the unloading of the udlfb module from kernel. rmmode gives
module busy error. However, the driver works on reattaching device at
this stage.

Plans for next week:
-- Framebuffer: Will integrate compression. Standalone tests were
already done however integrating into driver requires some corner case
considerations.
-- Framebuffer: Code cleanup. There are many unused functions and
operations which came with udlfb but were replaced, commented out or
never reached
-- Android: Code cleanup. Reorganise code to decrease workload in USB
reader thread with seperate threads for decompress and UI updates.

Demo video links.
-- First demo - without vertical sync and not full resolution:
https://www.youtube.com/watch?v=ukQJ9EKEcm4

-- Final demo - with vertical sync and full resolution :
https://www.youtube.com/watch?v=drMWtB0ipd0