Android-based Remote Display Weekly Report

Google Summer of Code
1

Hello mentors,

Here goes the 1st weeks update for Android-based Remote Display project:

  • I’m done with integrating 3 modules together and tested all together in bone.
  • Added some configuration settings in both driver and app so the user can optionally drop frames to speed up video play time.
  • Did some experiments for comparing the transfer speed of bone and my laptop which shows we need at least 4x speed up in USB transfer for bone. Which means the compression times + transfer time should be 1/4th of the uncompressed transfer time.
  • Redesigned input driver functions and data structures to pass more than one input event at a time. It is necessary for mouse drag drop feature and meta key(SHIFT, CTRL etc) functions and took care of those events from application side. So we can now use the android app as a fully functional mouse. Mouse single/double/right click, drag-drop are all in place.
  • Issues:
  • Audio driver can’t be unloaded without disconnecting from USB port
  • Abrupt disconnection from USB causes framebuffer driver crash.

Also available here

2

Week #2:

  • This week I worked mainly on keyboard features. I added most of the key’s in both android and driver side.
  • Added a popup for special keys like function key, PgDn, PgUp, DELETE, INSERT etc.
  • With that most of the works on input driver is finished. A demo available here.
  • Issues:
    • Android doesn’t fire any event for some of the important soft keys like “BackSpace” key. So for now I’ve added an extra key in the “Special Key Popup” for “BackSpace”. I’ll try to find an efficient solution for “BackSpace” key using “TextWatcher” class.
    • Audio driver unloading issue still exists. I couldn’t find any solution for that. So far it seems ALSA attach the driver with the device and as long as the device is connected ALSA considers the driver is being used by the device.
  • Plan for Next Week: Focus on Audio driver. Make the android device work as a fully functional USB speaker.
3
  • Week #3:
    • This week I worked on audio driver. I fixed the audio driver fast-forward problem. So the driver can play audio file correctly.
    • I found the reason of audio driver unload issue and practically it is not possible to unload the audio driver as long as the device is connected to USB port.
    • Issues:
      • Sometimes there is continuous popping noise with the original audio data. So even though the audio goes on perfectly, there are some background noise.
      • The audio driver still causes problem in bone when I try to play something for a 2nd time. It seems to be a tricky issue as I couldn’t solve it after putting enormous time on it.
4
  • Week #4:
    • This week I mostly tried to understand the usage of ffmpeg library. For laptop’s USB, we can get decent speed for both audio and video, but not when we run both together. In bone, we get enough bandwidth for audio, but not for video. Playing both audio and video together in bone is very slow.
    • Also trying to fix some issues related to smaller resolution with the hope of little speed up in smaller resolution.
5

  • Week #5:

    • This week I was more involved in doing some performance testing on multiple platform rather than coding. I tested the USB transfer rate on multiple architecture using virtual machine. The architecture I tested are on Ubuntu, Arch Linux and Debian. The test results lead us to believe the smaller cache size might be responsible for higher page fault in bone. Since we can’t reduce the number of page faults in bone some optimization needs to be done to solve our problem. Right now I’m thinking of lazy update meaning a page won’t be transferred over USB immediately on the hope that it’ll be updated again in near future. I’m not sure how effective this might be or how complex the solution will be. Also it’ll reduce the FPS rate. Next week I’ll put some time on this. I’m also open to better suggestions.
6
  • Week #6:
    • I implemented the lazy update method on framebuffer driver. So now the framebuffer driver doesn’t send to much data over USB connection and as a result we have an accurate timing of the video play time. But as a side effect, frame rate is hampered a little bit which won’t matter while working in terminal or reading or writing codes or mouse movement.
    • The above update doesn’t help much if we also use audio stream. So I dynamically reduced the frame rate when there is audio data. If there is no audio data, the frame rate will be raised to normal level which is now 10 FPS.
    • Also fixed some bugs on android side.
    • ToDo: I’ll try some optimization to increase the frame rate a bit more.
7
  • Week #7:
    • Finally I fixed the long awaited audio issue in bone. So now all the drivers and the android application are fully functional.
    • Doing code cleanup. Now my plan is to do code cleanup and documentation and any change mentioned by the mentors.
8
  • Week #8:
    • I updated the build script significantly which will reduce users setup step. Same script can be used for building & installing the binaries and running X on frame buffer device. Also it will copy the ALSA .asoundrc file to the appropriate directory during the installation of the driver. So in short this script will reduce a lot of initial setup steps for user.
    • Did some code cleanup on frame buffer driver, audio driver and input driver.
9
  • Week #9:
    • I’m passing my time in writing some docs, code cleanup and issue fixing. Fixed some issues on android side and doing more tests to find out new issues.
10

how about the benchmarking, I would like to see a set of results for that?

11

Do you mean the raw transfer rate and those stuffs?

Azizul Hakim

12

Do you mean the raw transfer rate and those stuffs?

yes

13
  • Week #10:
    • Android soft keypad doesn’t fire any events for meta keys like ‘BACKSPACE’. So I had to add use a completely separate button for BACKSPACE key in my earlier version of the android application. It did the work but was confusing as there is already backspace key in the soft keypad and normal user would like to use that key instead of a completely different key for backspace. So now added some hidden TextField and used TextWatcher to track any change in the TextField. Based on this approach now we’re able to use all the keys of the soft keypad.
    • I updated the wiki page with some information. It’ll be great if I can get some suggestion on what more needs to be added or what else it should contain.
    • About the raw bandwidth and driver performance:
    • Raw transfer rate 3.5 Mbps, both in beaglebone and laptop
    • Using lazy update we get 10 FPS.
    • I’ve to reduce frame rate to 3 FPS when there is active audio stream.
14

Week #11:

  • Done with the framebuffer driver cleanup. Removed unnecessary functions specific to DisplayLink device. Also cleaned up audio driver codes.
  • Tested bandwidth with different packet size. No significant improvement though.
15

Week #12:

  • No massive change in code. Just trying to trigger bugs and fixed some minor bugs.
16

Final Report:

My final report is available here.

I only added the link as the report is kind of big.

Thanks

Azizul Hakim