SGX on 4.4.*-ti-rt kernel

Hi,

I try for the first time to enable SGX feauture on BBB using the kernel version 4.4.* to experiment and try to use it with weston.
I use jessie debian image (rcn) updated to kernel 4.4.16-ti-rt-r38
After the update, I install also sgx module:
$ apt-get install ti-sgx-ti335x-modules-4.4.16-ti-rt-r38

In this package there are:
pvrsrvkm.ko
bc_example.ko

I don’t know if I need also omaplfb.ko.that I find in documentation.
Is there in any package? What packages I need to install for this kernel version?

Regards,
Matteo

So, you just need the userspace bits:

git clone -b ti-img-sgx/1.14.3699939
git://git.ti.com/graphics/omap5-sgx-ddk-um-linux.git --depth=1

cd omap5-sgx-ddk-um-linux/

sudo su
export DISCIMAGE=/
export TARGET_PRODUCT=ti335x
make install
exit

#fix: libgbm.so.2
sudo ln -s /usr/lib/arm-linux-gnueabihf/libgbm.so.1
/usr/lib/arm-linux-gnueabihf/libgbm.so.2

sudo mkdir -p /lib/modules/4.4.11-00332-gce54280-dirty/extra/
sudo ln -s /lib/modules/`uname -r`/extra/ti335x/pvrsrvkm.ko
/lib/modules/4.4.11-00332-gce54280-dirty/extra/pvrsrvkm.ko

debian@beaglebone:~$ sudo /usr/bin/pvrsrvctl --start
PVR:(Error): OpenServices: PVRDRMOpenRender failed [0, ]
PVR:(Error): PVRSRVInitSrvConnect: PVRSRVConnect failed [0, ]
PVR:(Error): SrvInit: PVRSRVInitSrvConnect failed (4) [0, ]
/usr/bin/pvrsrvctl: SrvInit failed (already initialized?)
(err=PVRSRV_ERROR_INIT_FAILURE)

Yeah, that's as far as i got... TI even got that far in our jessie image too...

Regards,

Yeah, that’s as far as i got… TI even got that far in our jessie image too…

Yeah, you are right..... SGX module is an headache.... inclusion of proprietary 
code. This is the main problem... :-(

...

Well,
with the same commands, have the same error...  good.

another little step:

The problem seems due to a wrong pvrsrvkm.ko in the package.

I have opened a thread in TI forum to know what are the exactly module 
version to build.
[]
And response is that we have to use the latest stack, integrated into the 
last Processor SDK:
<Processor SDK>****

So
in my host
- download ti-processor-sdk-linux-rt-am335x-evm-03.00.00.04 and enter in 
the right path
- copy board-support/extra-drivers/ti-sgx-ddk-km-1.14.3699939 directory in 
my BBB

in my BBB:
$ cd 
ti-sgx-ddk-km-1.14.3699939/eurasia_km/eurasiacon/build/linux2/omap_linux/
$ export DISCIMAGE=/
$ export TARGET_PRODUCT=ti335x
$ export KERNELDIR=/usr/src/linux-headers-4.4.16-ti-rt-r38/
$ make

So, copy by hand pvrsrvkm.ko in the right place: ****<your kernel 
>/extra/ti335x
because "make install" doesn't work.. Arrrrrgggh

Finally:
$ sudo /usr/bin/pvrsrvctl --start
module starts properly and kernel says:
[  208.971076] [drm] Initialized pvr 1.14.3699939 20110701 on minor 1
[  208.974968] PVR_K: UM DDK-(3699939) and KM DDK-(3699939) match. [ OK ]

But when I try to execute any example placed in 
omap5-sgx-ddk-um-linux/targetfs/
sound like

libEGL warning: DRI3: xcb_connect failed
libEGL warning: DRI2: xcb_connect failed
libEGL warning: DRI2: xcb_connect failed
Unable to initialise egl
    egl error 'EGL_NOT_INITIALIZED' (0x3001)
    

Is possible that we have to update also userspace bin?

Regards,
Matteo

Sounds like it’s not using the right libs, maybe check using ldd? It’s trying to connect to X11, whereas these SGX drivers do not support X11, the only backends are Wayland, DRM, and GBM.

Matthijs

sudo mkdir -p /lib/modules/4.4.11-00332-gce54280-dirty/extra/
sudo ln -s /lib/modules/uname -r/extra/ti335x/pvrsrvkm.ko
/lib/modules/4.4.11-00332-gce54280-dirty/extra/pvrsrvkm.ko

You can also just modprobe the module (don’t forget to run depmod unless make install already does so) and then pass the --no-module argument to pvrsrvctl.

debian@beaglebone:~$ sudo /usr/bin/pvrsrvctl --start
PVR:(Error): OpenServices: PVRDRMOpenRender failed [0, ]

I asked someone I know at imgtec; this means “it failed to open /dev/dri/renderD128 or whatever render node it was assigned”. Does anything show up in /dev/dri at all? Anything interesting in kernel log?

I find it a bit disconcerting that the git repo doesn’t work even though the one from the SDK ought to be a snapshot thereof afaik. (He agrees that Matteo’s result with the driver from SDK looks like the kernel driver loaded okay, but the userspace binaries are trying to use mesa libs instead of the sgx ones.)

Matthijs

sudo mkdir -p /lib/modules/4.4.11-00332-gce54280-dirty/extra/
sudo ln -s /lib/modules/`uname -r`/extra/ti335x/pvrsrvkm.ko
/lib/modules/4.4.11-00332-gce54280-dirty/extra/pvrsrvkm.ko

You can also just modprobe the module (don't forget to run depmod unless
make install already does so) and then pass the --no-module argument to
pvrsrvctl.

Oh i had forgotten about --no-module, I had found it once before in
ti's rc.pvr init script..

ti's rc.pvr:

debian@beaglebone:~$ sudo /usr/bin/pvrsrvctl --start
PVR:(Error): OpenServices: PVRDRMOpenRender failed [0, ]

I asked someone I know at imgtec; this means "it failed to open
/dev/dri/renderD128 or whatever render node it was assigned". Does anything
show up in /dev/dri at all? Anything interesting in kernel log?

debian@beaglebone:~$ sudo modprobe pvrsrvkm

(nothing in dmesg)

debian@beaglebone:~$ lsmod | grep pvr
pvrsrvkm 379674 0
debian@beaglebone:~$ ls /dev/dri/*
/dev/dri/card0 /dev/dri/controlD64

debian@beaglebone:~$ sudo /usr/bin/pvrsrvctl --no-module --start
PVR:(Error): OpenServices: PVRDRMOpenRender failed [0, ]
PVR:(Error): PVRSRVInitSrvConnect: PVRSRVConnect failed [0, ]
PVR:(Error): SrvInit: PVRSRVInitSrvConnect failed (4) [0, ]
/usr/bin/pvrsrvctl: SrvInit failed (already initialized?)
(err=PVRSRV_ERROR_INIT_FAILURE)

X.org was running, so let's kill that and it's friends (x11-common removed)

no change...

I find it a bit disconcerting that the git repo doesn't work even though the
one from the SDK ought to be a snapshot thereof afaik. (He agrees that
Matteo's result with the driver from SDK looks like the kernel driver loaded
okay, but the userspace binaries are trying to use mesa libs instead of the
sgx ones.)

Regards,

Those two nodes are from tilcdc obviously, so nothing in log and nothing in
/dev/dri, that strongly suggests it's not matching against the device tree.

How was this module built?

Matthijs

https://github.com/rcn-ee/ti-sgx-omap5/blob/master/build-ti335x-4.4.x.sh

against a mirror of this tree/branch:

http://git.ti.com/gitweb/?p=graphics/omap5-sgx-ddk-linux.git;a=shortlog;h=refs/heads/ti-img-sgx/1.14.3699939/k4.4

make ARCH=arm CROSS_COMPILE= KERNELDIR=/build/buildd/linux-src clean
make ARCH=arm CROSS_COMPILE= KERNELDIR=/build/buildd/linux-src

/build/buildd/linux-src is where we just built the kernel, so
zImage/modules exist in that directory..

build log:

https://gist.github.com/RobertCNelson/1a3a9ca14e537a706905027d6d9b60c0#file-ti-sgx-ti335x-modules-4-4-23-ti-r51_1jessie_armhf-deb

Regards,

So libEGL is Mesa's Wayland extensions. Mesa is a 3d graphics library. Not
sure how SGX even comes in at this context ?

So libEGL is Mesa's Wayland extensions.

No.

Mesa is a 3d graphics library. Not sure how SGX even comes in at this
context ?

Well it's not going to if you accidently use the wrong libEGL.so, but

there's also one among the SGX userspace libs.

Matthijs

So libEGL is Mesa's Wayland extensions.

No.

<https://en.wikipedia.org/wiki/EGL_(API)&gt;

Mesa is a 3d graphics library. Not sure how SGX even comes in at this
context ?

Well it's not going to if you accidently use the wrong libEGL.so, but

there's also one among the SGX userspace libs.

Matthijs

Yeah I know what EGL *is* but never used libEGL. Did a search wound up on
Mesa's webpage where it said that. But now I can not find the same
information. Which is funny because all I did was highlight "libEGL" from
your post,right click, and google searched. Both times.

How was this module built?

https://github.com/rcn-ee/ti-sgx-omap5/blob/master/build-ti335x-4.4.x.sh

I belive to have discovered what is the problem:

in build-ti335x-4.4.x.sh script, when specify make_options, there isn’t
“TARGET_PRODUCT=ti335x” option.

If not specify TARGET_PRODUCT, makefile compiles this module for the Jacinto SoC as default.

Matteo

Nice find Matteo, I missed that the makefile grew a device option…

Regards,

That worked awesome, pvr's now talking:

[ 188.742066] [drm] Initialized pvr 1.14.3699939 20110701 on minor 1
[ 398.460105] PVR_K: UM DDK-(3699939) and KM DDK-(3699939) match. [ OK ]

tagging and pushing out:

4.4.24-ti-r56 with sgx module build fix..

#one line:
git clone -b ti-img-sgx/1.14.3699939
git://git.ti.com/graphics/omap5-sgx-ddk-um-linux.git --depth=1

cd omap5-sgx-ddk-um-linux/

sudo su
export DISCIMAGE=/
export TARGET_PRODUCT=ti335x
make install
exit

#fix: libgbm.so.2 (one line:)
sudo ln -s /usr/lib/arm-linux-gnueabihf/libgbm.so.1
/usr/lib/arm-linux-gnueabihf/libgbm.so.2

sudo modprobe pvrsrvkm
sudo /usr/bin/pvrsrvctl --start --no-module

Regards,

The libraries are built for egl/wayland, so remember no x..

Regards,

What's the status of wayland on the BBB with Debian Jessie?

...or is that more "build from source" fun, like the SGX drivers
themselves?

Well, my wayland script in jessie bombed:

https://github.com/RobertCNelson/boot-scripts/blob/master/tools/wm/weston-drm.sh

lots of tilcd noise over dmesg..

I'm pretty sure we will have to back-port wayland from stretch..

But this scares me:

http://git.ti.com/gitweb/?p=graphics/omap5-sgx-ddk-um-linux.git;a=commit;h=8d4ba04a2da59989b21aacca9defdf387e321695

the gcc 4.9 -> gcc 5.x was so much "FUN" in stretch over the summer, ^
might mean we have to just use stretch..

Regards,

I’m pretty sure we will have to back-port wayland from stretch…

But this scares me:

http://git.ti.com/gitweb/?p=graphics/omap5-sgx-ddk-um-linux.git;a=commit;h=8d4ba04a2da59989b21aacca9defdf387e321695

the gcc 4.9 → gcc 5.x was so much “FUN” in stretch over the summer, ^
might mean we have to just use stretch…

I agree with you… this commit could be determinant for choose a right strategy…this library now are built with gcc 5.x!!!

IMHO could be a good idea to work in stretch.
From debian.org site, on 2016-Nov will starts the freeze transition.
So packets could be considered quite “stable” and I think it’s better to use a tested environment, for the wayland complexity.

Personally I don’t like the back-port operation. The risk is to obtain a custom wayland version and discover new problems generated by backport, without debian support.

Should be fantastic to have a minimal stretch image with kernel 4.4 + SGX, only console runlevel 3, without X, as base to start to try wayland.

Other opinions are welcome.

Regards,
Matteo

… ops… I use a wrong word…gramatical mistake… sorry…

It would be fantastic to have a minimal stretch image…

Like:

http://elinux.org/Beagleboard:BeagleBoneBlack_Debian#microSD.2FStandalone:.28testing.29.28Bone.27s.29

PS, it's 4.9.0-rc1 based so make sure to run:

cd /opt/scripts/tools/
git pull
sudo ./update_kernel.sh --ti-channel --lts-4_4

(and the correctly built sgx module is now in the repo..)

Regards,