How to figure out the minimal set of services for an embedded Beaglebone Black app on Debian Wheezy

[Super_Twang]

Hello all,

Is there a good resource (besides this list! that would help me understand what to remove/disable from the stock Debian Wheezy distro, on a BeagleBone Black, to arrive at a barest-minimum/rock-solid setup to run a dedicated app, as an embedded device?

Background:
I’ve developed a C++ app for the Bbb, which I intend to sell as a headless, embedded device that only runs this app when turned on. The Bbb, with a Debian/Wheezy distro, arrives as a much more general purpose device with a lot of services/options that aren’t needed, and to keep the system lean and mean, and as reliable as possible, aren’t desired.

I am a seasoned software developer, but don’t know a whole lot about overall linux system configuration.
I understand systemctl, but in many cases I don’t know how to pick which services are needed for my use case.
Also, are there things besides services I should consider disabling? (eg Device Tree overlays?)
Is there any system maintenance that I would need to worry about over the long term (years/decades) to keep the device reliably up and running “forever”?

Fwiw, my app does the following:

• acts as a specialized, single purpose webserver that serves up a web app on a particular address/port communicating with it using websockets

• based on interaction with the web app, sends information out some GPIO pins (in my case, installation control signals)

• when (optionally) connected to the internet, hits a live server to check for available updates (if found, download it)

• maintains the watchdog timer to self-reset in case of problems

To facilitate this, and also retain bare-bones remote access to the device (for remote troubleshooting/config), the system

• has a wifi USB dongle that creates a private network serving the web app

• fires up a DHCP server allowing easy connection on the private network

• can receive incoming ssh connections (mainly for diagnostics, setup, debugging)

• when connected to the live internet, updates a free dynamic dns server with its local ip, so can be accessed on the ‘live internet’ (assuming port forwarding has been properly set up)

Thank you in advance for any pointers you can provide,

ST

[Taceant_Omnes]

https://buildroot.org/

[yoder]

You could start with the console image[1] and then add what you need. The console has just the bare bones (so to speak) needed to get booted.

–Mark

[1] http://elinux.org/Beagleboard:BeagleBoneBlack_Debian#BBW.2FBBB.2FBBG_.28All_Revs.29

[Super_Twang]

@Mark

I’ve done what you said and started with a console image. That was a huge boost towards a minimal setup… thanks for the tip!!!

That said, even with a console image though I’m seeing lots of stuff I wouldn’t think I’d need ( using dpkg --list, and systemctl list-units --all ). These are things like support for the audio subsystem, time synchronization, dbus, ofono, avahi-daemon, etc.

I’ve been painstakingly (LOTS of out of date/irrelevant info out there to sift through) researching each item, trying to remove it, testing the system without it, and sometimes having to slowly back out my steps by restoring from a backup SD card (can take hours) when I remove something I didn’t know I needed. Its hard not to want a better way, even though I’m gaining some low-level linux knowledge by doing things this way.

I suspect there is still a fair bit more I can remove for my use case, but I don’t have a lot of reference points. Here’s one of the only references I’m working from: I was ssh’ed into a mios system recently and did a ‘ps aux’ command, and saw less than 40 procs running. With the debian console image, I’m still seeing 115 (after some manual reduction from the console image). Unfortunately I don’t know distros well enough to know if I’m comparing Apples and Oranges here.

I suppose it begs kind of a philosophical question (which I may pose in a different post)… "Do you need to remove every last unnecessary thing from a running linux distro, or is a system that works reliably, despite some wasted cpu & added complexity, “Good enough” for a commercial product? I come from an Occam’s Razor mindset (ie use only what you need), and prefer to build upwards from nothing, adding only what I need, rather than subtractively trying to remove things I don’t know whether I need.

If you (or someone else) has any insights, or any kind of validation that my painstaking process of continued subtraction is worthwhile, I’d love to hear it!

[RobertCNelson]

Just because it's installed : does not mean : it's running/taking cpu
cycles...

If you want to smart out really small, start with my "barefs":

https://eewiki.net/display/linuxonarm/BeagleBone+Black#BeagleBoneBlack-RootFileSystem(smallflash)

It's just dpkg + apt...

Regards,

[Super_Twang]

@Taceant

https://buildroot.org

That looks VERY promising and definitely along the lines of the way I’d like to think; Build up from nothing, ie additive rather than subtractive. Also, as a real bonus, it would seem to automate the creation of a cross-compilation toolchain (I’ve been doing all of my compiles on my BeagleBone Black, while my 2.8Ghz, 8-core 64x86 machine sits there jealous and idle.)

Despite its promise, however, I haven’t tried this approach yet for a couple of primary reasons:

1. because of all of the BeagleBoard-specific stuff I’m relying on – most notably the PRUs. Buildroot looks more geared towards building distros for general purpose linux machines. Am I wrong? Is anyone using buildroot for BBB-specific debian linux distros?

2. I’ve never gotten down to the kernel-building level of linux, and, to be quite honest, am a little scared of all of the options I don’t understand.

Part of this is also pragmatic – I’m on deadline. When I have some more spare cycles, I’ll dig deeper. Thanks again for the tip.

[Super_Twang]

@Robert

Thanks for the tip. Are the ‘*fs’ filesystem packages actually functionally complete OS’s? Does your barefs boot the BBB into a state into which I can ssh? In my local network setup, I believe I’d need a dhcp client on the ethernet port (I don’t control the router). Oh wait, I could probably start with a usb connection and apt-get that, couldn’t I?

I’ve seen mentions of your various ‘*fs’ (filesystems?), but, although I’m a seasoned software developer, I’m new enough at linux system administration that I don’t fully grasp the various terminology differences between the ways that parts/wholes of linux come packaged: Like: ‘file systems’, ‘kernels’, (the various kernel flavors) and ‘images’… I don’t fully grasp where one starts and the other leaves off. In other words, I wasn’t sure if the ‘*fs’ filesystems were overlays of some sort, rather than the functionally complete foundations. I had been working under the assumption that the ‘images’ were the smallest starting point, short of getting into my own kernel compilation (by which I’m a little daunted).

So, if indeed something like the barefs is functionally complete, is the following correct? A Linux installation is: a specific version of the compiled Linux kernel, paired with a set of installed packages (including hardware specific drivers, and general utilities) delivered as files and folders in a ‘filesystem’ (ie your barefs). An ‘image’ that is just a filesystem stored in a way that can be easily read/written to/from media in whole.

Best,
ST

[RobertCNelson]

@Robert

Thanks for the tip. Are the '*fs' filesystem packages actually
functionally complete OS's? Does your barefs boot the BBB into a state
into which I can ssh? In my local network setup, I believe I'd need a dhcp
client on the ethernet port (I don't control the router). Oh wait, I could
probably start with a usb connection and apt-get that, couldn't I?

The *fs's listed on eewiki.net are just a generic fs no kernel stuff
installed by default.. It's up to you do that stuff..

I've seen mentions of your various '*fs' (filesystems?), but, although I'm
a seasoned software developer, I'm new enough at linux system
administration that I don't fully grasp the various terminology differences
between the ways that parts/wholes of linux come packaged: Like: 'file
systems', 'kernels', (the various kernel flavors) and 'images'... I don't
fully grasp where one starts and the other leaves off. In other words, I
wasn't sure if the '*fs' filesystems were overlays of some sort, rather
than the functionally complete foundations. I had been working under the
assumption that the 'images' were the smallest starting point, short of
getting into my own kernel compilation (by which I'm a little daunted).

file system = generic file system (aka you set eveything up)
images = generic file system with a custom/specific kernel pre-installed
(along with bootloader) (aka i have it setup for you out of the box)

So, if indeed something like the barefs is functionally complete, is the
following correct? A Linux installation is: a specific version of the
compiled Linux kernel, paired with a set of installed packages (including
hardware specific drivers, and general utilities) delivered as files and
folders in a 'filesystem' (ie your barefs). An 'image' that is just a
filesystem stored in a way that can be easily read/written to/from media in
whole.

barefs is as bare as we could get it and still use dpkg/apt... any bit
smaller or one less package would break dpkg/apt...

it's up to you to install ssh..

Regards,

[William_Hermans]

barefs is a file system. It is a minimal file system. A file system is just that the file system. No kernel, and no boot loaders. Also there are no kernel modules( drivers ) on this file system initially.

Typically, this means you need to build your own kernel. Either that or find a kernel and driver modules all rolled up into a single archive file.

Roberts guide for this whole process is here : https://eewiki.net/display/linuxonarm/BeagleBone+Black#BeagleBoneBlack-RootFileSystem

But the above covers the whole process of building the bootloaders, and kernel( modules too ) from scratch, Then putting it all onto a given fs which was likely created with buildroot.

[Super_Twang]

@Robert [Preamble: Thank you for prepping the various images/flavors/recipes/etc I know you’re generous enough to design/maintain/share, as well as your willingness to help folks like me]

Ok good. Your clarification definitely helps! So an ‘image’ == kernel + bootloader + filesystem. Good.

Your barefs instructions pushed me farther upstream than I have been so far – hence my confusion so far. To date, I’ve been using my Mac laptop to format disk images, and bootstrapping everything on my live debian/BBB system once I’ve booted and ssh’ed into it (or connected via USB console). I have been able to swap kernels while logged into a ‘live’ bbb system ( apt-get install linux-image-* ), so, misreading your comments about having apt-get, and dpkg, I made the mistaken assumption that I might be able to do the same with the live filesystem… [ Though my mind wondered: where would my shell go? Could I get an answer for a ‘pwd’ or would it be a ‘stale file handle’? Would my system abort, and be bricked until I put in an unspoiled working sdcard? – those kinds of questions… ]

So – restating for clarity and for anyone else reading – Per your and William’s comments, and a full re-read of your wiki page, the commands in your recipe need to be executed on a (non-virtualized debian) developer machine. Following them will allow one to prep a custom image – using your ARM-compatible filesystem as one ingredient, then adding a bootloader and kernel to complete it. With that image, I can boot up the BBB, and use apt-get/dpkg to add packages from there.

The (brain) fog is starting to clear here!

Unfortunately, I think I’m missing a key requirement to use your barefs… a (non-virtualized OS) linux development machine. I have only a Mac + Vbox/Debian, and of course the several BBB’s I’m trying to configure. Could I use one of the BBBs as my ‘developer machine’? I’ve heard “Not advisable”, but don’t entirely know why … speedwise for compiles?

Or, without a non-virtualized-os linux developer machine, would you say that your pre-built console images on a live bbb remain the best place to start? ( From which to work subtractively towards the most minimal, stable/embeddable single-purpose system?
Or (if even possible) would you suggest I try a BBB as my developer machine, and give your barefs recipe a try? [ I basically need sshd, a dhcp ethernet client & pru support for a custom cape (uio_pruss version) ].

Best
ST

[Super_Twang]

@William
[Preamble: Thanks also for your clarifications here, and, all over this forum. I think I’ve learned about 90% of what I know from conversations between you, RobertCNelson, John3909 and a handful of others.]

Is there a source you’d recommend for:

find a kernel and driver modules all rolled up into a single archive file.

Kernel-wise, I’m looking specifically for 4.4.x-bone-rt-* (From what I gather I need the and to get 3.8-style uio_pruss support). I don’t know much about the bootloader I’d need (uBoot?). But basically I’d be trying to build Robert’s ‘barefs’ from the ground up to run a single-purpose server app + web app controls + driving a custom cape, in as reliable a package I can assemble. I’m still searching for a hardware watchdog solution (in another thread you’re on) to augment the RevC’s PMIC design to handle anything the power system throws at it, so the system can always rebound gracefully.

Best,
ST

[William_Hermans]

Is there a source you’d recommend for:

| find a kernel and driver modules all rolled up into a single archive file.

No, not really. Technically you can rip a kernel any associated kernel modules from any working system image. But it may not always be so simple. I think Robert used to keep modules tar’d up for various kernels, but I do not know where to look for them any more. Also, since kernel modules have to be compiled against the kernel they’re meant for . . . it makes things a bit more difficult.

Anyway, it’s probably better to start off with a console image from Robert, and just start apt-get remove --purge unneeded packages. A good way to know what to remove is to pipe the output of dpkg to a file ( dpkg -l > /path/to/file.ext )

, and start googling the package names. After a while you will know what packages are required.

You can also search the web for “reduce debian” and you’ll run into links like this: https://wiki.debian.org/ReduceDebian. So just start reading. Theres also a really good read on reducing Ubuntu, which mostly applies to Debian as well ( since Ubuntu is based on Debian ).

If you’re going to use Roberts barefs though. You’d be best off following that whole guide I linked to, and compiling your own kernel, kernel modules, and boot loaders.

[William_Hermans]

Oh, and at least one gotcha. Using barefs, you’ll either need a serial debug cable, or have the ability, and know how to setup, and use g_serial. Simply because openssh-server is not installed. So attempts to ssh in will fail.

Anyway, the very first thing one would need to do once they gain access to a terminal is to apt-get install openssh-server.