It depends...
If your target system only runs some limited subset of program, no, of course
not. What is taking up the 3GB on the SD card are things like the usuall
"zillion" utilities, libraries, and so on. Also, an embedded Linux appliance
(like your router or Blu-Ray player), is going to use BusyBox rather than have
a /bin and /usr/bin populated with all the usually little programs (ls, cp,
mv, rm, etc.). Instead, /bin, /sbin, /usr/bin and /usr/sbin will have links
to BusyBox. Basically, it is just like a Linux boot init "ramdisk" that never
bothers to mount a real "root" file system and continues to a full multi-user
boot with all the trimmings.
Next time you fire up your Beagle Bone, use the free and top commands to have
a look at what is running and how (RAM) memory is being used. For example, my
Raspberry Pi fruitloops is currently using just under 1 GB of RAM, running
about 100 processes:
Tasks: 109 total, 1 running, 60 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 948308 total, 908924 used, 39384 free, 142516 buffers
KiB Swap: 102396 total, 21248 used, 81148 free. 138932 cached Mem
It is also using 21GB of a 32GB SD card. This is Raspbian, and includes a
pretty typical Linux multi-user system, eg systemd, ssh server, X11, a
compilete native G++ toolchain and libraries, with multiple cross-build G++
toolchain and libraries, for Arduino (arm, avr, xtensa) and OpenMRN (arm),
along with a full LaTeX install, Doxygen, etc. -- this is my general purpose
build box for both native Raspbian (and also Beagle Debian targets), and for
various little MCUs: Avr, Arm (samd), and esp32 Arduino supported, and various
ARM-based MCUs supported by OpenMRN.