Possible eMMC firmware bug or hw issue - recent Seeed Studio BBBs with 6.1.x kernels

Thank you very much for your work and information.
As I understand it is only an issue of kernel 6.1, so nothing to do with 5.15 kernels?
Best Regards,

Eh, This only affects BBBs or is my BVF going away to the graveyard in weeks also?

What about all the derivatives, like BBG(W), Blue and friends?

one more question:
How-to install different kernel without access to internet?
I am able to ssh to the BBB and upload files with scp. But the BBB has no direct access to internet for using apt install e.g.
sudo apt update
sudo apt install bbb.io-kernel-5.10-ti-am335x
Regards,

With MK2704 Kingston 4gb eMMC

You’re doing product development wrong. Connect the board to the Internet, install what you need, test, repeat. Then remove it from the Internet.

… and a big sigh of relief was heard across the land …

Mine has a Micron device of some description.

Hi Robert,

in january I decided to go with kernel 6.1 and debian 12. Unfortunately I have not seen the discussion around problems with emmc. During my tests everything worked fine. I have no active writing activities to the emmc, so problems seem to arise later.
Only the debian internal system seems to write some changes. On previous versions I never had an issue with the emmc.
After some month I observed after a powercycle the BBB won’t start again.
For some BBBs (approx 600 km away from home ;-( ) without direct internet connection my wish was to prepare a tgz file or s.th. equivalent and bring it via scp to that BBB and switch to a 5.10 kernel to avoid crashing the emmc’s.
Here at home I just tried copying all kernel specific files in /boot/ to a target BBB and changed the ‘uname=…’ in /boot/uEnv.txt and rebooted. Seemed to work.
But I am not sure if other problems will come up.
Which would be the best way, if no help is available at the other location?

Thank You again!
Best Regards,

All of our serious work is air gapped from the “public internet”. Targets live on their “mock” internet environment. You need a metal box with dual internet ports and a server on the air-gap side. You can use apt and dpkg commands as well as git mirror to set every thing up. When they go on public internet for live testing the targets are under a microscope.

Typically we now use yocto builds so its already installed and configured. This all depends on the many issues and project requirements.

Just grab the matching linux-image-(version).deb you need, then upload it and install via dpkg on your external device. https://rcn-ee.net/repos/debian/pool/main/l/linux-upstream/

Example https://rcn-ee.net/repos/debian/pool/main/l/linux-upstream/linux-image-5.10.168-ti-r78_1bookworm_armhf.deb

Thank you! Great answer!
That’s what I needed, :wink:
Regards,

eMMC name: MK2704
eMMC date: 08/2023
eMMC rev: 0x7
eMMC hwrev: 0x0
eMMC fwrev: 0x0100000000000000
eMMC oemid: 0x0100
eMMC manfid: 0x000070
eMMC life_time: 0x02 0x01
eMMC serial: 0x1f15007b

Crap, life_time just turned from 0x01 → 0x02…

Got over a week out of it… with MMC_QUIRK_BROKEN_SD_DISCARD

MMC_QUIRK_TRIM_BROKEN - eMMC long term failed
MMC_QUIRK_BROKEN_SD_DISCARD - eMMC failing..

Okay, grabs another fresh bbb… and lets try the next flag combination…

Regards,

1 Like

Robert ‘BBB killer’ Nelson… :wink:

1 Like

What’s nice, that board that did fail, had a phy bug, so i was able to verify a new board with new gpio reset, and that the updated gpio reset fixed it… So add it my ci, as a phy tester…

Okay, new board up and running… 6.10.0-bone6

eMMC name: MK2704
eMMC date: 08/2023
eMMC rev: 0x7
eMMC hwrev: 0x0
eMMC fwrev: 0x0100000000000000
eMMC oemid: 0x0100
eMMC manfid: 0x000070
eMMC life_time: 0x01 0x01
eMMC serial: 0x1f5420d9

Trying: MMC_QUIRK_TRIM_BROKEN | MMC_QUIRK_BROKEN_CACHE_FLUSH

+	/*
+	 * Kingston EMMC04G-MK27 are showing fast wear rates
+	 * 2024???? : testing MMC_QUIRK_TRIM_BROKEN (eMMC died)
+	 * 20240708 : testing MMC_QUIRK_BROKEN_SD_DISCARD (eMMC failing)
+	 * 20240717 : testing MMC_QUIRK_TRIM_BROKEN | MMC_QUIRK_BROKEN_CACHE_FLUSH (testing)
+	 */
+	MMC_FIXUP("MK2704", CID_MANFID_KINGSTON, 0x0100, add_quirk_mmc,
+		  MMC_QUIRK_TRIM_BROKEN | MMC_QUIRK_BROKEN_CACHE_FLUSH),
+

Regards,

Sadly, got distracted by grief/life, so i missed catching the failure…

[   14.571349] mmc1: Card stuck being busy! __mmc_poll_for_busy
[   14.775700] mmc1: cache flush error -110
[   15.819799] mmc1: Card stuck being busy! __mmc_poll_for_busy
[   15.825589] mmc1: tried to HW reset card, got error -110
[   15.830937] mmcblk1: recovery failed!
[   15.834658] I/O error, dev mmcblk1, sector 8192 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
[   15.843789] Buffer I/O error on dev mmcblk1p1, logical block 0, lost async page write
[   15.851722] Buffer I/O error on dev mmcblk1p1, logical block 1, lost async page write

Another BBB eMMC bites the dust… so let’s try more quirks…

Regards,

Next board for testing…

eMMC name: MK2704
eMMC date: 04/2023
eMMC rev: 0x7
eMMC hwrev: 0x0
eMMC fwrev: 0x0100000000000000
eMMC oemid: 0x0100
eMMC manfid: 0x000070
eMMC life_time: 0x01 0x01
eMMC serial: 0x59923d72
+	/*
+	 * Kingston EMMC04G-MK27 are showing fast wear rates
+	 * 2024???? : testing MMC_QUIRK_TRIM_BROKEN (eMMC died)
+	 * 20240708 : testing MMC_QUIRK_BROKEN_SD_DISCARD (eMMC failing)
+	 * 20240717 : testing MMC_QUIRK_TRIM_BROKEN | MMC_QUIRK_BROKEN_CACHE_FLUSH (eMMC died)
+	 * 20240808 : testing MMC_QUIRK_TRIM_BROKEN | MMC_QUIRK_BROKEN_SD_DISCARD | MMC_QUIRK_BROKEN_CACHE_FLUSH | MMC_QUIRK_ERASE_BROKEN (eMMC failing)
+	 * 20240826 : testing MMC_QUIRK_TRIM_BROKEN | MMC_QUIRK_BROKEN_SD_CACHE
+	 */
+	MMC_FIXUP("MK2704", CID_MANFID_KINGSTON, 0x0100, add_quirk_mmc,
+		  MMC_QUIRK_TRIM_BROKEN | MMC_QUIRK_BROKEN_SD_CACHE),
+

Testing… 6.11.0-rc5-bone3

Regards,

You’re going through an awful lot of BBB’s, aren’t you?

and another just bit the dust!

eMMC life_time: 0x02 0x01
eMMC serial: 0x59923d72

I don’t know what scares me more,

You, still searching for the right combination of quirks,
or the fact that you still haven’t run out of BBBs to test destructively. :smiling_face:

I hope you’re getting sponsored or at least you’re able to rework
new eMMCs onto those BBBs.