[Beaglebone] SD Card Corruption on Read Only File System

Hi,

My beaglebone’s sd card is corrupted on read only file system after 27 days of work.
Here below you can see the error:

[ 4.718902] mmcblk0: error -110 transferring data, sector 415849, nr 24, cmd response 0x900, card status 0x200b00
[ 8.636077] mmcblk0: error -110 transferring data, sector 415872, nr 1, cmd response 0x900, card status 0x0
[ 8.646362] end_request: I/O error, dev mmcblk0, sector 415872
[ 12.546051] mmcblk0: error -110 transferring data, sector 415865, nr 8, cmd response 0x900, card status 0x200b00
[ 16.453765] mmcblk0: error -110 transferring data, sector 415872, nr 1, cmd response 0x900, card status 0x0
[ 16.464050] end_request: I/O error, dev mmcblk0, sector 415872
[ 16.470703] Kernel panic - not syncing: Attempted to kill init!

The only region which I wrote is /dev/shm/ and as I know it does not related with SD Card region?
I am really shocked about SD card corruption in read-only file system.

Do you have any suggestion?

Regards

Are you unmounting the SD card before power down? If not then you should be.

Gerald

Thank you Grald,

I don’t know when will be a power down. All power downs are unwanted, unpredicted but real. I tried to protect SD cards with mounting the whole filesystem read-only.
Why SD cards broken although they used under read-only file system.

Do you have any explanation? I can not predict the real reason of corruption. If you have any suggestion with reasons it would be great.

Regards.

2013/1/9 Gerald Coley <gerald@beagleboard.org>

Ah, this looks more like the SD card's controller/nand is actually
shot, not necessarily just a corrupted file system..

Regards,

I concur with Robert, that's not a file system issue, that's an SD card
internal issue.

Buy better SD cards. Samsung and SanDisk make nice ones in 4 and 8 GB
sizes for reasonable prices.

Can you provide the entire boot output to the serial port? Are you
sure you're using read only file systems?

-Andrew

Oh, these are indeed exactly the same errors that I got from my BBone. Tried several microSD card vendors, they all went down the drain with the same error after a month or so.

I didn’t mount my cards read-only. My I/O was all taking place to a RAM disk. Normally seeing these error messages after close to a month of operation - ranging from 22 to 28 days of continous power-on (no logged power supply failures during any of these periods).

Strange and undesired.

Just my two cents to shed light on things.

This is a very very bad situation. I think there is no valid solution. I am suspecting about a hardware design or scenario problem and I will try to find a solution. If I can find a solution I will share it.

Regards

2013/1/9 Torkel M. Jodalen <la6vja@gmail.com>

özen özkaya <ozenozkaya@gmail.com> [13-01-10 08:37]:

This is a very very bad situation. I think there is no valid solution. I am
suspecting about a hardware design or scenario problem and I will try to
find a solution. If I can find a solution I will share it.

Regards

> Oh, these are indeed exactly the same errors that I got from my BBone.
> Tried several microSD card vendors, they all went down the drain with the
> same error after a month or so.
>
> I didn't mount my cards read-only. My I/O was all taking place to a RAM
> disk. Normally seeing these error messages after close to a month of
> operation - ranging from 22 to 28 days of continous power-on (no logged
> power supply failures during any of these periods).
>
> Strange and undesired.
>
> Just my two cents to shed light on things.
>
> --
> For more options, visit http://beagleboard.org/discuss
>
>
>

--
Özen Özkaya

--
For more options, visit http://beagleboard.org/discuss

Hi

from the manpage of "mount"

       -r, --read-only
              Mount the filesystem read-only. A synonym is -o ro.

              Note that, depending on the filesystem type, state and
              kernel behavior, the system may still write to the
              device. For example, Ext3 or ext4 will replay its
              journal if the filesystem is dirty. To pre‐ vent this
              kind of write access, you may want to mount ext3 or ext4
              filesystem with "ro,noload" mount options or set the
              block device to read-only mode, see command blockdev(8).

May it possible, the your system still writes to the filesystem while
slowly but shureley weare out the nand flash of the sd card.
So that finally this results in the errors?

Just a shot in the dark...

Best regards,
mcc

For anecdotes, I've been running exclusively SanDisk Ultra Mobile and
Samsung Plus microSD cards and have yet to have one die on me. I've
been running A3, A5, and A6 bones since November 2011, many of them
with syslog getting hit quite hard with no rate limit. Uptimes of a
week or more between power down/up is not uncommon (usually due to
physically having to move the bones and unplugging causes a clean
shutdown with short transfer to lithium battery).

All are running with custom 5V 2A DC sources, regulated down from 24V
AC-DC power supplies via a custom cape. Some also have lithium
batteries for clean shutdown when DC power goes away. Power might be
your problem, worth trying to fix that first.

-Andrew

I am using Ubuntu and in every case during an upgrade I have ended up with a corrupt SD card, Kingston, Sandisk, all Class 10, result is the same on -XM and Beaglebone. Sometimes fsck fixes it but eventually they fail and have to be rebuilt. No such problem on Pandaboard or ODROID-X that use full-size SD cards. Regards Sid.

Power might be your problem, worth trying to fix that first.

Yep, on that assumption I'm already trying a different type of power supply, fed via the usb rather than power port.

These are A8 bones I believe.

Lots of variables to control! Will let you know how the new power supply goes, but of course it could be a month or two before I know :slight_smile:

Jon

Well, I doubt they are A8, as A6A is the latest revision. From a power standpoint there is really little difference between USB and DC power as the both go to the same point. Other than the fact that USB power limits the current to 500mA and as such limits the processor speed to 600MHz. The PMIC uses internal FETs to select one versus the other to feed all the various regulators.

Gerald

I had the same problem, and i found a solution that helped me.

SD Cards cant handle many write cycles.
For example i was making INSERT’s into my MySQL database for more then several times per second. That was bad idea. So i changed the MySQL table type to MEMORY (table is on RAM) and when i need to save taht data (eg.before shut down) i change table type for example to MyISAM (table is in SD card).
Other thing is you should put your folders like var, tmp to RAM, because many programs uses this what leads to writing to SD card and we don’t want to do that.
Solution:

echo "tmpfs /tmp tmpfs nodev,nosuid 0 0" >> /etc/fstab echo "tmpfs /var/log tmpfs nodev,nosuid 0 0" >> /etc/fstab

With these changes my Beaglebone REV.A6 (with Robert Nelson Ubuntu 12.04) works for more than a two months with standart SD card (Kingston 4GB Calss 4) without any crashes and believe me, it performs a lot of work.

2013 m. sausio 9 d., trečiadienis 13:53:45 UTC+2, Özen Özkaya rašė:

Here is some more information:
http://www.j1nx.nl/howto-optimize-linux-sd-card-rootfs/

2013 m. sausio 11 d., penktadienis 10:18:29 UTC+2, Deimantas Žvirblis rašė:

Great information, a while ago I posted in other thread about a similar situation with some BBxM I had working remotely, running a web server and mysql service, due to the heavy load or W/R on SD card I had to replace them like every 5 days. The solution I followed was by exporting the mysql DB to a external USB HD drive, which improved the whole system and also I haven’t had to change the SD for about 4 months. Now I’m working in porting the project in the BBxM to a BB revC5 NAND which leads to no corruption at all due to the filesystem format implemented in NAND memories :slight_smile:

Yep. Works great.

Gerald

Wow, this could be a definitive solution for me, can I ask you some hint Gerald?

did you find some manual on how to prepare the NAND to boot with angstrom?
What they mean with “Note: There’s currently no software support for BeagleBone Memory Expansion Cape. This cape is sold for development purposes only.”

thanks in advance,
Luca

Someone educate me on this please - isn’t NAND what is used in SD cards, and won’t any NAND suffer the same concerns with wear levelling, bad black management, and number of write cycles?

You may be moving the direct control out of the SD card’s built in controller to somewhere else, but ultimately the issues remain and must be addressed, yes?

Just wondering.

If it helps I have been running small embedded systems using MIDE modules

and CF cards for over eleven years with very few failures. And I did not

even use a filesystem that is flash friendly (I used ext3). The filesystem

was mounted rw and I am using Debian linux with normal logging enabled to

the MIDE/CF cards.

All these tails of short life seem to be worst cases.

David

Yes David,
but Compact Flash technology is very different from SD or NAND integrated chips.
Me too I used to run systems with CF without almost any problem related to the storage for long time.

Here with 3 beaglebone A6 mounting a kingstone class10 SD 4Gb from where I run the same angstrom version given out of the box, except for some package I removed such as cloud9 and graphical stuff.
My app uses very few disk writes and with an uptime of about 85 days (that is about the time I switched them on first time). Now several files all around the filesystem are unaccessible and I can get messages like “EXT4 i/o error…” even if I try to “ls” them. I tried to reboot one of them an it never came up again.
In the office I have a 4th board installed about in the same period of the others, the only difference is the uptime. Because of it is the development board sometime I used to reboot it.
Now my tough is maybe at boot time an implicit and silent fsck is reparing time to time mistakes of the partition and the result is a well running system… uhm…

Now i just started a disk stress test on 2 boards with a fresh installation on new sd cards, one have a cronjob to reboot every 2 days the other not. Let’s see what will happen.

keep in touch
Luca