Weird behaviour: input/output error and no commands after few minutes

GoogleGroups
1

Hi there, new to Beaglebone Blue but not happy so far.

Using newest image bone-debian-8.7-iot-armhf-2017-03-19-4gb
Flashing to MMC, working fine.
Configuring wifi to my local AP in /etc/network/interfaces
Running apt-get update, apt-get upgrade without problems.
Rebooting from that, all is ok, connecting wlan0 automatically

But after a while, without myself being able to pinpoint why, the terminal on 192.168.7.2 starts to behave strangely.
I cannot run commands, they only return “input/output error”
Cloud9 and NodeRed returns “Cannot / GET” in the browser and there is only one way which is a powercycle.

How do I debug that?

2

Get, and connect a serial debug cable. Connect to your support PC, use puTTY or another serial client software to view the kernel output as it’s running, and what errors are being displayed, when this happens.

3

That makes sense, will try that.

But are there no log-file where this kernel output is stored?

4

Well it could be, but I do not think Robert persists logs between reboots.
Maybe I'm wrong. So then assuming if what I say is true, you'd have to hope
the error is actually logged. Then the problem is that if you lose all
connectivity, how do you propose to see that information ?

With a serial debug cable, you'll be directly connected to a tty terminal,
so networking issues will not effect that. Which means not only can you get
additional debug info, but you can also log into the system, for further
troubleshooting if needed.

You can try to use:
root@wgd:~# more /var/log/messages
or
root@wgd:~# more /var/log/kern.log

But I do not think either of these will do you any good. You can check to
make sure though. Do keep in mind there will be a lot of info in there that
does not have anything to do with what you're looking for. So you may run
out of time before you get done reading through the files.

However, if you have another Linux machine, you could scp those files to a
remote system. If you are using Windows, there is a program called WinSCP I
believe that should allow you to traverse into the /var/log directory, and
copy those files out to your desktop.

5

A couple of other things you could try.

  1. Occasionally installing Linux to an sdcard for this purpose can fail silently. You could try to re-image the sdcard in hopes that this might be the case in your situation.

  2. Pick a slightly newer, or slightly older sdcard image, install that. Then see if the problem persists.

6

Will try that if attaching the cable does not work…

Now comes the noob question: How do I attach the cable, seems that there is plenty BeagleBone Black tutorials on that, but not sure how to do it on the blue one.

7

On Thu, 18 May 2017 06:01:49 -0700 (PDT), Niels Jakob Buch
<njbuch@gmail.com> declaimed the following:

Will try that if attaching the cable does not work....

Now comes the noob question: How do I attach the cable, seems that there is
plenty BeagleBone Black tutorials on that, but not sure how to do it on the
blue one.

  Based on the spec sheet, you'll likely have to wire the interface to a
4-pin JST-SH connector, since UART0 appears to be the fourth socket on the
outside long edge of the card.

  I can't tell if the card has male or female connectors, so I don't know
if it may be possible to just run bare jumpers into the connector.

8

UART0 looks to be tied to a 4pin JST-SH connector. How exactly that works out, you’ll have to tell me. I do not own a beaglebone blue. Software wise though, it should be exactly the same.

9

Argh, waiting on a pile of JST-XX connectors to get the Blue-world up and running. But thanks!

10

So, I actually have no idea of the pin pitch on those JST headers. But the
serial debug cable I bought years ago, and still use. Has fairly thin slip
on connectors. You only need RXD, TXD, and ground connected. The cable I
bought was a Prolific PL2303HX cable very similar to the one stocked at
Adafruit https://www.adafruit.com/product/954

I actually bought mine off ebay for less than $3, but shipping took a lot
longer too. Adafruit is stating on their page there that they're now using
SilLabs something or another because of driver stability. But I've never
experienced a problem with the PL2303HX cable.

Anyway, my point is that you may not need a JST connector for this purpose.

11

I just checked and the JST-SH connector pins, inside the terminal are only 1mm apart. https://is.alicdn.com/img/pb/863/022/524/524022863_862.jpg

I guess I will wait for the wire assemblies.

12

Any update on this problem. I ordered a beaglebone blue from Arrow and I’m having the same problems. Writing/reading to the eMMC just randomly fails. They sent me another one, this ones a little better… but eventually i get input/output errors. I thought it was the power supply maybe at first, but I hooked them up to a known googd power supply that’s 12V 5A (60w) and it still does this.

13

I also am having the same issue. I thought it was broken hardware so I bought a new one and the same thing happened again. I found that I can usually induce the errors by transferring a large amount of files using scp.

I’m going to check UART0 for additional information once I get the cabling set up. I really hope this gets patched soon, it’s a very irritating bug.

One time I was able to get some debug information before the I/O errors consumed everything, not sure if it’s helpful in any way.

Here’s the last few lines given from dmesg

[14188.582576] wlan0: RX AssocResp from 06:18:d6:57:8c:90 (capab=0x431 status=0 aid=2)
[14188.593611] wlan0: associated
[14188.641133] wlcore: Association completed.
[25043.096569] hrtimer: interrupt took 136276 ns
[30864.563462] mmcblk1: error -110 sending status command, retrying
[30864.597101] mmcblk1: error -110 sending status command, retrying
[30864.638167] mmcblk1: error -110 sending status command, aborting
[30864.644451] blk_update_request: I/O error, dev mmcblk1, sector 3487760
[30864.651172] blk_update_request: I/O error, dev mmcblk1, sector 3487768
[30864.657840] blk_update_request: I/O error, dev mmcblk1, sector 3487776
[30864.664510] blk_update_request: I/O error, dev mmcblk1, sector 3487784
[30864.671146] blk_update_request: I/O error, dev mmcblk1, sector 3487792
[30864.677767] blk_update_request: I/O error, dev mmcblk1, sector 3487800
[30864.684382] blk_update_request: I/O error, dev mmcblk1, sector 3487808
[30864.690996] blk_update_request: I/O error, dev mmcblk1, sector 3487816
[30864.697652] blk_update_request: I/O error, dev mmcblk1, sector 3487824
[30864.708291] Aborting journal on device mmcblk1p1-8.
[30864.769510] mmcblk1: error -110 sending status command, retrying
[30864.817684] mmcblk1: error -110 sending status command, retrying
[30864.865245] mmcblk1: error -110 sending status command, aborting
[30864.871498] blk_update_request: I/O error, dev mmcblk1, sector 8192
[30864.877900] Buffer I/O error on dev mmcblk1p1, logical block 0, lost sync page write
[30864.886607] EXT4-fs error (device mmcblk1p1): ext4_journal_check_start:56: Detected aborted journal
[30864.895940] EXT4-fs (mmcblk1p1): Remounting filesystem read-only
[30864.902042] EXT4-fs (mmcblk1p1): previous I/O error to superblock detected
[30865.101035] mmcblk1: error -110 sending status command, retrying
[30865.115710] mmcblk1: error -110 sending status command, retrying

And here are the last few lines of /var/log/syslog

Aug 4 20:37:33 beaglebone connmand[794]: ntp: time slew -0.003904 s
Aug 4 20:37:33 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:39:03 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:39:04 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:40:34 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:40:35 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:42:05 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:42:14 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:43:44 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:43:56 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:45:26 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:45:27 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:46:57 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:46:57 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:48:27 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:48:32 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:50:02 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:50:13 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:51:43 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:51:47 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:53:17 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:53:17 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:54:47 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:54:37 beaglebone connmand[794]: ntp: time slew -0.010474 s
Aug 4 20:54:47 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:56:17 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:56:17 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:57:47 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:57:47 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 20:59:17 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 20:59:22 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:00:52 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:01:02 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:02:32 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:02:32 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:04:02 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:04:03 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:05:33 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:05:33 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:07:03 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:07:03 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:08:33 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:08:33 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:10:03 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:10:12 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:11:42 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:11:41 beaglebone connmand[794]: ntp: time slew -0.009679 s
Aug 4 21:11:51 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:13:21 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:13:22 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:14:52 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:14:53 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:16:23 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:16:35 beaglebone rsyslogd-2007: action ‘action 17’ suspended, next retry is Fri Aug 4 21:18:05 2017 [try http://www.rsyslog.com/e/2007 ]
Aug 4 21:17:01 beaglebone CRON[4839]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Aug 4 21:17:01 beaglebone kernel: [30864.563462] mmcblk1: error -110 sending status command, retrying
Aug 4 21:17:01 beaglebone kernel: [30864.597101] mmcblk1: error -110 sending status command, retrying
Aug 4 21:17:01 beaglebone kernel: [30864.638167] mmcblk1: error -110 sending status command, aborting
Aug 4 21:17:01 beaglebone kernel: [30864.644451] blk_update_request: I/O error, dev mmcblk1, sector 3487760
Aug 4 21:17:01 beaglebone kernel: [30864.651172] blk_update_request: I/O error, dev mmcblk1, sector 3487768
Aug 4 21:17:01 beaglebone kernel: [30864.657840] blk_update_request: I/O error, dev mmcblk1, sector 3487776
Aug 4 21:17:01 beaglebone kernel: [30864.664510] blk_update_request: I/O error, dev mmcblk1, sector 3487784
Aug 4 21:17:01 beaglebone kernel: [30864.671146] blk_update_request: I/O error, dev mmcblk1, sector 3487792
Aug 4 21:17:01 beaglebone kernel: [30864.677767] blk_update_request: I/O error, dev mmcblk1, sector 3487800
Aug 4 21:17:01 beaglebone kernel: [30864.684382] blk_update_request: I/O error, dev mmcblk1, sector 3487808
Aug 4 21:17:01 beaglebone kernel: [30864.690996] blk_update_request: I/O error, dev mmcblk1, sector 3487816
Aug 4 21:17:01 beaglebone kernel: [30864.697652] blk_update_request: I/O error, dev mmcblk1, sector 3487824

14

I don’t have my logs with me, but I believe mine looked pretty much just like that. Unfortunately both of mine spits out errors just trying to update Debian. Using an SD Card delays the error a little longer, but eventually it gets some disk access errors and goes into read only mode. I wonder if there is a way to slow down read/write and if that would fix it. I mean, besides updating it I don’t really care about writing much after that.

I contacted the RMA email on the beagleboard site, and Arrow. Never heard back from beagleboard. Arrow said they would have the RMA people contact me, been a week already. I bought this for testing until their new X15 comes out, but at this rate I think I’ll wait a major revision of X15 after it first comes out.

Jeff

15

Same problems on new Beagle Bone Blue.
Unpacked, powered up, worked like 5 minutes until IO errors.

Kind regards,

Bm

16

I have tried:

  • 4 distinct power supplies (3 x 5V, 1x 12V)
  • SD card and built in memory
  • 3 different OS images

– latest ‘iot’ image from Latest Software Images - BeagleBoard
– image referenced in BeagleBoneBlue FAQ, https://debian.beagleboard.org/images/bone-debian-8.7-iot-armhf-2017-03-19-4gb.img.xz.
– image referenced on Straswon Design, https://rcn-ee.com/rootfs/bb.org/testing/2017-03-07/iot/bone-debian-8.7-iot-armhf-2017-03-07-4gb.img.xz

Nothing helped. The input/output errors show usually after some writes to mmc and dmesg is flooded with errors.

I have also found on StrawsonDesign github this thread: Read-only File System Error · Issue #87 · beagleboard/librobotcontrol · GitHub and StrawsonDesign mentioning that:

I’ve found this on about 10 out of 70 beaglebone blues, many of those boards also show flash errors when running badblocks. I think it’s a hardware issue with the emmc as I don’t see it when running off an SD card.

In my case it also doesn’t work with SD cards. I concluded that my BBBL is faulty and returned it.

Kind regards

17

I have the same problem. I ordered two of those but both of them have I/O error.
This happens with the eMMC and the SD card as well.
dmesg shows memory block errors.

Is it the hardware? Did I get 2 out of those 10 faulty ones?

18

Just received BeagleBone Blue and had the same issue as described here when trying to perform a distribution upgrade, which led me here.

My steps to recreate the problem:-

  1. I created this image https://rcn-ee.net/rootfs/bb.org/testing/2019-03-24/stretch-console/bone-debian-9.8-console-armhf-2019-03-24-1gb.img.xz on 16GB SD card using https://etcher.io/
  2. Booted beagle using the new image on SD card
  3. ssh to the beaglebone blue
  4. enabled wifif to connect to my wifi
  5. sudo apt-get -y update
  6. sudo apt-get -y dist-upgrade

After running step 6 it fails with input\output error after about 5-10 seconds.

``Did anyone have any success on finding a solution?
How did you guys get on when receiving a new BeagleBone blue.

after this point the device is servilely hindered as not many commands work

19

The company I ordered from replaced it. Their guess was it had bad memory somewhere, or some other faulty component. Replacement had the same issue. Gave up after that.

20

I think I managed to get past this issue. I tried another SD card, it was a really old one I had lying around, 2GB in size. I’m pretty sure there was nothing wrong with the previous 16GB card as it was used in my action camera. Fsck also found no errors.
Using the 2GB card I went through the same process as when using the 16GB card. This time round it went through the distribution upgrade procedure and seems to be working ok.
I no face another problem with arducopter but that’s for another forum.

Maybe it’s the issue is something to do with the SD cards class?