Hi Robert,
thank your for your fast response.
If you can get us a serial log, it would help shine more light on the
actual issue…
Actually, when the board froze, there wasn’t much to see. However, I found some interesting entries in kern and syslog:
kern.log had a lot these:
Oct 4 14:31:11 omap kernel: [30064.859466] smsc95xx 1-2.1:1.0: eth0: kevent 2 may have been dropped
syslog reported a stack exception:
Oct 5 14:32:51 omap kernel: [ 3647.893157] swapper: page allocation failure: order:3, mode:0x4020
Oct 5 14:32:51 omap kernel: [ 3647.893249] [] (unwind_backtrace+0x0/0xf8) from [] (warn_alloc_failed+0xc4/0x114)
Oct 5 14:32:51 omap kernel: [ 3647.893249] [] (warn_alloc_failed+0xc4/0x114) from [] (__alloc_pages_nodemask+0x4e4/0x72c)
Oct 5 14:32:51 omap kernel: [ 3647.893280] [] (__alloc_pages_nodemask+0x4e4/0x72c) from [] (__get_free_pages+0x10/0x24)
Oct 5 14:32:51 omap kernel: [ 3647.893310] [] (__get_free_pages+0x10/0x24) from [] (kmalloc_order_trace+0x24/0x8c)
Oct 5 14:32:51 omap kernel: [ 3647.893341] [] (kmalloc_order_trace+0x24/0x8c) from [] (__alloc_skb+0x54/0xe0)
Oct 5 14:32:51 omap kernel: [ 3647.893371] [] (__alloc_skb+0x54/0xe0) from [] (rx_submit+0x24/0x1e4)
Oct 5 14:32:51 omap kernel: [ 3647.893371] [] (rx_submit+0x24/0x1e4) from [] (usb_hcd_giveback_urb+0x64/0xf8)
Oct 5 14:32:51 omap kernel: [ 3647.893402] [] (usb_hcd_giveback_urb+0x64/0xf8) from [] (ehci_urb_done+0x9c/0xb0)
Oct 5 14:32:51 omap kernel: [ 3647.893432] [] (ehci_urb_done+0x9c/0xb0) from [] (qh_completions+0x248/0x454)
Oct 5 14:32:51 omap kernel: [ 3647.893463] [] (qh_completions+0x248/0x454) from [] (ehci_work+0xd0/0xa88)
Oct 5 14:32:51 omap kernel: [ 3647.893463] [] (ehci_work+0xd0/0xa88) from [] (ehci_irq+0x27c/0x29c)
Oct 5 14:32:51 omap kernel: [ 3647.893493] [] (ehci_irq+0x27c/0x29c) from [] (usb_hcd_irq+0x34/0x84)
Oct 5 14:32:51 omap kernel: [ 3647.893524] [] (usb_hcd_irq+0x34/0x84) from [] (handle_irq_event_percpu+0x68/0x230)
Oct 5 14:32:51 omap kernel: [ 3647.893554] [] (handle_irq_event_percpu+0x68/0x230) from [] (handle_irq_event+0x40/0x60)
Oct 5 14:32:51 omap kernel: [ 3647.893585] [] (handle_irq_event+0x40/0x60) from [] (handle_level_irq+0xb0/0x13c)
Oct 5 14:32:51 omap kernel: [ 3647.893615] [] (handle_level_irq+0xb0/0x13c) from [] (generic_handle_irq+0x30/0x40)
Oct 5 14:32:51 omap kernel: [ 3647.893615] [] (generic_handle_irq+0x30/0x40) from [] (asm_do_IRQ+0x58/0xac)
Oct 5 14:32:51 omap kernel: [ 3647.893676] [] (asm_do_IRQ+0x58/0xac) from [] (__irq_svc+0x38/0x100)
Oct 5 14:32:51 omap kernel: [ 3647.893676] Exception stack(0xc086ff20 to 0xc086ff68)
Oct 5 14:32:51 omap kernel: [ 3647.893707] ff20: 00000000 00060db8 00060db8 00000000 c08e1d74 c0daa260 00000003 c0daa260
Oct 5 14:32:51 omap kernel: [ 3647.893707] ff40: 00000003 c08e027c c0daa260 00000000 00000000 c086ff68 c007bccc c007bcd4
Oct 5 14:32:51 omap kernel: [ 3647.893737] ff60: 600d0053 ffffffff
Oct 5 14:32:51 omap kernel: [ 3647.893737] [] (__irq_svc+0x38/0x100) from [] (omap3_enter_idle+0xb4/0x128)
Oct 5 14:32:51 omap kernel: [ 3647.893768] [] (omap3_enter_idle+0xb4/0x128) from [] (cpuidle_idle_call+0xd4/0x24c)
Oct 5 14:32:51 omap kernel: [ 3647.893798] [] (cpuidle_idle_call+0xd4/0x24c) from [] (cpu_idle+0xc0/0xf4)
Oct 5 14:32:51 omap kernel: [ 3647.893829] [] (cpu_idle+0xc0/0xf4) from [] (start_kernel+0x324/0x330)
Oct 5 14:32:51 omap kernel: [ 3647.893859] [] (start_kernel+0x324/0x330) from [<80008044>] (0x80008044)
Oct 5 14:32:51 omap kernel: [ 3647.893859] Mem-info:
Oct 5 14:32:51 omap kernel: [ 3647.893890] Normal per-cpu:
Oct 5 14:32:51 omap kernel: [ 3647.893890] CPU 0: hi: 186, btch: 31 usd: 62
Oct 5 14:32:51 omap kernel: [ 3647.893920] active_anon:7661 inactive_anon:112 isolated_anon:0
Oct 5 14:32:51 omap kernel: [ 3647.893920] active_file:27561 inactive_file:31379 isolated_file:7
Oct 5 14:32:51 omap kernel: [ 3647.893920] unevictable:0 dirty:3841 writeback:3893 unstable:0
Oct 5 14:32:51 omap kernel: [ 3647.893920] free:47467 slab_reclaimable:3913 slab_unreclaimable:1610
Oct 5 14:32:51 omap kernel: [ 3647.893920] mapped:1802 shmem:136 pagetables:229 bounce:0
Oct 5 14:32:51 omap kernel: [ 3647.893981] Normal free:189868kB min:16384kB low:20480kB high:24576kB active_anon:30644kB inactive_anon:448kB active_file:110244kB inactive_file:125516kB unevictable:0kB isolated(anon):0kB isolated(file):28kB present:501760kB mlocked:0kB dirty:15364kB writeback:15572kB mapped:7208kB shmem:544kB slab_reclaimable:15652kB slab_unreclaimable:6440kB kernel_stack:728kB pagetables:916kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 5 14:32:51 omap kernel: [ 3647.894012] lowmem_reserve: 0 0 0
Oct 5 14:32:51 omap kernel: [ 3647.894012] Normal: 289554kB 90608kB 216kB 032kB 064kB 0128kB 0256kB 1512kB 11024kB 02048kB 04096kB = 189868kB
Oct 5 14:32:51 omap kernel: [ 3647.894073] 59081 total pagecache pages
Oct 5 14:32:51 omap kernel: [ 3647.894073] 0 pages in swap cache
Oct 5 14:32:51 omap kernel: [ 3647.894073] Swap cache stats: add 0, delete 0, find 0/0
Oct 5 14:32:51 omap kernel: [ 3647.894104] Free swap = 0kB
Oct 5 14:32:51 omap kernel: [ 3647.894104] Total swap = 0kB
Oct 5 14:32:51 omap kernel: [ 3647.911224] 131072 pages of RAM
Oct 5 14:32:51 omap kernel: [ 3647.911224] 47739 free pages
Oct 5 14:32:51 omap kernel: [ 3647.911224] 8637 reserved pages
Oct 5 14:32:51 omap kernel: [ 3647.911254] 5169 slab pages
Oct 5 14:32:51 omap kernel: [ 3647.911254] 51757 pages shared
Oct 5 14:32:51 omap kernel: [ 3647.911254] 0 pages swap cached
Oct 5 14:32:51 omap kernel: [ 3647.911346] swapper: page allocation failure: order:3, mode:0x4020
Oct 5 14:32:51 omap kernel: [ 3647.911407] [] (unwind_backtrace+0x0/0xf8) from [] (warn_alloc_failed+0xc4/0x114)
Oct 5 14:32:51 omap kernel: [ 3647.911437] [] (warn_alloc_failed+0xc4/0x114) from [] (__alloc_pages_nodemask+0x4e4/0x72c)
Oct 5 14:32:51 omap kernel: [ 3647.911468] [] (__alloc_pages_nodemask+0x4e4/0x72c) from [] (__get_free_pages+0x10/0x24)
Oct 5 14:32:51 omap kernel: [ 3647.911468] [] (__get_free_pages+0x10/0x24) from [] (kmalloc_order_trace+0x24/0x8c)
Oct 5 14:32:51 omap kernel: [ 3647.911499] [] (kmalloc_order_trace+0x24/0x8c) from [] (__alloc_skb+0x54/0xe0)
Oct 5 14:32:51 omap kernel: [ 3647.911529] [] (__alloc_skb+0x54/0xe0) from [] (rx_submit+0x24/0x1e4)
Oct 5 14:32:51 omap kernel: [ 3647.911560] [] (rx_submit+0x24/0x1e4) from [] (usb_hcd_giveback_urb+0x64/0xf8)
Oct 5 14:32:51 omap kernel: [ 3647.911590] [] (usb_hcd_giveback_urb+0x64/0xf8) from [] (ehci_urb_done+0x9c/0xb0)
Oct 5 14:32:51 omap kernel: [ 3647.911590] [] (ehci_urb_done+0x9c/0xb0) from [] (qh_completions+0x248/0x454)
Oct 5 14:32:51 omap kernel: [ 3647.911621] [] (qh_completions+0x248/0x454) from [] (ehci_work+0xd0/0xa88)
Oct 5 14:32:51 omap kernel: [ 3647.911651] [] (ehci_work+0xd0/0xa88) from [] (ehci_irq+0x27c/0x29c)
Oct 5 14:32:51 omap kernel: [ 3647.911651] [] (ehci_irq+0x27c/0x29c) from [] (usb_hcd_irq+0x34/0x84)
Oct 5 14:32:51 omap kernel: [ 3647.911682] [] (usb_hcd_irq+0x34/0x84) from [] (handle_irq_event_percpu+0x68/0x230)
Oct 5 14:32:51 omap kernel: [ 3647.911712] [] (handle_irq_event_percpu+0x68/0x230) from [] (handle_irq_event+0x40/0x60)
Oct 5 14:32:51 omap kernel: [ 3647.911743] [] (handle_irq_event+0x40/0x60) from [] (handle_level_irq+0xb0/0x13c)
Oct 5 14:32:51 omap kernel: [ 3647.911773] [] (handle_level_irq+0xb0/0x13c) from [] (generic_handle_irq+0x30/0x40)
Oct 5 14:32:51 omap kernel: [ 3647.911804] [] (generic_handle_irq+0x30/0x40) from [] (asm_do_IRQ+0x58/0xac)
Oct 5 14:32:51 omap kernel: [ 3647.911834] [] (asm_do_IRQ+0x58/0xac) from [] (__irq_svc+0x38/0x100)
Oct 5 14:32:51 omap kernel: [ 3647.911834] Exception stack(0xc086ff20 to 0xc086ff68)
Oct 5 14:32:51 omap kernel: [ 3647.911865] ff20: 00000000 00060db8 00060db8 00000000 c08e1d74 c0daa260 00000003 c0daa260
Oct 5 14:32:51 omap kernel: [ 3647.911865] ff40: 00000003 c08e027c c0daa260 00000000 00000000 c086ff68 c007bccc c007bcd4
Oct 5 14:32:51 omap kernel: [ 3647.911895] ff60: 600d0053 ffffffff
Oct 5 14:32:51 omap kernel: [ 3647.911895] [] (__irq_svc+0x38/0x100) from [] (omap3_enter_idle+0xb4/0x128)
Oct 5 14:32:51 omap kernel: [ 3647.911926] [] (omap3_enter_idle+0xb4/0x128) from [] (cpuidle_idle_call+0xd4/0x24c)
Oct 5 14:32:51 omap kernel: [ 3647.911956] [] (cpuidle_idle_call+0xd4/0x24c) from [] (cpu_idle+0xc0/0xf4)
Oct 5 14:32:51 omap kernel: [ 3647.911987] [] (cpu_idle+0xc0/0xf4) from [] (start_kernel+0x324/0x330)
Oct 5 14:32:51 omap kernel: [ 3647.912017] [] (start_kernel+0x324/0x330) from [<80008044>] (0x80008044)
Oct 5 14:32:51 omap kernel: [ 3647.912017] Mem-info:
Oct 5 14:32:51 omap kernel: [ 3647.912017] Normal per-cpu:
Oct 5 14:32:51 omap kernel: [ 3647.912048] CPU 0: hi: 186, btch: 31 usd: 62
Oct 5 14:32:51 omap kernel: [ 3647.912048] active_anon:7661 inactive_anon:112 isolated_anon:0
Oct 5 14:32:51 omap kernel: [ 3647.912078] active_file:27561 inactive_file:31379 isolated_file:7
Oct 5 14:32:51 omap kernel: [ 3647.912078] unevictable:0 dirty:3841 writeback:3893 unstable:0
Oct 5 14:32:51 omap kernel: [ 3647.912078] free:47467 slab_reclaimable:3913 slab_unreclaimable:1610
Oct 5 14:32:51 omap kernel: [ 3647.912078] mapped:1802 shmem:136 pagetables:229 bounce:0
Oct 5 14:32:51 omap kernel: [ 3647.912109] Normal free:189868kB min:16384kB low:20480kB high:24576kB active_anon:30644kB inactive_anon:448kB active_file:110244kB inactive_file:125516kB unevictable:0kB isolated(anon):0kB isolated(file):28kB present:501760kB mlocked:0kB dirty:15364kB writeback:15572kB mapped:7208kB shmem:544kB slab_reclaimable:15652kB slab_unreclaimable:6440kB kernel_stack:728kB pagetables:916kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 5 14:32:51 omap kernel: [ 3647.912170] lowmem_reserve[]: 0 0 0
Oct 5 14:32:51 omap kernel: [ 3647.912170] Normal: 289554kB 90608kB 216kB 032kB 064kB 0128kB 0256kB 1512kB 11024kB 02048kB 04096kB = 189868kB
Oct 5 14:32:51 omap kernel: [ 3647.912231] 59081 total pagecache pages
Oct 5 14:32:51 omap kernel: [ 3647.912231] 0 pages in swap cache
Oct 5 14:32:51 omap kernel: [ 3647.912231] Swap cache stats: add 0, delete 0, find 0/0
Oct 5 14:32:51 omap kernel: [ 3647.912231] Free swap = 0kB
Oct 5 14:32:51 omap kernel: [ 3647.912261] Total swap = 0kB
Oct 5 14:32:51 omap kernel: [ 3647.929260] 131072 pages of RAM
Oct 5 14:32:51 omap kernel: [ 3647.929290] 47739 free pages
Oct 5 14:32:51 omap kernel: [ 3647.929290] 8637 reserved pages
Oct 5 14:32:51 omap kernel: [ 3647.929290] 5169 slab pages
Oct 5 14:32:51 omap kernel: [ 3647.929290] 51757 pages shared
Oct 5 14:32:51 omap kernel: [ 3647.929290] 0 pages swap cached
Oct 5 14:32:51 omap kernel: [ 3647.929321] smsc95xx 1-2.1:1.0: eth0:
and in addition a lot of:
kevent 2 may have been dropped
As a quick workaround to test, what i think is going on…
Under load it’s probably the smsc95xx chipset running out of kernel
memory, bump: 8192 → 16384 in /etc/sysctl.conf
I’ve run some extensive test with various settings and it turned out that increasing this value eases the problem. With the suggested 16384
the board didn’t freeze anymore, however, I got some “path not found” abort messages on my windows client during copy.
With a setting of 65536, I was able to move a 15Gig file under full load without problems.
For my use-case for the BB (fileserver with additional torrent-daemon),
you think, is it wise to set this parameter to the largest possible value?
Best,
Kurt
2011/10/5 Robert Nelson <robertcnelson@gmail.com>