Subject: large ethernet transfers killing ARMLinux

From: Nicholas Clark <[email-masked]>
Date: Fri, 12 Feb 1999 15:49:25 +0000 (GMT)

I think I've now had four reset button job lockups from ARMLinux where I've
throwing large files over Ethernet. The most recent today was ftping a 83M
file in from Solaris box on the local Ethernet - after screens of # marks
everything locked up with 72M downloaded. After previous crashes I had a look
at /var/log/Messages and there's nothing at all there from the time of the
crash (no clues :-() I'm using an I cubed EtherH card in the network card
slot of an issue 3 motherboard 610 (October 1996) with a revision K
StrongARM, 38M in all and one 8.4G IBM HD (if any of that helps) with Kernel
2.0.35.

Willy appears to be getting similar (ie no warning, no clues) lockups with
an Ether3/4 card in his machine.

Any suggestions on what I should do? Or what further info I should supply?

Nick
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Hwa Jin Bae <[email-masked]>
Date: Fri, 12 Feb 1999 08:34:14 -0800

I too have noticed that large ethernet transfers can lock up
ARMLinux (2.2.1 and previous ones).  This can happen quicker
if disk I/O is active at the same time.  But without too much
disk I/O active, it still happens.   It can happen when you
use non-flow controlled "flooding" via UDP or ICMP against
ARMLinux ethernet device, but you can also cause it happen
with TCP programs (ttcp for example) under stress.  I've seen
this happen with DEC21143 ethernet on a couple of different
SA110 based computers.  The result is quite consistent failure.
Right now this is the only instability I see in the current
kernel release.

-- 
Hwa Jin Bae
PSO Systems Inc
[email-masked]   
http://www.pso.com

---

From: Nicholas Clark <[email-masked]>
Date: Fri, 12 Feb 1999 17:27:05 +0000 (GMT)

> Right now this is the only instability I see in the current

Interesting - so if these are the same thing then it's probably not anything
wrong with Acorn or third party hardware.
Most recent lock up was 191K into my ftp (from Easynet in London to a machine
in Liverpool University, so even going off JANET I've got enough bandwith to
kill it! :-(

Nick

---

From: Hwa Jin Bae <[email-masked]>
Date: Fri, 12 Feb 1999 10:05:52 -0800

Well... I am not saying that it is not related to a particular
HW or driver.  I am not saying that it is generic ARM Linux
kernel problem.   At least in my environment I am inclined to
believe that it may be related to PCI bus master related driver level
issues.  That's where I have been trying to debug.  Modifying
the latency timer and cache line values in the PCI config registers
help a little, but does not fix the problem.  I'm still looking...

-- 
Hwa Jin Bae
PSO Systems Inc
mailto:[email-masked]
http://www.pso.com

---

From: Russell King - ARM Linux Admin <[email-masked]>
Date: Fri, 12 Feb 1999 19:44:49 +0000 (GMT)

> StrongARM, 38M in all and one 8.4G IBM HD (if any of that helps) with Kernel

This is very interesting, since I just cannot reproduce any here.  The
following FTP was performed between a NetWinder (disk build #13, kernel
2.0.35) and my RiscPC (2.0.36) on a 10bT connection via a Allied Telesyn
CentreCOM hub.  I've repeated this several times in succession, and these
are the FTP messages I got each time.

62126480 bytes received in 109 secs (5.5e+02 Kbytes/sec)
62124288 bytes received in 108 secs (5.6e+02 Kbytes/sec)
62125916 bytes received in 110 secs (5.5e+02 Kbytes/sec)
62123240 bytes received in 109 secs (5.6e+02 Kbytes/sec)

The interface stats are:

eth0      Link encap:10Mbps Ethernet  HWaddr 00:C0:32:00:7C:F9
          inet addr:192.168.0.3  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:181863 errors:0 dropped:0 overruns:0
          TX packets:52588 errors:0 dropped:0 overruns:0
          Interrupt:11 Base address:0xae00

I have also used in the past ttcp between to ARM Linux boxes, and have not
noticed any crashes.

However, having said that, the EBSA-110 crashed while running 2.2.1 this
week, but I haven't been able to reproduce this problem.  I believe that
this particular problem was caused by the ethernet driver dying.

Hwa - do you have the SysRQ stuff enabled in your kernel?  Does it still
work after the crash?
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |  http://www.arm.linux.org.uk/~rmk/armlinux.html    /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Hwa Jin Bae <[email-masked]>
Date: Sat, 13 Feb 1999 10:14:35 -0800

> Hwa - do you have the SysRQ stuff enabled in your kernel?  Does it still

My target machine only has serial console (no keyboard port).  However
I also have a CATS machine, so I will try to see what happens on that
later on.  

One thing I noticed in tulip.c driver -- it has a "while (1) ;" in a
couple
of places when transmit error happens.  I think original writer of 
this driver wanted to debug if the error happened.  Unfortunately
when it does happen (and I can make it happen in my stress test
on my HW ), the driver busy loops forever in the kernel.  I took
it out and the driver keeps on working (no more hangups!).

However there is still one more problem -- when sending a lot of data
out from ARM linux via tulip to outside machines, back-to-back,
flooding as fast as possible, the tulip transmitter seems to
halt the driver.  Subsequent ifconfig down/up revives it.   I am
trying to debug this now.  This seems to be the only problem I can
think of, in terms of stability.  

Oh... one more thing when I had IDE in PIO mode, when IDE and ethernet
are both busy it seems to behave badly.  I ported latest PDC20246
(promise) driver code that does UDMA a little better and I am
exclusively running in UDMA mode with Quantum drives now.  It works
well.

Other than that, 2.2.1 is quite stable, very good work all of you!
Thanks.

---

From: Hwa Jin Bae <[email-masked]>
Date: Sat, 13 Feb 1999 11:38:13 -0800

A little update on the Ethernet under heavy load problem.
I finally traced down the "problem" to interrupt handling routine
tulip_interrupt().   When there are a lot of packets coming in
from the net (because my stress test program or benchmark
program is flooding the ethernet port), the interrupt handler
doesn't handle this gracefully.  It has a single global that
indicates the desired work load to handle each time the
handler is called (perhaps to limit spending too much
time in interrupt handler).  This is 'max_interrupt_work'
which is aliased to work_budget.  Needless to say when
flooding, we run out of work_budget.  The handler clears
all interrupts when this happens and returns.  I am not
sure if this is graceful enough -- it certains makes the
driver no longer function properly (I/O doesn't happen
correctly any more).  

To solve my particular problem (of surviving all the torture
test I have here), I hacked up a couple of parameters:
namely RX_RING_SIZE and max_interrupt_work.  I increased
these values and so far everything seems to work very
stable.  No, I'm not going to say what numbers I'm using
because it's such a horrible way of "fixing" the problem.
I'm just mentioning this here in case someone else wants
to "pass" things like netperf and netbench (gasp!).