Subject: StrongARM and AMD LANCE PCI NETCARD

From: christophe leroy <[email-masked]>
Date: Wed, 24 Feb 1999 16:21:24 +0000

Has anyone tried a PCNET AM79C971 based card on EBSA285 ?
Mine doesn't works, it is detected by driver but sends nothing
to the network

christophe
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Philip Blundell <[email-masked]>
Date: Wed, 24 Feb 1999 21:56:42 +0000

>Mine doesn't works, it is detected by driver but sends nothing

It probably needs DMA cache-sync operations adding.

p.

---

From: "Dr . David Alan Gilbert" <[email-masked]>
Date: Thu, 25 Feb 1999 09:40:47 +0000

>Mine doesn't works, it is detected by driver but sends nothing

As Phil said the problem is probably due to cache consistency (or the lack
of it)
on the EBSA.

Intel, Alpha and SPARC PCI systems provide cache coherency between the PCI bus
and the processor - so if the Net card writes to main memory and then the host
looks at that location it will pretty much immediatly see the change.

On systems based around the StrongARM/21285 combination if the location
happened to be previously in the processor cache then even after the write
by the net card, the processor will see the old value. This confuses the hell
out of the network drivers which presume they are on a system like Intel.

The tulip driver has been suitably frobbed to perform cache invalidation
where needed;
but this is not true of other drivers.

One way to see if this is the problem is to disable the processor cache by
editing
head-armv.S - there is somewhere where it sets up the cache and write
buffers; with
a line something like (I dont have the source in front of me!):

  orr r0,r0, #????? ; Set bits in MMU registers  I*******DW*

(No I can't remember the details - but the thing to look for is this string
of letters
at the end with separators with a DW near one end).  You want to add a bic
below it
to clear the DW bits to disable the data cache and writebuffer.

Dave

---

From: "Conor O'Gorman" <[email-masked]>
Date: Thu, 25 Feb 1999 09:55:06 +0000

Hello,

I've just recently signed up on the list, so bear with me, thanks.

> out of the network drivers which presume they are on a system like Intel.

This is a bit unfortunate? no?

Whats the root cause of this? the 21285's PCI implementation? or the SA?

Doesn't this mean that most (if not all) IO/DMA device drivers need
patching to run on ARMLinux?

Conor

---

From: "Olof Backing" <[email-masked]>
Date: Thu, 25 Feb 1999 11:07:22 +0100

> This is a bit unfortunate? no?

No.

> Whats the root cause of this? the 21285's PCI implementation? or the SA?

Could be, but try out the modification suggested earlier first to 
determine if it's cache related.

> patching to run on ARMLinux?

No, just map an uncached, unbuffered memmory space for the PCI environment 
and the problem will go away (if it is a cache related issue, that is...)

/Olof

Olof Backing     Net Insight AB              Phone: +46-8 449 22 36
                 Ingenjörsvägen 3, 2tr       Fax:   +46-8 449 22 40
                 S-117 43 Stockholm          mailto:[email-masked]
                 Sweden                      http://www.netinsight.se
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Philip Blundell <[email-masked]>
Date: Thu, 25 Feb 1999 11:05:20 +0000

>patching to run on ARMLinux?

The cause is simply the way the StrongARM and the 21285 are designed.  To 
provide a fully coherent cache without involvement from software, logic must 
be provided in the chipset to snoop cycles on the bus and take appropriate 
action to synchronise caches when necessary.  Some processor architects felt 
this was a worthwhile feature to implement in silicon; others didn't.  The ARM 
isn't by any means the only processor to be this way.  Some MIPS systems have 
the same behaviour; I think some 68000s do as well and I'm sure there are 
others.

The only drivers that are affected are those performing DMA to main memory.
Yes, they need patching to work with Linux/ARM (or similar systems) but the 
changes are usually trivial.

Programmed I/O isn't affected.

p.

---

From: Philip Blundell <[email-masked]>
Date: Thu, 25 Feb 1999 11:10:29 +0000

>and the problem will go away (if it is a cache related issue, that is...)

It's not quite that simple.  We already map PCI I/O and memory uncached.  The 
difficulty with DMA-enabled network drivers is that they want to access main 
memory which obviously needs to be cached for performance reasons.  So 
explicit synchronisation instructions are needed.

We've previously kicked around the idea of adding a GFP_UNCACHED flag to 
__get_free_page to help with this but nobody has implemented it yet.

p.

---

From: Russell King <[email-masked]>
Date: Thu, 25 Feb 1999 11:13:33 +0000 (GMT)

Dr . David Alan Gilbert said:
> of it) on the EBSA.

Beware!  Be very aware!  There is an AM79C971 driver in the kernel (called
am79c971.c) for EBSA-110 systems *only*.  Do not use this on non-EBSA-110
systems.  It will not work.

--
Russell King ([email-masked])
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Philip Blundell <[email-masked]>
Date: Thu, 25 Feb 1999 11:43:57 +0000

>systems.  It will not work.

That's a different driver.  I was talking about the pcnet32 one.

p.

---

From: christophe leroy <[email-masked]>
Date: Thu, 25 Feb 1999 12:21:35 +0000

> systems.  It will not work.

Err, there is only a AM79C961A driver for EBSA110.
I'm using LANCE PCI Driver

Christophe

---

From: "Dr . David Alan Gilbert" <[email-masked]>
Date: Thu, 25 Feb 1999 12:37:49 +0000

>and the problem will go away (if it is a cache related issue, that is...)

Unfortunatly life ain't that easy.

With DMA your DMAing into main memory.  This wouldn't be too difficult to fix
if the sections being DMA'd into were allocated separatly - we could add some
magical flag to say that it needed to be uncached etc.

Unfortunatly many device drivers allocate a block of memory which is accessed
both by the kernel (and which needs to be fast) and by the device on the bus.
Some locations in the block are used by one and not the other and thus
you dont want to alter the cache/buffering settings.

Normally you find that most drivers consist of things which set stuff up in
shared memory and then 'kick' the other end - either by writing to an IO
register
or by writing some magical flag into a piece of shared memory.
You need to find those places in the code and cause a cache flush at that
point.
Sometimes you need to be careful to flush most of the data and then flush
the magical
flag to get the ordering correct.

Then you need to find whereever reads something back (normally in interrupt
code
which reads the magical flag back) and do a cache flush/invalidate before
the read.

(Unfortunatly I can't do much more work on this at the moment since the
cache in my
EBSA285 seems to be ill....)

Dave

---

From: Russell King <[email-masked]>
Date: Thu, 25 Feb 1999 13:40:18 +0000 (GMT)

> I'm using LANCE PCI Driver

That was not clear in your original mail.

---

From: Alan Cox <[email-masked]>
Date: Thu, 25 Feb 1999 14:22:24 +0000 (GMT)

> __get_free_page to help with this but nobody has implemented it yet.

This doesnt help you in general. Having the ring buffer uncached on a lance
is the wrong solution (tm) for most platforms. Its better to flush the page
out of cache and push it to memory when writing. That gets you things like
burst writes.

Alan

---

From: "Olof Backing" <[email-masked]>
Date: Thu, 25 Feb 1999 15:29:21 +0100

> This doesnt help you in general. Having the ring buffer uncached on a
> lance is the wrong solution (tm) for most platforms. Its better to flush
> the page out of cache and push it to memory when writing. That gets you
> things like burst writes.

Accept my apologies. I re-read the code once again (and not as fast as the 
last time) and found that only some parts of the PCI stuff had uncached 
MMU entries and that there did, indeed, exist calls to SysDcacheFlush() in 
my code at those points mentioned above.

/Olof

---

From: Philip Blundell <[email-masked]>
Date: Thu, 25 Feb 1999 14:46:24 +0000

>out of cache and push it to memory when writing. That gets you things like

The idea for cached memory was for descriptor lists and things that you touch 
frequently but usually only a single word at a time.  For those it seems 
better to have an uncached area than push it out of the cache all the time.  
Burst writes aren't a big deal if you're only transferring one word, and a 
single-word uncached read is actually faster than a cache line fill.

For packet buffers and the like I agree with you.

p.

---

From: Paul Koning <[email-masked]>
Date: Thu, 25 Feb 1999 10:59:26 -0500

>>>>> "Philip" == Philip Blundell <[email-masked]> writes:

 >> This doesnt help you in general. Having the ring buffer uncached
 >> on a lance is the wrong solution (tm) for most platforms. Its
 >> better to flush the page out of cache and push it to memory when
 >> writing. That gets you things like burst writes.

 Philip> The idea for cached memory was for descriptor lists and
 Philip> things that you touch frequently but usually only a single
 Philip> word at a time.  For those it seems better to have an
 Philip> uncached area than push it out of the cache all the time.
 Philip> Burst writes aren't a big deal if you're only transferring
 Philip> one word, and a single-word uncached read is actually faster
 Philip> than a cache line fill.

 Philip> For packet buffers and the like I agree with you.

I'll second that.

I spend my days writing code for a MIPS-based router.  As Philip
mentioned earlier, MIPS systems certainly tend to avoid snoopy DMA
designs, and a 68040 I worked on likewise.  The reason for that is
simple: DMA coherence at best gives you a MAJOR complexity hit in the
hardware design, and at worst a major performance hit as well.  (For
example the 68040 supported it but at the cost of a clock tick per DMA 
cycle, not to mention the need to couple the processor bus much more
closely to the other buses in the system.)

For routers, it turns out the best design is to have all I/O related
memory (packet buffers too) uncached.  Sometimes if you don't mind the 
software hair it helps a little to access a header or descriptor via a 
cached alias address but this is very hard to get right.  

For a general purpose system I'd agree that the packet buffers
*probably* should be cacheable -- especially if the CPU in question
can do cache push efficiently.  (The 68040 is an example of one that
cannot -- a null cache push takes 256 cycles!)  But all I/O control
structures such as device descriptor rings or queues are best put in
uncached space; the performance impact is somewhere between trivial
and nil for this sort of system, and the benefits in software
reliability are large indeed.

	paul

---

From: Russell King <[email-masked]>
Date: Thu, 25 Feb 1999 16:27:46 +0000 (GMT)

>  Philip> For packet buffers and the like I agree with you.

However, there is a good reason why it's preferable to have the packet
buffer cached on the StrongARM - the StrongARM core switches between
a high speed clock and the bus clock.  If you're doing several sequential
reads or writes, you really want the cache to be on.  Otherwise the
StrongARM will switch from high to bus, back to high clock for each and
every access.

This incurs extra synchronisation delays etc, so it's not a case of
'minimal performance impact'.

It really depends on the size of the structure you're dealing with though.