Subject: EBSA285 and Cacheing

From: Dave Baukus <[email-masked]>
Date: Thu, 09 Sep 1999 17:39:08 -0500

I'd like to address the issue of data caching on the ebsa285.
(kernel is 2.2.10-rmk2)

We have been running our ebsa285s with data caching off. With caching disabled
the stock ncr53c8xx SCSI driver works well and the lance ethernet driver works
nicely w/ a few changes. Everything falls apart, however, when we turn on 
data caching. The NCR scsi shuts its self down after detecting
cache coherency problems, and the lance ethernet simply does not work.

In a naive attempt to solve the problem we've tried several combinations of
the following:
	1.) There is a driver out there called bigphysmem; it allocates a fixed
	chunks of memory at boot time and provides a malloc() interface to it.
	We turn off caching on the sections managed by this driver.
	We modified the scsi and ether drivers to use the bigphysmem malloc().

	2.) we modified kmalloc et.al. to turn off caching on the
	returned pages.

With any combination of these changes the kernel will boot
and the drives work reasonably well, but not 100%.

One class of problems seems to be that some file are corrupted.
for example the shell complains when it tries to source in
its config files:

bash: .: /etc/profile.d/lang.sh: cannot execute binary file

With caching turned off, the bigphysmem modified drivers
work as expected (it boots w/ no errors).

Finally to the questions:
	1.)	Is anybody actually running an ebsa285
		with caching enabled ?

		If so what type of changes are
		required to the drivers ?

	2.)	Are we way off base here ?
		Shouldn't the bigphysmem or changes to kmalloc()
		solve the problem ?

		What subtlety are we missing ?

--
[email-masked]
	Inet Technologies Inc.
	Plano, Texas
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Jonathan Foster Kliegman <[email-masked]>
Date: Thu, 9 Sep 1999 18:57:18 -0400 (EDT)

We use an EBSA-285 with caching on.  You have to be careful which
drivers you use and any references to DMA data must be protected by
cache flushes.  Look at the tulip.c driver -  you'll see an example of
all the cache flushes used.  You'll need to modify any of your drivers
which use DMA to behave in a similar way.  Basically, you need to flush
the cache immediately after setting up data structures read by the
device and you need to invalidate the cache before reading those data
structures.  

-Jon

Excerpts from linux-arm: 9-Sep-99 EBSA285 and Cacheing by Dave [email-masked]

---

From: Benjamin Herrenschmidt <[email-masked]>
Date: Fri, 10 Sep 1999 10:30:38 +0200

>		What subtlety are we missing ?

This is a generic problem with all SA110/21285 platform (EBSA, Netwinder,
CATS, ...)

I discussed this recently with Philip and Russel. The problem is that the
SA110/21285 pair doesn't handle cache coherency. That means that when a
devices accesses the memory via the PCI (bus master), nothing is done to
make sure the datas are coherent with the CPU cache.
(The architecture of the SA110 cache, based on virtual addresses, would
probably have made this almost impossible without a bunch of hacks in the
silicon).

Some drivers have been "fixed" for this behaviour by adding various cache
flushes and invalidate on memory ranges at specific locations. Basically,
you need to invalidate a range which will be written by the device and
flush a range which will be read by the device. This is how the tulip
driver was fixed and I did fix the pcnet32 driver the same way recently
(I can send you patches). I've seen a fixed sym53c8xx in the CVS recently.

The problem with this solution is that it's far from perfect. It imposes
to place those flush and invalidate in critical locations which can make
the drivers quite unmaintainable. Also, when dealing with "shared
regions" where both the CPU and the device will read&write small datas
(words), there are still some potential coherency issues. The CPU flushes
cache lines, so when flushing, let's say, a word, the entire cache line
containing this word will be flushed, possibly corrupting whatever the
device wanted to write somewhere else in this same cache line. The
current fixed tulip and pcnet32 seems to work fine, probably because the
SA110 support a mecanism of "half cache lines" (2 dirty bits per line),
making the half cache line the same size as a ring descriptor entry for
those drivers.

However, a better solution (and probably more efficient, especially for
the symbios driver) would be to allocate those shared regions in
non-cachable space. Flushes and invalidate would still be required for
data buffers, but ring descriptors or SCSI controller scripts should be
in non-cachable space.

I tried implementing a vmalloc_uncached (with the help of Russel) but
unfortunately, virt_to_bus can't get the physical address of a vmalloc'ed
area, and I didn't want to walk the page tables.

I had to stop hacking on this (more urgent work to do), but I beleive
there is still the possibility to use the low-level _ioremap function to
create a second, uncachable, mapping for a given kmalloc'ed region. If
you make sure to invalidate it once, and then only use the address
returned by _ioremap, this should work.

I'll do more experiments with this next week,

Benjamin.