Subject: memory mapping

From: Mark van Doesburg <[email-masked]>
Date: Wed, 2 Sep 1998 14:18:18 +0200

This needs to be fixed anyway.  Basically we need to read the
	bit from the 21285 that tells it whether or not to be the PCI
	"central function", and if not avoid doing all the PCI config
	stuff.  Then __virt_to_bus() and so on need to be taught that
	the offset might not be constant.  I think the overhead is small
	enough we can ignore it but you could always make it a config
	option if people were that worried.

OK, thats relatively easy to fix.

	It would be compatible for user processes.  It doesn't all that
	much matter where the kernel is loaded across different
	architectures so you could change it if you really wanted to.

Great ! This is really the answer i hoped for.

	No matter where you map the kernel you aren't going to be able
	to get at the whole of the 4GB PCI memory space.  I would have
	thought the best thing to do would be to leave the kernel where
	it is and ioremap() memory in as and when you want it rather
	than trying to always map the whole lot at once.

I assume you don't want to reserve memory below address 0xc0000000, that
would seem verry messy to me. Not mapping below 0xc0000000 would not give
me the memory area i need.

I don't see why i wouldn't be able to accces the entire 4GB PCI memory space,
consider the following memory mapping;

0x00000000 - 0x00007fff		for catching NULL pointer access
0x00008000 - 0x5fffffff		user code/data/stack
0x60000000 - 0x6fffffff		SDRAM access for the kernel
0x70000000 - 0x7fffffff		21285 registers, PCI I/O, ...
0x80000000 - 0xffffffff		PCI memory mapped I/O

This would give me access to everything, all possible SDRAM, PCI I/O,
PCI mem and all the 21285 registers. PCI memory may not be obvious but
it really is accessible in a portable way with this mapping. In
linux/Documentation/IO-mapping.txt the following is said;

---
This memory is called "PCI memory" or "shared memory" or "IO
memory" or whatever, and there is only one way to access it: the
readb/writeb and related functions. You should never take the
address of such memory, because there is really nothing you can
do with such an address: it's not conceptually in the same
memory space as "real memory" at all, so you cannot just
dereference a pointer.
---

The writeb and related functions should be able to toggle bit 15 in the
PCI address extension register (offset 0x140).

Note: ioremap would be really easy to write with this mapping.

	>Would this 1.5GB be per process or for all processes together ?
	>(Is the MMU register reloaded on a task switch ?)

	It's per process, and yes.  This is why context switches are so
	slow on the ARM; we need to flush the cache because the memory
	mappings have changed.

That's what i assumed, i just wanted to be sure.

It seems like my problems are solved.

	p.

Thanks,

Mark.
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Mark van Doesburg <[email-masked]>
Date: Thu, 3 Sep 1998 16:41:03 +0200

Philip Blundell writes:
	> From what I've seen he seems to be planning most of a cluster-type 
	> architecture rather than actual multiprocessing, in which case cache coherency 
	> doesn't enter into it.  We've been through the reasons that real SMP is bad 
	> news on current ARM machines quite recently, but a Beowulf-type cluster that 
	> used PCI for message passing would be a neat toy.

	Would this really require all memory to be accessible at all times by
	everything?

That would be the preferred solution, not the only one.

Mark.

---

From: Mark van Doesburg <[email-masked]>
Date: Thu, 3 Sep 1998 16:50:17 +0200

No, Beowulf is a NUMA (Non-Uniform Memory Architecture) system -
	processors can't access remote memory directly.  They
	communicate by message passing. Conventional Beowulf systems
	communicate over 100MBit ethernet, so communicating via PCI
	should be significantly faster.

	-- Matthew Wilcox <[email-masked]>

It should be possible to run Beowulf software on it since the shared
memory network driver still is a normal network driver. The advantages
of this system are the speed of the network and the ability to create a
shared memory segment between processors. This makes it possible to use
this system for finer grained problems than is possible on Beowulf
systems.  The main advantage of this system over SMP is the increased
memory bandwidth.

Mark.

---

From: Mark van Doesburg <[email-masked]>
Date: Thu, 3 Sep 1998 17:37:16 +0200

Another option incidentally might be to dedicate a region of say
	1MB on each CPU for message passing and have only that area
	mapped into PCI space rather than the whole of RAM.

	p.

That is a good idea if remapping the kernel really is a problem. It
would still mean i have to use the DMA channel (and thus have to flush
caches on both sides) It would however be a lot better than putting all
messages in the host memory.

I would still map the whole sa ram on the PCI bus since the host does
not have the memory mapping problem. (the intel uses segmentation and
the alpha/sparc are 64 bit processors) But only the actual message areas
would have to be mapped on the sa's.

But it would probably be more work than remapping the kernel and still
be less flexible. (Simply adding an additional message frame would
require the destination to map that area as well, the destination has to
be told to do so and of course ioremap/iounmap would be more
complicated, since i can have a page in the physical
0x80000000-0xffffffff area mapped two times on different bus addresses)

Mark.

---

From: Mark van Doesburg <[email-masked]>
Date: Thu, 3 Sep 1998 19:26:23 +0200

Why couldn't you use the DMA channel to write between StrongARM
	processors?  PCI works a lot better if you can arrange to have
	your data pushed rather than pulled, as it were.

Data will be pushed. However this can be done by writing to the PCI memory
space as well. When i use the DMA processor i would have to flush the
cache on the processor that is going to push the data into the other
processors memory. I would also have to program the DMA processor to transfer the data and receive the interrupt of the DMA processor when it is done. This
is a lot of overhead since most packets will consist of 20 bytes data and a
TCP/IP header. (and a 4 byte hardware header)

	>I was thinking about making it bufferable. This would mean i still get
	>burst writes.

	Yup, that's true.  It's still better to make it cacheable as
	well if you can because that way you get write merging as well.

I will write each value only once so merging is not an issue. Also
the messages (not the packets) will be read/write and must be updated
before an interrupt to the other processor is generated. This would
mean flushing the cache if the memory was cached.

	>And arch/arm/kernel/head-armv.S to do the intial page mapping ?

	Yup, and obviously arch/arm/Makefile so that the linker did the
	right thing.  That's about it I think.

Thanks for pointing that one out.

	p.

Now, with all the information i have it seems like remapping the kernel
to 0x60000000 is the best thing to do in my case. Unless someone can
give a good reason not to do this i will simply remap the kernel.

greetings,

Mark.