Subject: RE: switched to strongARM and 160 Mb and Debian 2.2 doesn't boot

From: George France <[email-masked]>
Date: Mon, 28 Aug 2000 13:00:31 -0400

Hello Nico;

We are having the same problem. On Friday, We removed my 32 MB chips from my
only bank on the iPAQ and soldered on some 64 MB chips. 

Best Regards,

--George

> This restriction is only SA11x0 related, and the limit is 
> actually 32 MB
> for each of the four memory banks.  This is due to my original and
> incorrect design which I plan to fix once I'll be sure to do it the
> right way.  Since the generic discontigous memory support 
> moved a bit in
> latest kernels, I wanted to wait a bit more.
> 
> In short, this doesn't concern other architectures than SA1100 based.
> 
> 
> Nicolas
> 
> 
> 
> unsubscribe: body of `unsubscribe linux-arm-kernel' to 
> [email-masked]
> 

unsubscribe: body of `unsubscribe linux-arm-kernel' to [email-masked]

---

From: Erik Mouw <[email-masked]>
Date: Mon 28 Aug 2000 19:22:31 +0100 (MDT)

> only bank on the iPAQ and soldered on some 64 MB chips. 

You know this will void the warranty of your iPaq? ;-)

But same over here on the LART with an extra 64MB board: I can only use
the first 32MB of the 64MB RAM expansion.

Nico, is there an easy way to fix the 32MB per bank limit? Can you point
us where to look in the code?

Erik

-- 
J.A.K. (Erik) Mouw, Information and Communication Theory Group, Department
of Electrical Engineering, Faculty of Information Technology and Systems,
Delft University of Technology, PO BOX 5031,  2600 GA Delft, The Netherlands
Phone: +31-15-2783635  Fax: +31-15-2781843  Email: [email-masked]
WWW: http://www-ict.its.tudelft.nl/~erik/

---

From: Nicolas Pitre <[email-masked]>
Date: Mon, 28 Aug 2000 15:17:24 -0400 (EDT)

> us where to look in the code?

It's kind of scattered.

As a prerequisite, we need to change the PCMCIA code to ioremap() the
PCMCIA address spaces.  We need to get rid of most of the PCMCIA region
from 0xe0000000 to 0xf7ffffff.  They are probably excessively wide and the
PCMCIA code must know how large it wish them to be.

Nicolas

---

From: Erik Mouw <[email-masked]>
Date: Mon 28 Aug 2000 21:26:57 +0100 (MDT)

> PCMCIA code must know how large it wish them to be.

Why? From 0xc0000000 to 0xdfffffff is 512MB of memory, which is the
maximum physical memory on an SA11x0 CPU. No need to move the PCMCIA
space. However, there are some SA11x0 boards that use 0xd0000000 to map
the flash memory. So either we limit the total DRAM to 256MB, or we move
the flash region.

Erik

-- 
I'm not a lawyer. I don't even play one on TV.
  -- Linus Torvalds on the gcc mailing list

---

From: Russell King - ARM Linux Admin <[email-masked]>
Date: Mon, 28 Aug 2000 21:07:15 +0100 (BST)

> us where to look in the code?

See linux/include/asm-arm/arch-sa1100/memory.h - its to do with the
virtual to physical memory translation "compression" that is done.

Note, however, that there are several problems with modifing that:

1. It may cause machines with only 4MB in the first bank to fail to boot.
2. Effectively, you make 0xc0000000 through to 0xdfffffff available for
   the direct mapped region; unfortunately, 0xd0000000 upwards is used
   for IO (typically flash, SCR, BCR and the SA1111 on Assabet).
   See linux/arch/arm/mm/mm-sa1100.c for that.

In all, getting >32MB per bank will probably not be trivial.
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |            http://www.arm.linux.org.uk/            /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |

---

From: Russell King - ARM Linux Admin <[email-masked]>
Date: Mon, 28 Aug 2000 21:11:05 +0100 (BST)

> the flash memory. So either we limit the total DRAM to 256MB, or we move

If 0xdfffffff is the top of your memory, and 0xe0000000 is where IO-type
stuff starts, which nibble does the vmalloc space fit in between these?

Yes, vmalloc sits above the kernel direct mapped memory, so 0xe0000000
upwards has to change as well.
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |            http://www.arm.linux.org.uk/            /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |

---

From: "J.D. Bakker" <[email-masked]>
Date: Mon, 28 Aug 2000 22:30:38 +0200

>   See linux/arch/arm/mm/mm-sa1100.c for that.

Maybe I'm not seeing the bigger picture, but wouldn't changing

#define __virt_to_phys(x)	(((x) & 0xf9ffffff) | ((x) & 0x06000000) << 2)
#define __phys_to_virt(x)	(((x) & 0xe7ffffff) | ((x) & 0x18000000) >> 2)

to

#define __virt_to_phys(x)	(((x) & 0xf9ffffff) | ((x) & 0x06000000) << 1)
#define __phys_to_virt(x)	(((x) & 0xe7ffffff) | ((x) & 0x18000000) >> 1)

yield 64M banks, while using only the 0xCxxxxxxx region for DRAM ? I admit
I don't know too much about the new memory mappings, but 0xC0000000 -
0xCFFFFFFF seems otherwise empty, right ?

JDB
[who should maybe stick to hardware, but...]

--
LART. 250 MIPS under one Watt. Free hardware design files.
http://www.lart.tudelft.nl/

---

From: Russell King - ARM Linux Admin <[email-masked]>
Date: Mon, 28 Aug 2000 21:42:48 +0100 (BST)

> 0xCFFFFFFF seems otherwise empty, right ?

Not quite correct, but:

#define __virt_to_phys(x)	(((x) & 0xf3ffffff) | ((x) & 0x0c000000) << 1)
#define __phys_to_virt(x)	(((x) & 0xe7ffffff) | ((x) & 0x18000000) >> 1)

is probably what you were meaning, and yes, it still fits in the 0xc0000000
to 0xcfffffff space.  But what about the vmalloc area, when you've got all
the banks used and full (ie 256MB of memory)?

You need to move stuff at 0xd0000000 out of the way at least.
   _____

---

From: "J.D. Bakker" <[email-masked]>
Date: Mon, 28 Aug 2000 23:03:07 +0200

At 22:42 +0200 28-08-2000, Russell King - ARM Linux Admin wrote:
>is probably what you were meaning,

Duh. Yes, you're right; shows me once again to test theories before posting
them. OTOH, a LART with two 16M banks and this change (and the obvious one
to mmzone.h) boots and runs happily ;-)

> and yes, it still fits in the 0xc0000000

That's not likely to be the normal mode of operation on SA boards (famous
last words). With increasing SDRAM densities and tough SDRAM timing
requirements on a loaded bus I expect the norm to be 1 or 2 banks filled.
That being said it would be silly to have a default config that breaks
when, say, bank 3 has 64M in it yes.

>You need to move stuff at 0xd0000000 out of the way at least.

Agreed. BTW, why isn't DRAM mapped contiguously any more ? And how do other
platforms handle this when faced with, say, 1GB DRAM and the standard 3/1
split ?

JDB.
[who hasn't touched these code paths since 2.2.x]

--
Jan-Derk Bakker, [email-masked]

The lazy man's proverb:
    'There's no business like slow business !'

---

From: Russell King - ARM Linux Admin <[email-masked]>
Date: Mon, 28 Aug 2000 22:07:32 +0100 (BST)

> platforms handle this when faced with, say, 1GB DRAM and the standard 3/1

The problem is all to do with those two macros above.  As they stand, they
should compile down to:

	bic	%0, x, #0x0c000000
	and	%1, x, #0x0c000000
	orr	%0, %0, %1, lsl #1

(but I imagine that as always GCC will produce far worse code than that) ;(

These macros are used extensively within the memory management system, and,
if they are slow, you have a slow machine.  Therefore, we do everything we
can to make these as fast as possible.  Normally, I'd like them to be a
pure phys = virt - offset type arrangement, but the SA1100 platforms are
the only ones which don't allow this.
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |            http://www.arm.linux.org.uk/            /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |

---

From: Erik Mouw <[email-masked]>
Date: Mon 28 Aug 2000 23:21:22 +0100 (MDT)

> the banks used and full (ie 256MB of memory)?

This patch allows 64MB/bank; tested on the LART, YMMV.
Oh, and #include <vmalloc-dislaimer>.

--------------------------------------------------------------------
diff -urN linux-2.4.0-test6-rmk5-np2-vanilla/include/asm-arm/arch-sa1100/memory.h linux-2.4.0-test6-rmk5-np2/include/asm-arm/arch-sa1100/memory.h
--- linux-2.4.0-test6-rmk5-np2-vanilla/include/asm-arm/arch-sa1100/memory.h Thu Jan 13 22:30:31 2000
+++ linux-2.4.0-test6-rmk5-np2/include/asm-arm/arch-sa1100/memory.h Mon Aug 28 23:02:36 2000
@@ -29,10 +29,11 @@
 #define __phys_to_virt__is_a_macro
 
 /*
- * The following gives a maximum memory size of 128MB (32MB in each bank).
+ * The following gives a maximum memory size of 256MB (64MB in each bank).
  */
-#define __virt_to_phys(x)  (((x) & 0xf9ffffff) | ((x) & 0x06000000) << 2)
-#define __phys_to_virt(x)  (((x) & 0xe7ffffff) | ((x) & 0x18000000) >> 2)
+#define __virt_to_phys(x)   (((x) & 0xf3ffffff) | ((x) & 0x0c000000) << 1)
+#define __phys_to_virt(x)   (((x) & 0xe7ffffff) | ((x) & 0x18000000) >> 1)
+
 
 /*
  * Virtual view <-> DMA view memory address translations
diff -urN linux-2.4.0-test6-rmk5-np2-vanilla/include/asm-arm/arch-sa1100/mmzone.h linux-2.4.0-test6-rmk5-np2/include/asm-arm/arch-sa1100/mmzone.h
--- linux-2.4.0-test6-rmk5-np2-vanilla/include/asm-arm/arch-sa1100/mmzone.h Thu Aug 24 12:52:38 2000
+++ linux-2.4.0-test6-rmk5-np2/include/asm-arm/arch-sa1100/mmzone.h Mon Aug 28 23:02:37 2000
@@ -20,9 +20,9 @@
 extern pg_data_t sa1100_node_data[];
 
 /*
- * 32MB max in each bank, must fit with __virt_to_phys() & __phys_to_virt()
+ * 64MB max in each bank, must fit with __virt_to_phys() & __phys_to_virt()
  */
-#define NODE_MAX_MEM_SHIFT 25
+#define NODE_MAX_MEM_SHIFT 26
 #define NODE_MAX_MEM_SIZE  (1<<NODE_MAX_MEM_SHIFT)
 
 /*
--------------------------------------------------------------------

> You need to move stuff at 0xd0000000 out of the way at least.

OK, so where is a good place? (Is the memory map on ARM linux somewhere
defined?)

Erik

---

From: Russell King - ARM Linux Admin <[email-masked]>
Date: Mon, 28 Aug 2000 22:35:21 +0100 (BST)

> This patch allows 64MB/bank; tested on the LART, YMMV.

Ok, it looks mostly sane, but you do want to look at
include/asm-arm/arch-sa1100/vmalloc.h as well - it defines the
area for vmalloc/ioremap stuff.

> OK, so where is a good place?

That is up to Nico/individual platform people, but there are a large
number of SA1100-based platforms that have stuff down at 0xd0000000.

> (Is the memory map on ARM linux somewhere defined?)

It's architecture dependent, so there is no "one true reference".
However, in the spirit of Linux, it's in the source.
arch/arm/mm/mm-sa1100.c defines the memory mapping above the so called
"direct mapped" kernel memory that virt_to_phys and friends work on.
In test6-rmk5 (?) and higher, its defined by two tables - a common
"standard_io_desc" which is for all SA1100 devices, and the individual
platform tables (eg, "assabet_io_desc").  Glue the relevent tables
together, and sort by virtual address and you have the memory map.

(Note that that is not the whole story, but as a generalisation,
0x0000-0x0fff are for vectory-type things, 0x1000 to 0xbfffffff
is userspace, 0xc0000000 upwards is kernel direct mapped memory,
and on top of that is IO).
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |            http://www.arm.linux.org.uk/            /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |

---

From: Mike Touloumtzis <[email-masked]>
Date: Mon, 28 Aug 2000 15:09:15 -0700

> the only ones which don't allow this.

The Cirrus Logic 7110 and descendants have the same issue; I'm currently
using the same technique as SA1100 with the EP7211, so I'm following
this discussion with interest.  The original Psion port used a table
lookup for virt_to_phys and phys_to_virt, but I imagine that's a
performance nightmare.  The EP7211 has two DRAM banks (each 256MB,
so they're unlikely to be contiguous), and to add insult to injury,
the EDB7211 eval board has not-contiguous RAM within the first bank!

The recent kernels also have VALID_PAGE(), which looks like another
potential problem.  The SA1100 now has: 

#define VALID_PAGE(page)        ((page - mem_map) < max_mapnr)

but from looking at the NUMA code and the other macros in mmzone.h,
it looks like there could be per-node mem_map arrays, so the above
code wouldn't work.  Am I missing something?

miket

---

From: Russell King - ARM Linux Admin <[email-masked]>
Date: Mon, 28 Aug 2000 23:23:03 +0100 (BST)

> potential problem.  The SA1100 now has: 

Yes, it is another problem, but one which (hopefully) the numa people
are tackling at this present time (they were looking at it - I've got
their private discussion archived here).
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |            http://www.arm.linux.org.uk/            /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |