Subject: EBSA285 in Add-in Mode + Local ide disk

From: Gaixia Zhang <[email-masked]>
Date: Fri, 10 Sep 1999 09:16:14 -0500

Hi,

Has anybody ever tried to run EBSA285 in Add-in Mode and access
the PC PCI-ide drive as a local disk? Is it possible? I mean to let
EBSA285 access the disk directly other than through NFS as Mark did?

I am trying to include the disc driver into ARM-Linux kernel, I think I
should at least select the following options as I did for my host PC:
     Enhanced IDE/MFM/RLL disk/cdrom/tape/floppy support
     Include IDE/ATA-2 DISK Support
     Generic PCI IDE chipset support
but the option "Generic PCI IDE chipset support" would not show up
in Add-in Mode, it is only available for Host Mode.

I remember John Kliegman ever mentioned in order to add a Ethernet
card for EBSA285 in Add-in Mode, we have to do some interrput
reassignment so that EBSA285 can get interrupt inputs from the Ethernet
card. Do we also need to do interrput reassignment so that EBSA285 can
get interrupt inputs from PCI-ide chipset? If so, will the host still
function?

Or can I add a new PCI-SCSI/PCI-ide card and a new disc drive so that
EBSA285 can access the new disc drive as a local disk?

Thanks,
Gaixia Zhang

________________________________________________________
NetZero - We believe in a FREE Internet.  Shouldn't you?
Get your FREE Internet Access and Email at
http://www.netzero.net/download/index.html
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Philip Blundell <[email-masked]>
Date: Fri, 10 Sep 1999 18:52:52 +0100

>the PC PCI-ide drive as a local disk? Is it possible?

In theory yes, so long as you can arrange for your host (or other cards) not 
to start fighting over the interface.  All PCI interfaces are more or less 
equal.

>in Add-in Mode, it is only available for Host Mode.

I've never been convinced that the hard dichotomy between "host" and "plug-in" 
mode is necessary or desirable.  But this point of view seemed rather 
unpopular last time I aired the idea.

>get interrupt inputs from PCI-ide chipset?

You will need to find a way for the EBSA-285 to see the right interrupt 
numbers.  This is going to be hardware dependent and in some cases it may not 
even be possible -- if the IDE interface is built in to your motherboard 
chipset it may not be capable of generating interrupts on the PCI bus.  With a 
plugin card you will stand more of a chance.

p.

---

From: Russell King - ARM Linux Admin <[email-masked]>
Date: Fri, 10 Sep 1999 20:39:25 +0100 (BST)

> EBSA285 access the disk directly other than through NFS as Mark did?

I wouldn't have thought so.  In order to use IDE with Linux, you need
an interrupt.  The PCI southbridges (which contain the IDE interface
on PCs) which I have come across internally route the IDE interrupt
to the interrupt controller (also in the Southbridge), which means
that it never appears on the PCI bus.

This means that Linux on an add-in EBSA285 in a PC will never be able
to see the interrupt, and therefore will not be able to function.

Yes, you could write your own IDE driver that didn't rely on interrupts,
but the performance would be so dire to make it undesirable.

There is a falicy persistently introduced by Phil into his patches
(against my judgement) which will bite hard with the later 2.3
kernels.  He allows you to configure a host EBSA to be used in plug-in
mode.  This is 100% bad if you're using a PNP BIOS with a PNP OS,
since the EBSA285 will set your PCI bus up for you, possibly even
before the PC's BIOS has looked at it.  The result could be all
hell breaking loose when the OS starts on the host, or when the
BIOS refiddles the PCI bus.  My many complaints did nothing, so
people using his "development" patches do so at their own (potentially
disasterous) risk.
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |  http://www.arm.linux.org.uk/~rmk/armlinux.html    /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Jonathan Foster Kliegman <[email-masked]>
Date: Fri, 10 Sep 1999 15:36:07 -0400 (EDT)

Excerpts from linux-arm: 10-Sep-99 EBSA285 in Add-in Mode + Lo.. by
Gaixia [email-masked] 
> EBSA285 can access the new disc drive as a local disk?

For the work we did with ethernet cards we always used a dedicated card
for the EBSA.  Using the same card at the same time as the host is
asking for trouble and I don't recommend it.  With a network card its
just lost traffic - with a disk controller you may corrupt your disk.

As for the interrupts, you have to realize that not all interrupts are
shared between all cards.  If you have a schematic of your motherboard,
you'll be able to see which slots share interrupts.  If not, you can do
some trial and error.  We had to make sure that both the EBSA and the
NIC were in a pair of slots which shared ethernet cards (on a Dell P6 I
believe this was the the bottom most slot and the 3rd from the bottom).  

We also had to hardcode in the tulip driver to use IRQ_IN1.  You may
have to do some trial and error there as well.

Finally, if your new card does not have an ArmLinux supported driver
(see Russel's page) then you should run it in Programmed I/O mode only -
don't use DMA!  (see the current EBSA285 and caching thread).

-Jon

---

From: Russell King - ARM Linux Admin <[email-masked]>
Date: Fri, 10 Sep 1999 20:51:38 +0100 (BST)

> unpopular last time I aired the idea.

It is not desirable for the following reasons:

1. Linux will assign addresses to any non-setup peripheral,
   potentially causing havoc with the PCs setup of the PCI
   bus.  It is not for Linux to play with the configuration
   registers in add-in mode, but for host mode it must do.
   (generic code).
2. virt_to_bus is heavily dependent on the mapping of the
   memory to the PCI bus.  The BIOS sets the base address
   of the EBSA285's memory on the PCI bus.  It is again
   not for Linux to second-guess the BIOS, nor the PNP bios
   in the host PC.  Since virt_to_bus and friends are
   referenced many times, I feel that it is not desirable
   to slow down the kernel (possibly considerably) by having
   to calculate non-constant offsets for the virtual to PCI
   mapping.

> You will need to find a way for the EBSA-285 to see the
> right interrupt numbers.  This is going to be hardware
> dependent and in some cases it may not even be possible
> -- if the IDE interface is built in to your motherboard 
> chipset it may not be capable of generating interrupts
> on the PCI bus.

Probably won't be capable, and what a good argument against
combining host and add-in mode.  Host and add-in mode are
two completely separate beasts.

> With a plugin card you will stand more of a chance.

No.  With a host card you stand more of a chance of getting
the interrupts.
   _____

---

From: Philip Blundell <[email-masked]>
Date: Fri, 10 Sep 1999 21:50:17 +0100

>   to calculate non-constant offsets for the virtual to PCI

What do you estimate the slowdown would be from introducing a variable into 
these calculations?

>Host and add-in mode are two completely separate beasts.

But, fundamentally, they are not.  The only difference is that you have one 
PCI agent (let's call it the "host", or "central function") which sets up the 
configuration space.  Other than that, all devices are equal in a PCI sense.  
Obviously in a particular implementation there might be issues like interrupts 
that prevent everything from working in quite the way you might hope, but not 
all the world is a PC.

It did use to be the case that you would run into trouble because drivers read 
the PCI configuration space directly to get I/O addresses, interrupt numbers 
and the like.  But now that these come from the PCI database which can be 
massaged in advance, you can remap them in any way you choose for a "plugin" 
card.

If you know that the kernel will be running as a host implementation, you can 
take some shortcuts like hardwiring the offsets in virt_to_bus, and having a 
configuration option for that is just fine.  But I see no reason to consider 
the two modes as completely separate.

>No.  With a host card you stand more of a chance of getting

I think you misunderstood what I wrote there.

p.

---

From: [email-masked]
Date: Fri, 10 Sep 1999 23:43:46 +0100 (BST)

> What do you estimate the slowdown would be from introducing a variable into 

I haven't sat down and worked through all the logic yet, but at least one
of two LDRs will be to obtain the value will be from an as yet uncached
location.  Say we're running at 275MHz on a 50MHz bus.  You're looking at
at least 275/50*8 (two cache lines will need to be read), = 44 cycles per
(currently it's only one or two cycles).  If you have an application which
is sending over ethernet, say, then you're going to be hitting the data cache
hard, so this may be more like 88 cycles per.  At 1MB/s, you're going to have
to do this 692 times, which means you've wasted 29064 cycles per second doing
a calculation above what it needs to be on a host.

However, you wouldn't want to cache the setting, since the base register is
free to be changed by any PNP BIOS or OS at any time.  Only the host knows
what the current state of the PCI bus mapping is.

I do not expect to receive a reply which says "it's only 29064 cycles per
second", because if I do, I'll bounce all your "lets save a cycle here and
there" patches back at you! ;O

> that prevent everything from working in quite the way you might hope, but not 

Oh I wish that everyone else knew that.  However, unless I am mistaken, this
thread was talking about an EBSA in a PC, which is after all how most people
are going to use it when in add-in mode.  In order not to upset the host
configuration, everything to do with the handling of the PCI bus should be
done correctly, and definitely not how your patch does it.

> the two modes as completely separate.

Maybe it doesn't have to be separate if it's done in the right way, but so far
I have not seen "the right way", despite pointing this flaw out in a previous
email.  I have not seen that there is demand for a combined host and add-in
kernel to date either.

I think before even looking into/touching this area/allowing patches to
touch this area, I want to see the generic PCI code stabilise.  Martin Mares
has some patches outstanding for it.
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |  http://www.arm.linux.org.uk/~rmk/armlinux.html    /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Philip Blundell <[email-masked]>
Date: Sat, 11 Sep 1999 09:55:54 +0100

>what the current state of the PCI bus mapping is.

I don't have the PCI specification here to check, but while that may be true 
in some theoretical sense, in practice all reasonable implementations keep 
BAR and irq line values cached rather than going to config space every time.  
If the mapping for a device is changed it is reasonable to have to restart 
that device.

>done correctly, and definitely not how your patch does it.

I have never claimed that the patch you are thinking about does everything you 
need for a plug-in card implementation; it is necessary, but not sufficient.  
Last time the issue was raised you refused to even discuss it so I put that 
on the back burner.

>has some patches outstanding for it.

Well, let us know when it has stabilised to your satisfaction.

p.

---

From: [email-masked]
Date: Sat, 11 Sep 1999 18:32:02 +0100 (BST)

> If the mapping for a device is changed it is reasonable to have to restart 

Hardware devices do not - the hardware checks the BARs each time.

> Last time the issue was raised you refused to even discuss it so I put that 

Last time, I did not refuse to discuss the problem.  I may have missed replying
to some message, but that's not a refusal.
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |  http://www.arm.linux.org.uk/~rmk/armlinux.html    /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Philip Blundell <[email-masked]>
Date: Sat, 11 Sep 1999 18:47:11 +0100

>Hardware devices do not - the hardware checks the BARs each time.

No, but the drivers for those devices do.

p.

---

From: [email-masked]
Date: Sat, 11 Sep 1999 18:53:31 +0100 (BST)

> No, but the drivers for those devices do.

If you rommify Linux into the EBSA285, how do you cater for the case where
Linux starts up on the EBSA285 before the PC BIOS has setup the PCI cards?

Or, if you're running a PNP OS, chances are (because it's from Redmond) that
it'll have it's own ideas about how to change the PCI bus mappings a long
time after boot.  Again, how do you intend to handle this?
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |  http://www.arm.linux.org.uk/~rmk/armlinux.html    /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Philip Blundell <[email-masked]>
Date: Sat, 11 Sep 1999 20:57:52 +0100

>time after boot.  Again, how do you intend to handle this?

As with any other expansion card it will need a driver, and this can arrange 
to reset the card once its world is stable.

p.

---

From: [email-masked]
Date: Sat, 11 Sep 1999 23:15:27 +0100 (BST)

> to reset the card once its world is stable.

How do you define "stable"?  How does the driver determine that the world is
"stable"?
   _____

---

From: Philip Blundell <[email-masked]>
Date: Sun, 12 Sep 1999 07:54:05 +0100

>How do you define "stable"?  How does the driver determine that the world is

Well, as I mentioned before, drivers routinely cache BAR and other 
configuration values.  So the PnP OS must either perform the appropriate setup 
prior to loading the driver, or provide some mechanism for the driver to keep 
abreast of what's happening.

The fact that the card in question is running Linux isn't really significant 
at all.

p.

---

From: [email-masked]
Date: Sun, 12 Sep 1999 09:45:54 +0100 (BST)

> The fact that the card in question is running Linux isn't really significant 

You've totally missed my point.  We were talking about the Linux add-in side,
and the virt_to_bus macro, which must know what settings the host places into
the BAR at all times.

I'm not interested in the host driver for the EBSA285 add-in board, since while
there is a driver actively using the add-in card, the BARs must remain stable.
I am only concerned about the EBSA285 itself.
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |  http://www.arm.linux.org.uk/~rmk/armlinux.html    /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Philip Blundell <[email-masked]>
Date: Sun, 12 Sep 1999 10:06:19 +0100

>I'm not interested in the host driver for the EBSA285 add-in board, since 
>while there is a driver actively using the add-in card, the BARs must remain 
>stable.  I am only concerned about the EBSA285 itself.

Yes, and my point is that it really doesn't matter.  If you put an EBSA-285 in 
a PC then for it to be any use you must have a host driver -- if you don't 
want it to communicate with the host, it might as well be a standalone device.
So, all the time the EBSA is running this driver will be loaded and the PCI 
configuration space will be stable.

p.

---

From: [email-masked]
Date: Sun, 12 Sep 1999 10:45:30 +0100 (BST)

> configuration space will be stable.

You're still totally missing the point.

1. Power up PC with EBSA285 installed and Linux ROMMified
2. BIOS scans the PCI bus, and the PCI bus stops since EBSA285 sends
   retry cycles
3. Linux boots on EBSA285, sets up DC21285, sets up BAR masks, and
   reads BARs for virt_to_bus.
4. BIOS reads and possibly sets BARs (depending whether the BIOS is in PNP
   mode, read l-k for more information on this).
5. PC boots into PNP OS.
6. PNP OS reassigns/assigns all BARs.

The net result is that Linux on the EBSA285 has out of date BAR information,
or the BARs could be setup quite late.  How does the EBSA285 know when to
read the BARs for virt_to_bus?  Does this make my point crystal clear?
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |  http://www.arm.linux.org.uk/~rmk/armlinux.html    /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Philip Blundell <[email-masked]>
Date: Sun, 12 Sep 1999 15:14:37 +0100

>6. PNP OS reassigns/assigns all BARs.

7. Host driver for EBSA-285 loads
8. Linux on EBSA-285 is rebooted (or, indeed, started for the first time)
9. Linux re-reads BAR values

p.

---

From: [email-masked]
Date: Sun, 12 Sep 1999 15:28:30 +0100 (BST)

> 9. Linux re-reads BAR values

How do you intend to reboot Linux on the EBSA-285?  You can't assert the PCI
reset line to individual sockets on PCI.
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |  http://www.arm.linux.org.uk/~rmk/armlinux.html    /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Philip Blundell <[email-masked]>
Date: Sun, 12 Sep 1999 17:02:37 +0100

>reset line to individual sockets on PCI.

No, that's true.  Most PCI bridges, though unfortunately not the 21285, let 
you drive local reset from software (via a register bit).  I guess you could 
do the same with some external hardware, or have a boot ROM on the EBSA-285 
that didn't actually start Linux until it received some signal from the 
host.  Obviously the latter would be less robust in the face of a PnP OS 
wanting to arbitrarily reorganise PCI space some time after booting, but that 
might not be too much of a problem in reality.

p.

---

From: Paul Koning <[email-masked]>
Date: Mon, 13 Sep 1999 09:44:54 -0400

>>>>> "linux" == linux  <[email-masked]> writes:

 linux> Philip Blundell writes:
 >> What do you estimate the slowdown would be from introducing a
 >> variable into these calculations?

 linux>...
 linux> However, you wouldn't want to cache the setting, since the
 linux> base register is free to be changed by any PNP BIOS or OS at
 linux> any time.  Only the host knows what the current state of the
 linux> PCI bus mapping is.

Does any OS actually do such a bizarre thing (change the base register 
out from under a device that's in use)?  It is obvious that new
devices get new addresses and disappearing devices have their
addresses freed, but the notion of moving base addresses of active
hardware on the fly strikes me as absurd.  For one thing, depending on 
the device you cannot even do such a thing at all if there is any I/O
pending. 

	paul

---

From: [email-masked]
Date: Mon, 13 Sep 1999 22:08:08 +0100 (BST)

> the device you cannot even do such a thing at all if there is any I/O

Wrong end of the stick.  If you have the EBSA285 in add-in mode, then
it needs to know the values of the BARs.  The EBSA285 does not necessarily
know if the BARs have been set by the host - only the host knows this, and
yes, if the host has a driver for it loaded, then it shouldn't change them.

However, at what point does the EBSA285 know in fact that it's been
correctly configured so it can read the BARs?  There is no "initialisation
complete" bit as there is for the EBSA285 to tell the host it's ready.
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |  http://www.arm.linux.org.uk/~rmk/armlinux.html    /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: Jonathan Foster Kliegman <[email-masked]>
Date: Mon, 13 Sep 1999 17:22:25 -0700

> complete" bit as there is for the EBSA285 to tell the host it's ready.

There is no explicit bit but one can be implemented easily enough using
the mailbox registers (or doorbell's or some equivalent).   

For example:

Host + EBSA both get power.
Host pauses on PCI retries of EBSA
EBSA sets the first mailbox register (offset 0x50?) to 0, no/minimal OS
initialization yet.
EBSA sets PCI ready bit and busy-waits on the mailbox register.
Host continues booting, BIOS and OS scramble the BARs.
Host OS loads driver for the EBSA.
Host EBSA driver writes 0xED (or some other magic byte) into the first
mailbox register.
EBSA sees the mailbox register has the correct magic byte and continues
loading.  It can even set the mailbox register back to zero when it is
done to let the host know it has finished loading.
Host and EBSA are both booted and happy with a consistant PCI memory
view from both sides.

Mark's co285 patch uses a variant of this method (in the other
direction) to supply information about the EBSA-285 memory size to the
host.  We used a similar trick at CMU to coordinate readiness between
two drivers, one on the host, one on the EBSA.

Oh - and everywhere above that I said EBSA I really meant EBSA-285.  

And to butt back into the previous/related topic on HOST/ADD-IN mode, I
tend to agree with Phil here.  The add-in mode should be given more
access to the PCI bus.  We were able to do it with only a few patches
made to the add-in mode configuration (basically copying a few
memory-mappings and defines from the host mode).  As long as the
programmer is smart enough to only have a driver installed on either the
host or the EBSA there shouldn't be a problem.

-Jon