Subject: Context switch time

From: <[email-masked]>
Date: Wed, 14 Jul 1999 08:28:54 -0500 (CDT)

Hi,

Running kernel 2.2.10-rmk2-stk1, I have noticed that context switch
times are pretty terrible.  Running lmbench to measure this reports the
following on my 200MHz SA-110:

Context switching - times in microseconds - smaller is better
-------------------------------------------------------------
Host		     OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
			ctxsw  ctxsw  ctxsw ctxsw  ctxsw   ctxsw   ctxsw
--------- ------------- ----- ------ ------ ------ ------ ------- -------
armv4l-li Linux 2.2.10-  200    410    914   453    931     434     930
armv4l-li Linux 2.2.10-  196    415    917   434    955     435     939
armv4l-li Linux 2.2.10-  199    410    915   455    955     434     948
armv4l-li Linux 2.2.10-  198    413    911   433    947     434     934

I know I've heard both good and bad things about lmbench, but are these
numbers fairly accurate?  Looking at the context switch code in
proc-sa110.S shows comments pertaining to flushing the cache being
horribly slow, which I'm guessing is because to flush it you have to
read 16KB of data from the cache flush area.

Have there been many changes to the context switching code in 2.3 at
all that attempt to speed this up?  Thanks.

--
Kyle Mestery			| StorageTek's Storage Networking Group
[email-masked]		| http://www.freebsd.org/
[email-masked]		| http://www.netwinder.org/
	Protect your right to privacy: www.freecrypto.org

unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: "Adam 'WeirdArms' Wiggins" <[email-masked]>
Date: Wed, 14 Jul 1999 23:44:03 +1000 (EST)

On this note I'm writting a technical report (Uni of N.S.W. Aust)
on improving address space switches on the SA-1100. I was after a few
people to 'prove' read it to see I haven't done anything stupid. The
report is being done in rather a rush so those interested would have to
read it and get comments back within a day. I expect it to be ready for
proofing in the next half week or so. Please mail me privately if your
interested in proofing. I'm also after anyone interested implimenting the
idea's into linux. The current design is for a micro-kernel
implimentation.

	Cheers Adam

P.S. Good knowledge of the ARM Arch and the SA-1100 are required as well
as Mem Mang OS background.

---

From: Philip Blundell <[email-masked]>
Date: Wed, 14 Jul 1999 15:07:59 +0100

>read 16KB of data from the cache flush area.

Mostly I think lmbench is telling the truth.  Some of the delay will be due to 
flushing the cache; on the EBSA-285 I'd expect that to be about 70us or so.  
I don't know where the rest of the slowdown is coming from but it would be 
interesting to find out.  I usually get 2-way context switch times of about 
160us on my netwinder, for what that's worth. 

>all that attempt to speed this up?  Thanks.

No.  There's nothing can be done about the cache, other than making some 
effort to avoid flushing it when possible.  This helps a bit with the 
situation where you have one or more kernel threads (nfsd, kflushd, swapper or 
whatever) being scheduled a lot, and also for multithreaded user applications, 
but not a lot for general multitasking.  It might be possible to avoid some of 
the other latency but I don't know of anybody who's actively looking into it 
right now.  I did do a bit of work a month or so back to cut down on cache 
flushing under other circumstances, which helps with things like lat_proc, but 
not context switching as such.

p.

---

From: Philip Blundell <[email-masked]>
Date: Wed, 14 Jul 1999 15:55:59 +0100

>times are pretty terrible.  Running lmbench to measure this reports the

For interest's sake, here's the latest in my set of irregularly-collected 
lmbench results.  I ran the benchmark twice on my NetWinder under both 2.2.10 
and 2.3.9 and I've left in some of the historical ones as a comparison.

There are some funnies -- I've no idea what's going on with the page-fault 
latency measurement under 2.3.9, and it seems strange that select has suddenly 
got 200us slower.  Anyway...

p.

                 L M B E N C H  1 . 9   S U M M A R Y
                 ------------------------------------
		 (Alpha software, do not distribute)

Processor, Processes - times in microseconds - smaller is better
----------------------------------------------------------------
Host                 OS  Mhz null null      open selct sig  sig  fork exec sh  
                             call  I/O stat clos       inst hndl proc proc proc
--------- ------------- ---- ---- ---- ---- ---- ----- ---- ---- ---- ---- ----
alpha-lin Linux 2.1.132  226  4.1  8.5   45   70 0.39K 18.9   45 4.5K  27K  97K
arm32-net   NetBSD 1.3F  224  2.2  11.   62  101 0.27K  5.3  257 15.8K  42K  74K
armv4l-li   Linux 2.2.9  233  0.6  1.3   10   12 0.11K  4.5    4 3.3K  13K  58K
armv4l-li   Linux 2.2.7  275  0.5  1.1    8   10 0.08K  3.8    3 3.0K  14K  26K
armv4l-li  Linux 2.0.31  274  0.6  2.7   16   15 0.08K  5.5   50 3.8K  15K  66K
armv4l-li  Linux 2.2.10  275  0.5  1.1    8   10 0.08K  3.7    3 2.8K  12K  51K
armv4l-li  Linux 2.2.10  275  0.5  1.1    8   10 0.08K  3.7    3 2.8K  12K  51K
armv4l-li   Linux 2.3.9  275  0.5  1.1    8    9 0.10K  3.7    3 2.7K  10K  41K
armv4l-li   Linux 2.3.9  275  0.5  1.1    8   10 0.10K  3.7    3 2.7K  10K  41K
i586-linu   Linux 2.2.6  100  4.5  2.6   22   33 0.20K  4.5    9 1.6K   8K  45K
i586-linu Linux 2.2.0-p  133  1.1  2.1   16   26 0.13K  3.7    8 1.1K   7K  39K
sparc-lin Linux 2.1.130   33  10.7  17.  157  178 1.01K 56.5  105 10.5K  63K 226K

Context switching - times in microseconds - smaller is better
-------------------------------------------------------------
Host                 OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
                        ctxsw  ctxsw  ctxsw ctxsw  ctxsw   ctxsw   ctxsw
--------- ------------- ----- ------ ------ ------ ------ ------- -------
alpha-lin Linux 2.1.132   34    140    699   267   1515     335    1678
arm32-net   NetBSD 1.3F  381    617          821            879        
armv4l-li   Linux 2.2.9                                                
armv4l-li   Linux 2.2.7  162    342    768   360    777     352     779
armv4l-li  Linux 2.0.31  177    348    750   354    770     378     780
armv4l-li  Linux 2.2.10  157    330    731   345    746     355     755
armv4l-li  Linux 2.2.10  158    329    732   331    749     345     769
armv4l-li   Linux 2.3.9  160    334    732   350    735     344     768
armv4l-li   Linux 2.3.9  159    335    733   335    767     355     775
i586-linu   Linux 2.2.6   10     79    346   116    528     126     570
i586-linu Linux 2.2.0-p    8     75    292    82    404     104     462
sparc-lin Linux 2.1.130         517          532           1211        

*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
Host                 OS 2p/0K  Pipe AF     UDP  RPC/   TCP  RPC/ TCP
                        ctxsw       UNIX         UDP         TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
alpha-lin Linux 2.1.132    34   128  288   568         749       2265
arm32-net   NetBSD 1.3F   381   875  871  1256        1383       3278
armv4l-li   Linux 2.2.9         417  691                             
armv4l-li   Linux 2.2.7   162   325  451   612         823       1792
armv4l-li  Linux 2.0.31   177   359  490   709         985       1127
armv4l-li  Linux 2.2.10   157   315  821   590         793       1862
armv4l-li  Linux 2.2.10   158   316  894   588         770       1754
armv4l-li   Linux 2.3.9   160   321  565   625         770       1747
armv4l-li   Linux 2.3.9   159   325  632   598         770       1750
i586-linu   Linux 2.2.6    10    40   73   168         468        869
i586-linu Linux 2.2.0-p     8    29   62   163         239        742
sparc-lin Linux 2.1.130         268  576  1094        1564       4398

File & VM system latencies in microseconds - smaller is better
--------------------------------------------------------------
Host                 OS   0K File      10K File      Mmap    Prot    Page	
                        Create Delete Create Delete  Latency Fault   Fault 
--------- ------------- ------ ------ ------ ------  ------- -----   ----- 
alpha-lin Linux 2.1.132     55     22    170     40    56904         13.8K
arm32-net   NetBSD 1.3F                                35107  1934    5.0K
armv4l-li   Linux 2.2.9                                    0     2    1.9K
armv4l-li   Linux 2.2.7                                28045     1    1.6K
armv4l-li  Linux 2.0.31     65      4    156     10    27317          0.2K
armv4l-li  Linux 2.2.10                                25178     1    1.3K
armv4l-li  Linux 2.2.10                                25168     1    1.3K
armv4l-li   Linux 2.3.9                                 3842     1    0.0K
armv4l-li   Linux 2.3.9                                 3858     1    9.3K
i586-linu   Linux 2.2.6                                21500     5    2.1K
i586-linu Linux 2.2.0-p     40      6    143     12     9439          2.1K
sparc-lin Linux 2.1.130    555    100    763    234    37715          1.5K

*Local* Communication bandwidths in MB/s - bigger is better
-----------------------------------------------------------
Host                OS  Pipe AF    TCP  File   Mmap  Bcopy  Bcopy  Mem   Mem
                             UNIX      reread reread (libc) (hand) read write
--------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- -----
alpha-lin Linux 2.1.132   15   15    8     18     34     13     14   34    36
arm32-net   NetBSD 1.3F    5    5    9     14     75     52     52   72    98
armv4l-li   Linux 2.2.9   -1        -1     -1     -1     -1     -1   -1    -1
armv4l-li   Linux 2.2.7   10   23   18     35     88     62     62   88   116
armv4l-li  Linux 2.0.31    9   15   12     36     92     65     65   92   121
armv4l-li  Linux 2.2.10   10   23   19     37     92     64     64   92   121
armv4l-li  Linux 2.2.10   10   24   16     37     92     64     64   92   121
armv4l-li   Linux 2.3.9   10   23   18     37     92     64     64   92   121
armv4l-li   Linux 2.3.9   10   23   19     37     92     64     64   92   121
i586-linu   Linux 2.2.6   35    4   12     30     75     38     47   75    88
i586-linu Linux 2.2.0-p   37   24   18     35     90     39     46   90    89
sparc-lin Linux 2.1.130    7    2    3      7     18      9     10   18    16

Memory latencies in nanoseconds - smaller is better
    (WARNING - may not be correct, check graphs)
---------------------------------------------------
Host                 OS   Mhz  L1 $   L2 $    Main mem    Guesses
--------- -------------   ---  ----   ----    --------    -------
alpha-lin Linux 2.1.132   226    13    137         890
arm32-net   NetBSD 1.3F   224     8    334         347    No L2 cache?
armv4l-li   Linux 2.2.9   233     -      -           -    Bad mhz?
armv4l-li   Linux 2.2.7   275     7    278         289    No L2 cache?
armv4l-li  Linux 2.0.31   274     0    270         270    No L2 cache?
armv4l-li  Linux 2.2.10   275     7    269         279    No L2 cache?
armv4l-li  Linux 2.2.10   275     7    269         279    No L2 cache?
armv4l-li   Linux 2.3.9   275     7    269         280    No L2 cache?
armv4l-li   Linux 2.3.9   275     7    269         279    No L2 cache?
i586-linu   Linux 2.2.6   100    20    123         193
i586-linu Linux 2.2.0-p   133    15     80         147
sparc-lin Linux 2.1.130    33    90    888         889    No L2 cache?

---

From: "Neil A. Carson" <[email-masked]>
Date: Wed, 14 Jul 1999 09:04:40 -0700

It's interesting; in the past when doing these benchmarks, things like
the pipe throughput and local latencies became totally shadowed by ctx
sw times. I remember once a test where the remote latency was smaller
than the local one :-)

	Neil

---

From: Philip Blundell <[email-masked]>
Date: Wed, 14 Jul 1999 17:28:09 +0100

>sw times. I remember once a test where the remote latency was smaller

Yeah.  None of the lmbench programs I looked at make any attempt to factor out 
the context switch time from what they're measuring.  The tough bit is that 
faced with results like:

it's hard to know how much of the blame for the 632us AF_UNIX latency you can 
lay at the door of context switching, and how much you are picking up 
elsewhere.

BTW, there is of course another bad effect of the ARM cache that the 
comments in the code don't mention.  Because it's virtual mapped you have to 
flush the whole thing on every context switch -- even if you could do this 
instantaneously, as you pretty much can on non-StrongARM machines, both the 
kernel and the newly-running user program are starting out with completely 
cold caches, so you get desperately bad performance for a while as everything 
gets reloaded from main memory.  This might actually make it worthwhile to 
think about adding to the ARM6/7 support some of the code to work out when 
we can get away without flushing the cache and TLB.  I will investigate that 
when I can find some time.

p.

---

From: "Neil A. Carson" <[email-masked]>
Date: Wed, 14 Jul 1999 09:54:47 -0700

One thing that I was considering was making all binaries effectively
PIC, and placing them at different points in the address space.

An idea probably hated by unix hackers in general, but for small sets of
processes it seemd an interesting way of preserving the cache between
some context switches.

Eg if we only allow a process 512MB of VA space, we can map each new
pmap (sorry, a BSDism although I don't work on that any more either) at
say 4KB + (512 * (i%6)) or something like that. Even with 1GB per proc
you'd have enough room for 3...

Thoughts?

	Neil

---

From: <[email-masked]>
Date: Wed, 14 Jul 1999 12:26:58 -0500 (CDT)

> you'd have enough room for 3...
This sounds like it would be possible.  Another idea a colleague at work
suggested was to make everyone share the same virtual address mappings,
i.e. you don't have to flush the cache on a context switch.  There would
be one set of virtual mappings that everyone gets mapped to.

Of course, another option (an idea even more hated than Neals, I'm
guessing:) would be to make Linux be able to handle a flat address
space.  Then you can use the MMU for protection only, and have two sets
of page tables, one for the kernel, and one for everyone else.  User
processes can then splat each other, but not the kernel.  This makes the
system somewhat more functional than MS-DOS or MAC-OS.:)  Not a popular
option, but for some occasions, this might be nice. (i.e. embedded
systems running Linux).

---

From: "Neil A. Carson" <[email-masked]>
Date: Wed, 14 Jul 1999 10:34:56 -0700

> be one set of virtual mappings that everyone gets mapped to.

I don't get you; how does this differ from what we have at the moment?
Everything gets mapped to one address...

> option, but for some occasions, this might be nice. (i.e. embedded

This makes sense, but what makes even more sense is to run with a proper
CPU for the target :-) Running multi-user UNIX systems on things like
strongarms is, IMHO, scaling them up too much if they are used in a
scalable (ie non-UI or something like web serving) context.

Why not just use an SH-4 or something?

	Neil

---

From: Philip Blundell <[email-masked]>
Date: Wed, 14 Jul 1999 19:21:28 +0100

>you'd have enough room for 3...

Yes, I've thought about this in the past.  In fact by putting each task in a 
different domain you could avoid even having to flush the TLB.  I think it 
would always be a bit of a specialist interest and it's probably not suitable 
for "mainstream" kernels but it's certainly an idea worth pursuing if someone 
has the time and inclination.  Compiling everything as PIC would obviously 
cost a small amount but it's not the end of the world.

p.

---

From: Russell King - ARM Linux Admin <[email-masked]>
Date: Wed, 14 Jul 1999 20:00:57 +0100 (BST)

> got 200us slower.  Anyway...

2.3.9/2.3.10 are both seriously broken as far as the memory management,
page cache, and buffer cache go.  I wouldn't recommend running it r/w
on any filesystem whatsoever, unless you'd like to loose it.

Certainly both the loop and ramdisk devices just plain old do not work
(files read wrong data).

This is why I haven't rushed to do the 2.3.10 release yet.
   _____
  |_____| ------------------------------------------------- ---+---+-
  |   |        Russell King       [email-masked]      --- ---
  | | | |  http://www.arm.linux.org.uk/~rmk/armlinux.html    /  /  |
  | +-+-+                                                     --- -+-
  /   |               THE developer of ARM Linux              |+| /|\
 /  | | |                                                     ---  |
    +-+-+ -------------------------------------------------  /\\\  |
unsubscribe: body of `unsubscribe linux-arm' to [email-masked]

---

From: <[email-masked]>
Date: Wed, 14 Jul 1999 15:22:50 -0500 (CDT)

> Everything gets mapped to one address...
Oops!  I guess I meant pretty much the same thing as you said
previously.  I'm new at this stuff, and still getting my feet wet.
Basically, I was refering to have all processes mapped into the same
address space, but using the MMU for protection and such.  All of the
process's have the same physical<->virtual address mappings, they just
differ on which mappings are real.

I had another thought: When you go to fill a line in the cache
(32bytes), it's always filled a line at a time.  I haven't looked at the
Linux cache flushing code yet, but does it read all 32 bytes for each
cache line fill?  Would it be possible to speed up the cache flushing by
reading just one word, which would fill a cache line, and then skipping
ahead 32 bytes and reading another word, thus filling another cache line?
Instead of having 8 reads per cache line fill, you would now only have
1.  Just a thought.

---

From: <[email-masked]>
Date: Wed, 14 Jul 1999 15:43:32 -0500 (CDT)

On Wed, 14 Jul 1999 [email-masked] wrote:

Scratch the below, I just looked at the code and it appears to already
do this.  Oh well, it was worth a shot!:)