Re: [Hampshire] Kernel Memory Models

On Tue, Apr 29, 2008 at 08:58:23PM +0100, Adrian Bridgett wrote:
> On Tue, Apr 29, 2008 at 19:59:37 +0100 (+0100), Hugo Mills wrote:
> > On Tue, Apr 29, 2008 at 07:50:36PM +0100, Stephen Davies wrote:
> > > I was at a presentation today where the speaker said that one of the
> > > differences between SLES & RHEL was that they use a different way to
> > > allocate memory.
> > > He said that SLES uses a NUMA model whereas RHEL uses a 'flat' model.
> > > He indicated that the NUMA model gave a slightly better performance that
> > > that used by RHEL. This was observd in a DB Benchmark test. TPC-H
> > >
> > > Can anyone verify this statement?
> > > What do other stock kernels use and can the NUMA model be enable of a
> > > RHEL Kernel?
> >

> >    If I recall, it's a compile-time option, so you'd have to rebuild
> > your kernel.

> >

> >    I believe that multi-socket Athlon64 machines would benefit from a
> > NUMA kernel, because of the memory architecture typically implemented

>
> I remember seeing an Opteron talk a while ago (whilst they were still
> emulating it in software....) where they described it as SUMA
> (sufficently...) which I think fell somewhere between the two. It
> would benefit from NUMA, but it wasn't all that important.

I think it's NUMA, but the differences due to the non-uniformity
are sufficiently small that you can ignore them with small cost. Of
course, as you scale your systems up, that assumption becomes less and
less relevant. On a 2-socket machine, you probably won't notice. On an
8-socket machine, I'd guess it's a bit more noticable, and on a hunk
of really big iron, you probably both want and need the NUMA.

> >    I'm a little further adrift on the current Intel chips, but I
> > believe they're still using a flat memory architecture on the current
> > generation, and will be moving to something more NUMA-like fairly
> > soon.

>
> I believe the most recent ones are different now, but anything older
> is just chips in sockets - however there can be bus-contention between
> the cores across the bus.

The point there is that it's symmetric/uniform bus contention. The
latency to any one piece of RAM is the same from all CPUs. NUMA is the
case where the latency to a piece of RAM is different depending on
which CPU wants it (which it is, slightly, in the Athlon64/
Hypertransport case for multiple sockets).

From what I remember, Intel are planning on moving to a HT-like
CPU interconnect in either one or two generations (I think it's one).

> Only one way to tell for sure - try it and benchmark it for _your_
> setup.

Indeed. And use _your_ workloads.

Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- The English language has the mot juste for every occasion. ---