After the hardware was assembled I made a quick attempt to boot Nevara 105 from a DVD in order to see how things went (the installation proper will be made via the network).
The system booted, but was very slow. It took over 10 minutes to get to the first prompt (which asks about the kind of installation you want to perform, and which is usually reached in a few seconds). Older releases and OpenSolaris 08/11 behaved the same. A Linux system booted from an USB stick behaved normally, though.
I wrestled with this for two days, but then noticed something while running memtest86+ on the system. Since the system has 4GB of RAM, and quite a lot of the physical address space between 3 and 4 GB is used by PCI devices, quite a lot of physical memory is remapped to physical addresses above the 4GB mark. memtest shows the start and end address of the block it is currently testing (this is why start and end addresses above 4GB can show up there, even if the system has less than 4GB of RAM). While testing the relocated memory block memtest slowed to a crawl, while the memory below 4GB was tested at normal speed. It looks like memory accesses above 4GB are not covered by the processor cache. Linux seems to put it's kernel below this magic mark, and thus runs normally, while Solaris lands above it, and is less than usable.
h3. Memory layout
In order to explain what is going on here (and why it is bad) a small detour is in order.
Physical address space is a shared resource on most architectures, the Intel x86 platform (in 32 and 64 bit) included. It is shared between real, physical memory and IO memory. Physical memory is the kind that comes (usually and these days) in DIMM memory sticks that go into the appropriate slots on the main board. IO memory, on the other hand, is a way of talking to extension cards, for example network adapters and graphics cards.
These adapters register one or more areas of memory with the BIOS during system startup. Accessing these adresses results in reading or writing to these extension cards instead of real memory.
The use of this is that programs can treat extension cards just like normal memory.
The end result of this is that a given physical memory address can have one of three "backgrounds":
* A physical memory cell
* A device
* Nothing
The memory ranges that devices register usually live between the 3GB and 4GB physical address space. This was all well and good as long as practically no system had that much real memory, so there was no contention for address space.
However, two things happened in the last years: memory got incredibly cheap, and devices got more hungry for memory space. Modern graphics cards, for example, map a large chunk of their on board graphics RAM into the memory space, sometimes all of it, easily taking up half a gigabyte or more of address space.
There are two ways to handle address conflicts in this situation.
The easy way is to simply ignore the physical memory in the address ranges claimed by devices. The physical memory cell becomes inaccessible, and the storage it provides is lost. This is obviously not a popular solution.
The other way is to relocate the physical memory from the contended address spaces into non-contended space. This usually means "above the 4GB border". So although you may only have 4GB of physical RAM in the system some of it must be accessed at addresses above 4GB.
The exact layout of the memory is passed from the BIOS to the system in the so-called E820 memory maps.
On the MSI IM-GM45 board with 4GB of memory installed it looks like this:
BIOS-e820: 0000000000000000 - 0000000000099000 (usable)
BIOS-e820: 0000000000099000 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000bdc80000 (usable)
BIOS-e820: 00000000bdc80000 - 00000000bdc8e000 (ACPI data)
BIOS-e820: 00000000bdc8e000 - 00000000bdcd0000 (ACPI NVS)
BIOS-e820: 00000000bdcd0000 - 00000000bdce0000 (reserved)
BIOS-e820: 00000000bdcec000 - 00000000bde00000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 000000013c000000 (usable)
The lines marked with (usable)
represent physical memory, the others are memory ranges reserved for other purposes.
The last line shows the physical memory that has been relocated above the 4GB memory barrier. Almost one gigabyte of physical memory has been relocated.
h3. Processor caches
These days there are multiple layers of caches between the CPU core and the main memory. Data read from memory is held in the caches, for it might be needed again soon, and data written to memory is held for the same reasons.
Data written to memory addresses beloning to devices, however, may or may not be eligible to caching. While memory is expected to keep it's content stable (unless explicitly written to) devices may change the content of their memory maps as they see fit, and caching the values read or written would screw with the CPUs world view.
CPUs therefore contain a list of memory ranges and the cache policies associated with those ranges. These lists are called memory type range registers (MTRR) on Intel CPUs. Below is the list from a different system, also with 4GB of physical memory:
reg00: base=0x100000000 (4096MB), size= 512MB: write-back, count=1
reg01: base=0x120000000 (4608MB), size= 256MB: write-back, count=1
reg02: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
reg03: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg04: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1
reg05: base=0xcff00000 (3327MB), size= 1MB: uncachable, count=1
Three quarters of the memory range between 3 and 4GB do not appear on this list, and are thus considered uncacheable by the CPU. The memory above 4GB, however, is marked as cacheable (reg00
and reg01
).
Now the same list on the MSI system:
reg00: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
This is wrong in several ways, but the most important one is that the physical memory above 4GB is no longer cached by the CPU. The result of that is access to that memory becomes painfully slow.
This also explains why the problem goes away when only 2GB of memory are present: no physisical memory is relocated beyond the 4GB barrier, because not enough address contentions exist, so all physical memory is being cached again.
I have filed a support request with MSI and hope someone there understands the problem.
h3. [Update 1]
After a short discussion about supported operating systems, MSI has sent me a newer BIOS file. I'll try that this evening.
h3. [Update 2]
The new BIOS does indeed fix this issue. The new MTRRs look like this:
reg00: base=0x13c000000 (5056MB), size= 64MB: uncachable, count=1
reg01: base=0x00000000 ( 0MB), size=4096MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg04: base=0xbdd00000 (3037MB), size= 1MB: uncachable, count=1
reg05: base=0xbde00000 (3038MB), size= 2MB: uncachable, count=1
reg06: base=0xbe000000 (3040MB), size= 32MB: uncachable, count=1
There are still some weird edges about this, but by and large it does what it should: all memory (minus that for the onborad graphics) is cached now.
My thanks to the MSI support for the fast (and working) response.
I have to say that I am very pleased with the technical support I have received so far from MSI for my IM-GM45. Besides the MTRR issue I wrote about here, I had filed a second, minor request. The second request regarded the shared video memory setting
Tracked: Feb 13, 14:17