K8WE

From Gentoo Linux Wiki

[edit] General Information

• A Dual Opteron (2 x Socket-939/940), motherboard. The K8WE has support for virtually every hardware I/O Interface made,
  • (2 PCIe x 16, Firewire, PCI-X (133,100,33 (64 and 32 bit)), USB, SCSI ultra 320, SATAII, ATA-133 IDE, Floppy, GigE, audio, PS2, Serial, Parallel)

Detailed Spec :

• Chipset / Expansion
  • NVIDIA nForce nForce 2200 (CK8-04) (North) (On CPU1)
  • 2050 (South) (On CPU2)
  • Dual PCI-X busses.
  • Included is the AMD8131 (CPU2)
    • dual PCI-X 1.0 (1GBps) tunnel and AMD8132 a dual PCI-X 2.0 (2GBps) tunnel
    • each processor has its own I/O hub.
    • Translates to 2 full PCIe x16 lane slots with SLI support
  • 1 64 bit PCI-X 133Mhz Bridge A
  • 2 64 bit PCI-X 100Mhz Bridge B
  • 1 32 bit 33Mhz PCI v2.3
  • Two IEEE 802.3 Nvidia MAC 1000/100/10
    • eth0 on North and eth1 on South (connecting to eth0 can lead to kernel panics and crashes)
    • Jumbo, FullDuplex
  • TI TSB43AB22A IEEE 1394a
    • 2 connections (the internal pin header is non standard)
  • Watchdog (smsc47b397, advantechwdt)

• Memory
  • 2 local, glueless, full 168-pin directly on each processor 8 sockets
  • Non-Uniform Memory Architecture (NUMA)
  • Bi-directional system I/O interconnect known as HyperTransport
    • Supports up to 16GB ECC DDR266 DDR333 and DDR400
    • Optimal DDR400, CAS2.5-3

• I/O
  • LSI 53C1030 U320 SCSI
  • 2 68 pin connectors connected to PCI-X Bridge B

• Other
  • 2 dual USB2.0 EHCI 8 Ports
  • AC'97 2.3
  • PS2 Mouse/KB
  • 6 Fan headers

• Good resources

* Tyan manual download link:
* Community Site:
* Gentoo Forum thread:
* Blog-thread:

[edit] Known Good Configuration

• Not bragging, this is just how it is.
  • 16GB Fully populated DDR400 RAM
  • 3WARE 9550SX - 16ML
    • Fully populated with 16 drives, beware of IO wait states I would of gone with LSI if I were shopping around again

Large IO WAIT State with 3ware:

• I have had compleatly pinned IO scheduling on JFS with Deadline no preempt and only 1~2MB/Sec of throughput. Sync can elieveite some pain but on massive filesystems (+1.5TB) the link above is often correct about his testing even on a more recent reliese.
• I am running firmware from CodeSet 9.4.0.1, a more recent version is available

• Some pepople have experienced problems with both PCIe Slots being populated
  • I have dual GeForce 7800 GT
  • 1 Connected to a 30" Dell LCD
  • The second card runs 2 17" Sony's using nvidia-drivers xorg "Xinerama" "false" ( Composite and XINERAMA GLX NV-GLX RENDER ACCEL all TRUE)

• This is usually because of the BIOS setting I refer to in the BIOS section, X will fail
• Ensure /proc/mtrr hash each line set to "write-back"
  • Always try to seat one card at a time, then the other, or try reverse order

• I use the LSI320 for a boot disk
• Also connect the internal SATA to an External disk
• 18 Disks in total in use internally accessable, Firewire external PIONEER CDROM
• Cooling Zalman CNPS9500 LED Blue LED High Performance Quiet Cooler

Picture:

• These 2 fans operate as well as my Cooler Master watercooling did (until the hose broke and leaked!@!)
  • Never above 60'C @ 2.8Ghz

[edit] Caution

• Tyan published a warning about the 6 pin SSI EEB Power Connector that is not compatable with the 6 pin PCI-Express x16 standard Graphics Connector
  • Some non-polerized connecting plugs may fit the slot and damage the system
• The 6pin power connector on the S2895 can be left disconnected

Manual Addnum

[edit] BIOS

• Reset the BIOS and start over if your having trouble, After a default load, change these settings
• IOMMU!! This is a hotly contested setting
  • My Allocate is set to Auto
  • IOMMU Is Disabled
  • And the MEMHOLE is set to Hardware
    • This may only be possiable on newer Opterons, my last CPU's did not support this and forced me to use auto (probably software as hardware would not work well), This setting is also more costly in terms of Ram but is better performing over all
  • Make sure your using APIC, if your in PIC mode there will be lots of driver related issues that are difficult to track down.

• Keep changes to a minimum, though I usually set 1.1 legacy support for my USB keyboard and also "Self Healing" whatever that is :D
  • only need legacy support if USB keyboard or mouse is used
• Disable unused option ROM's
  • I have run out of option rom memory and could not boot as this system has so many overlays

• Make sure you calculate the amount of RAM you should have, ECC DIMMS will sighlently fail and as the number reporting your memory grows it get's more difficult to know how much you should have available.
  • This is for 16GB of RAM (Looks like 18:)
  • Memory: 16430916k/18350080k available (1838k kernel code, 344868k reserved, 822k data, 292k init)

• For a dual PCI-Express NVIDIA 7900GS setup, it was necessary to upgrade from BIOS v1.01 to BIOS v1.04. Before the upgrade the second was not recognized at all, or, if it was the only card but seated in the second slot (slot #3), produced serious video artefacts. After the upgrade, both cards were recognized without special kernel command line options using 2.6.20-gentoo-r8. For a single card setup BIOS v1.01 worked fine.

[edit] Failure to Boot

• Possiable caues

• It should boot, but it will not, despite harddrive BIOS priority settings
  • This can happen during install with USB/Firewire CDROMS (thouigh Firewire boot was removed in later BIOS)
• Insert a USB memory stick or two if a USB CDROM is not able to BOOT
  • This seems to be enough to kick in the boot loader (BIOS 1.05 with the "fix" DOES NOT work AT ALL for me)

• 1.05 gives me bootup problems

[edit] Kernel configuration

Linux localhost 2.6.20-gentoo-r6 #1 SMP Sat Apr 21 22:19:33 EST 2007 x86_64 Dual Core AMD Opteron(tm) Processor 290 AuthenticAMD GNU/Linux

[edit] NUMA Support

Processor type and features --->
 <*> /dev/cpu/*/msr - Model-specific register support
 <*> /dev/cpu/*/cpuid - CPU information support
 [*] MTRR (Memory Type Range Register) support
 [*] Symmetric multi-processing support
 [*] SMT (Hyperthreading) scheduler support
 [*] Multi-core scheduler support
 [*] Non Uniform Memory Access (NUMA) Support
 [*] ACPI NUMA detection
 [*] Page migration
 [*] Provide RTC interrupt
 [*] AMD MCE features
 [*] Function reordering

[edit] LSI 53C1030 U320 SCSI

Device Drivers --->
 Fusion MPT device support  --->
  <*> Fusion MPT ScsiHost drivers for SAS
 SCSI device support  --->
  <*> RAID Transport Class
  <*> SCSI device support
  [*] legacy /proc/scsi/ support
  <*> SCSI disk support
  <M> SCSI CDROM support
  [*] Enable vendor-specific extensions (for SCSI CDROM)
  <M> SCSI generic support
  <M> SCSI media changer support

[edit] SATA

Device Drivers --->
 Serial ATA (prod) and Parallel ATA (experimental) drivers  --->
  <M> ATA device support
  <M> NVIDIA SATA support

[edit] NIC

forcedeth.ko

• forcedeth.c: Reverse Engineered nForce ethernet driver. Version 0.59.
• full 1gb full duplex even though it is in the 100Mbit configuration page
  • Verified by switch LED's

Device Drivers --->
 Network device support  --->
  Ethernet (10 or 100Mbit)  --->
   <M> nForce Ethernet support

• Try alternate NIC if the system is unstable, eth0 has been known to panic
  • I would only move it if the problem occurs

[edit] Sound

• snd-intel8x0

Device Drivers --->
 Sound --->
  Advanced Linux Sound Architecture --->
   PCI Devices --->
    <M> Intel/SiS/nVidia/AMD/ALi AC97 Controller

[edit] USB

USB can cause bootup problems especially on 32 bit installs. Use noapic and/or nousb if the boot CD does not work.

• Also, LOAD Default BIOS Settings to fix CPU spinning when USB is in use
  • Seems to be fixed in more recent 2.6 releases

Device Drivers --->
 USB support  --->
  <M> Support for Host-side USB
  [*] USB device filesystem
  <M> EHCI HCD (USB 2.0) support
  <M> OHCI HCD support
  <M> USB Human Interface Device (full HID) support 

[edit] IEE1394 / Firewire

Works well, though it took later 2.6 releases to recognize the specific hardware ID of the firewire controller

Device Drivers --->
 IEEE 1394 (FireWire) support  --->
  <M> IEEE 1394 (FireWire) support
  [*] Build in extra config rom entries for certain functionality
  [*] IP-1394 Entry
  <M> OHCI-1394 support
  <M> SBP-2 support (Harddisks etc.)

1:05.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link) (prog-if 10 OHCI)

       Subsystem: Tyan Computer Unknown device 2895
       Flags: bus master, medium devsel, latency 64, IRQ 19
       Memory at a0104000 (32-bit, non-prefetchable) [size=2K]
       Memory at a0100000 (32-bit, non-prefetchable) [size=16K]
       Capabilities: [44] Power Management version 2

[edit] PATA

Standard settings apply

[edit] Sensors

Amazing Hardware monitoring support:

Device Drivers --->
 I2C Support --->
  <M> I2C Support
  <M> I2C Device Interface
 I2C Hardware Bus Support --->
  <M> AMD 756/766/768/8111 and nVidia nForce 
  <M> AMD 756/766/768/8111 and nVidia nForce 
  <M> AMD 8111  
  <M> Intel 82801 (ICH) 
  <M> Nvidia nForce2, nForce3 and nForce4
 Miscellaneous I2C Chip Support --->
  <M> EEPROM reader
 Hardware Monitoring Support --->
  <M> Hardware Monitoring support
  <M> AMD Athlon64/FX or Opteron temperature sensor
  <M> SMSC LPC47B397-NC

Standard lmsensors/libsensors configuration.

[edit] Other information

Bus options (PCI etc.)
 [*] Support mmconfig PCI config space access
 [*] PCI Express support
 [*] Message Signaled Interrupts (MSI and MSI-X)
 [*] Interrupts on hypertransport devices
 [*] Root Port Advanced Error Reporting support

[edit] Grub 32 bit

kernel /boot/vmlinuz root=/dev/sda1 ro apic apci=noirq iommu=memaper=3 swiotlb=65536

kernel /boot/vmlinuz root=/dev/sda4

[edit] make.conf USE FLAG

I live sort of dangeriously, you may not want to run so much experimental settings, but this is all fairly stable. No system crashes to the kernel, only occasional vmware-server problems but sshd always seems to accept my SYN

 
CFLAGS="-march=k8 -O3 -funroll-loops -pipe -fomit-frame-pointer"
ACCEPT_KEYWORDS="~amd64"
CHOST="x86_64-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j4"
FEATURES="ccache"
CCACHE_DIR="/mnt/raid1/ccache"
CCACHE_SIZE="50G"
USE="nv dri nvidia glx glitz pdf png cairo qt kde dvd alsa crypt cdr glib2 emacs 3dnow 3dnowext mmxext mmx sse sse2 sse3 X xcomposite pdf gtk gnome nptl nptlonly samba swat xinerama nsplugin arts -ipv6"
INPUT_DEVICES="keyboard mouse"
VIDEO_CARDS="nv nvidia vesa"
PORTDIR_OVERLAY="/usr/local/portage /usr/local/portage/xeffects/trunk /usr/local/portage/xeffects/experimental /usr/local/portage/crossdev"
  

[edit] lsmods output

 
localhost:~# lsmod
Module                  Size  Used by
loop                   63124  2
sr_mod                 18084  0
cdrom                  36520  1 sr_mod
sbp2                   24900  0
usb_storage            44164  0
nvidia               7760600  40
usbhid                 26080  0
hid                    23936  1 usbhid
snd_intel8x0           36520  1
snd_ac97_codec        107288  1 snd_intel8x0
ac97_bus                3392  1 snd_ac97_codec
snd_pcm                86088  2 snd_intel8x0,snd_ac97_codec
snd_timer              24776  1 snd_pcm
snd                    63656  6 snd_intel8x0,snd_ac97_codec,snd_pcm,snd_timer
soundcore               9504  1 snd
snd_page_alloc         10960  2 snd_intel8x0,snd_pcm
forcedeth              45000  0
sata_nv                21508  0
libata                112736  1 sata_nv
3w_9xxx                35396  2
ohci1394               35336  1
ieee1394              107064  2 sbp2,ohci1394
i2c_nforce2             6848  0
i2c_core               24640  2 nvidia,i2c_nforce2
ehci_hcd               31116  0
ohci_hcd               21252  0
usbcore               142192  5 usb_storage,usbhid,ehci_hcd,ohci_hcd
k8temp                  6912  0
hwmon                   4296  1 k8temp
sg                     34472  0
  

[edit] lspci output

 
00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)
00:04.0 Multimedia audio controller: nVidia Corporation CK804 AC97 Audio Controller (rev a2)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2)
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
00:0a.0 Ethernet controller: nVidia Corporation CK804 Ethernet Controller (rev a3)
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:05.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
02:00.0 VGA compatible controller: nVidia Corporation G70 [GeForce 7800 GT] (rev a1)
10:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
10:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
10:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
10:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
11:04.0 RAID bus controller: 3ware Inc 9550SX SATA-RAID
12:06.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
12:06.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
80:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
80:01.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
80:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
81:00.0 VGA compatible controller: nVidia Corporation G70 [GeForce 7800 GT] (rev a1)