Motherboards ASUS Z9PE-D16 Review/ Builld

Update: 29 April 2013

Since I had the opportunity to build a dual processor server for a client on the new ASUS Z9PE-D16 motherboard for dual LGA 2011 processors, I decided it would be worth taking some pictures, and maybe writing a review of the motherboard to share my thoughts and my experience building the system. I built this system several months ago, and it’s been interesting to note how I have been learning my camera over that time, some of my shots came out very out of focus, lighting could have been better, but here is what I have:

Since I had the opportunity to build a dual processor server for a client on the new ASUS Z9PE-D16 motherboard for dual LGA 2011 processors, I decided it would be worth taking some pictures, and maybe writing a review of the motherboard to share my thoughts and my experience building the system. I built this system several months ago, and it’s been interesting to note how I have been learning my camera over that time, some of my shots came out very out of focus, lighting could have been better, but here is what I have:

First, the specs of what I was building:
Norco RPC-4216 4U Rackmount EATX Server Chassis
16 Hot Swap SAS or SATA Drive Bays
Norco 3 120mm Fan Wall
3 High Performance 120mm Mid Mounted Fans
2 High Performance 80mm Exhaust Fans
Athena Power Atlas 800 Plus 800W Redundant 80+ Bronze Power Supply
ASUS Z9PE-D16 EATX Server Motherboard
Intel C602-A Chipset
PS2 Keyboard and Mouse Port
4 USB 2.0 Ports (1 Front, 1 Internal)
2 SATA 6Gbps Ports
4 SATA 3Gbps Ports
Intel Matrix RAID 0,1,5,10 Support
Aspeed AST2300 Graphics Chipset
VGA Port
Quad Intel i350 Gigabit Ethernet Adapters
2 PCIe 3.0 x16 Expansion Slots
4 PCIe 3.0 x8 Expansion Slots
PIKE Expansion Slot
ASUS ASMB6-IKVM Management Chipset
Dual Intel Core Xeon E5-2620 Hex Core Processors
2.0GHz, 15MB L3 Cache, 2 6.40GT/s QPI, HyperThreading, 95W
Thermaltake 92mm 4 Heatpipe Horizontal CPU Coolers
32GB 8 Channel DDR3 1333MHz Registered ECC System Memory
8 Kingston 4GB 240 Pin DDR3 1333MHz Registered ECC CL9 RDIMM RAM
Hitachi Deskstar 7K1000.D 2TB 7200RPM 32MB Cache SATA 6Gbps 3.5in Desktop Hard Drives
Maximum Read Speed: 185MBps, Maximum Write Speed: 182MBps
9TB Array Moved from an Existing File Server
Existing HighPoint RocketRAID 640 4 SATA 6Gbps PCIe 2.0 x4 RAID Controller Card
Existing 4 Hitachi Deskstar 3TB 7K3000 7200RPM 64MB Desktop Hard Drives
Norco 4 SATA to SFF-8087 Reverse Breakout Cable
Maximum Read Speed: 477MBps, Maximum Write Speed: 474MBps
HighPoint Technologies RocketU 1144A 4 USB 3.0 PCIe 2.0 x4 Controller Card
Plextor L890SA SATA LightScribe SuperMulti Optical Drive
Microsoft Windows Server 2008 R2 Standard (64bit Architecture) Operating System
5 Client Access Licenses
Nero Essentials and Power DVD Optical Disk Solution
Internet Software Pack:
Adobe Reader, Air, Flash, and Shockwave
Oracle Java Runtime Environment
Microsoft Silverlight
Three Year Limited In Shop Hardware Warranty, Lifetime Limited Technical Support
In Shop Configuration including Application Installation, Onsite Delivery and Installation

I ordered the board about 2 months before it was available, and sat back twiddling my thumbs until it shipped. Once I got it, I snapped some pictures of what arrived:

The Processors:



The Memory:


The Midplane and Breakout Cable:


Optical Drive:


Chassis:


Power Supply:


Motherboard:

 

Cooling the processors presented a rather unique challenge, LGA 2011 processors don’t ship with cooling solutions, and I didn’t think any of the 120mm solutions available would fit inside the 4U case. I had done some arithmetic before ordering, and I thought maybe I could fit a Thermaltake Contac29/30 in if I was lucky. I held off on ordering cooling until I took measurements and figured out what would work.

Case Opened:


SFF-8087 mSAS Connectors on Backplane:


Measuring the Depth of Chassis:


Long Standoffs:


Due to the longer than normal standoffs, I really didn’t have room for any 120mm cooler. One 92mm cooler was available specifically for servers, but it was out of stock in most places. After a lot of research discovering that the Contac21 by Thermaltake is rated for LGA 2011 as it is compatible with LGA 1366 and is considered a viable cooler for ASUS flexible mounting for LGA 2011, I chose to order the cooling and sort out how to mount it after it arrived, secretly hoping the included mounting screws would match the thread of the LGA 2011 mount.

Coolers:


Fans:


Cooling Surface to CPU Surface Size Comparison:


Cooler Thread Comparison:


As you can see, the threads did not match anything usual. The threads for the mount are M4, in the picture above the threads are M3 to the left and 6-32 (close to M4) to the right, tough luck for me. I started to experiment with different lengths and types of 6-32 screws to see if I could get anything to work with LGA 2011, but I was leery of trying anything too long because directly beneath the mounting holes is the motherboard surface, no holes through. Eventually I tried some standard motherboard standoffs, and they fit perfectly, not too long and just the right height, though I had to flip the brackets on the cooler to maintain pressure.

 

Assembly Pics:

 

Testing the Cooler Assembly:




I wish I could have kept the thumb screws, they look good, but I only had 4, not 8. Thermaltake needs to sell this kit though, little thumb screws and some nice silver standard size standoffs, maybe even tension springs on the screws.

Using the direct contact cooler, I applied the TIM to the cooler rather than the CPU (I am one of those BB sized beads on the CPU heat spreader TIM application type of guys) and mounted them up.

 

With that installed, the next step was to fill in the remaining components and install the cooling. I used an upgraded 120mm midplane and Rosewill blue LED fans to replace the stock, screaming but small delta fans. I was quite impressed with how much airflow the Rosewills push through and the quality of their construction. Unfortunately, as Rosewill is owned by Newegg, I don’t think they’re available in the UK. They also didn’t have 92mm fans, so I picked up the only blue LED ones available. Why blue fans you might ask? Because I think that subtle glow out the exhaust of a computer and in the cracks of the case is awesome, and that is all. I also had to use SATA power to Molex 4 pin power adapters to properly fill the 8 Molex power connectors on the backplane. Just in case something comes unplugged I alternated the adapters from side to side so one could become unplugged without taking the system down.

 

On boot I encountered an issue with the USB 3.0 card I had used having an issue in POST, I removed it until ASUS updated the BIOS, which I can report resolved the issue.



Once the system was up and running, I installed Windows Server 2008 R2 Standard, and to be sure of my cooling setup, I ran Prime95 and SpeedFan to stress the system and measure the temperatures. To be honest, it is the coolest and most stable server I’ve seen. Using an enthusiast cooler on a server system really has its benefits. Temperatures below are in Fahrenheit:



Sadly, I don’t have pictures of the installation on site, I can tell you it is running happily away as a domain controller for a small office of 15 computers or so and performing admirably with huge file transfer operations and indexing.

I was quite impressed with the Z9PE-D16, once the card compatibility issue was resolved in BIOS. The issue was most likely due to the use of a PCIe splitter to split the PCIe x4 lane into 4 PCIe x1 lanes for the USB 3.0 card (which supports 4 simultaneous USB 3.0 5Gbps connections, useful for the external hard drives and SSD the client uses). The board has support for the future use of a big, 16-24 port RAID card to move the OS off the onboard RAID. There are issues with the boot process with HighPoint RAID and the Intel RAID, which have been going on for a long time. The solution is to remove the 4 HighPoint RAID drives when booting and install them after Windows is running. I wish my client had listened to me and sprung for a hardware controller for all of the drive bays… next time…

Let me know if this sort of review/ build log is appreciated Hopefully it is beneficial to people building dual processor systems for their own use.

 

Looks pretty cool, although Ive never heard of most of the hardware

One question, due to the fact that servers are usually always on, how come the power supply wasn't very efficient?

 

80 Plus Bronze is still pretty efficient. Unfortunately, since non-OEM (non-Dell/HP/etc.) redundant power is a fairly small niche, the more efficient ratings take a while to trickle down while Dell and HP have big enough pockets to adopt the standards right away.

 

This is a really incredible motherboard, nice rig there

Which case did you use for the review ?

 

Quite a nice bit of kit.

 

It's in the specs. Norco RPC-4216. I am really quite pleased with it, though I personally would have gone for the RPC-4220 or RPC-4224 rather than waste the space on a full size optical drive, but it costs a bit more. One major downside I noticed is that because the optical drives come with built in flaps, if you were to fill one with a 4x2.5in drive bay, which I intend to do in this server down the road with SSD, it might sit back from the opening, unless it has multiple holes for mounting forward or back. Ideally I would use this, but the holes won't line up:


All the usual companies, Icy Dock, IStarUSA, Startech, they all would sit back 1/2in into the case because of that poor design... other than that, the Norco case rocks.

Edit: Also, solution:

 

Never owned a CPU cooler with exposed pipes, but is that really how you're supposed to apply TIM? Seems like a lot!

 

Hey man why don't you add a bunch of Delta high-rpm fans

In my server there are

1- 92*92*38 5700rpm in the front

1- 120*120*38 4500 rpm fan in the rear

 

Most of your pic links are broken.

 

Looks like my kind of hardware ...to a certain extent.

 

No! That is a ridiculous amount of TIM. Your cooling capability is likely being compromised, and I bet a load has squidged out the side, possibly onto key components.

 

Is this true for anyone else? The pics are hosted on my site, I have no problem getting to them and I haven't gotten any reports of issues from my server or hosting company (like bandwidth limits or anything).

It seems like a lot for me too, like I said, I'm a "dot the size of a BB" sort of applicator guy, but the truth is in the doing. The TIM included with these direct contact coolers is actually much thinner than standard TIM, and the layer on the bottom of the cooler is actually fairly thin, when I remove it, there are no big squished zones on the outside of the CPU contact area, so it's not squishing like you think. The big thing I've found when applying TIM to direct contact coolers is to prepack the crevices between the heatpipes and baseplate, this was the best way I found to do that. Prime95 for 3-4 hours and only hitting 115°F (46°C), that's the lowest stable temperature I've ever seen.

Also, I built a system for a friend with a Contac30 direct contact cooler, I tried dots between the heatpipes, a dot on the CPU heatspreader, lines between the heatpipes, and finally this, temperatures were 5°C lower with the final method of application. I'd say it's a one off situation, but I have built maybe a half dozen systems with direct contact cooling since, all of which have great temperatures, compared to their traditional BB applied non-direct contact cooled companions.

Lol, I removed the fans included with the case 2x 80x38mm in the rear and 4 80x25mm at the midplane. This server is going in the office itself, not a server closet, so unbearable screaming noises were not preferred. I did actually have to remove the blue fans though, this server is for a somewhat covert law enforcement location, so a blue glow in the windows at night was to be avoided.

 

Thankyou for sharing.

 

Thats a fairly beefy looking system right there

 

Pictures are working now by the way, think my connection had a brainfart because there were quite a few of them!