[OpenIndiana-discuss] Cram more drives into an HP xw8600

[Philip Robar]

My apologies for taking so long to respond. I started a first draft of this
and then forget to finish it before going on vacation.

Phil

On Thu, Jul 24, 2014 at 8:28 PM, Harry Putnam <reader at newsguy.com> wrote:

> Philip Robar <philip.robar at gmail.com> writes:
>
> First, thank you sir for your repeated good input.
>
> > Personally, I want one of the tray less x in y bays just because I think
> > that they look cool. (I lust for this case, but am too cheap to replace
> the
> > used server I recently purchased on eBay for a great price with a new
> > mini-ITX motherboard, CPU, ECC memory and the case:
> > http://www.u-nas.com/cases.html)
>
> Not sure I really get it about that nifty looking case... does it have
> its own OS, or would your workstation os be running it?
>

While the maker sells preconfigured NAS boxes using these cases [1], I'm
only interested in the case, which comes with the hot-swap, tray less drive
bays and a power supply. The 2, 4 and 8 bay versions are $130, $150 and
$200 respectively. Unfortunately they only take mini-ITX motherboards and
the system I went with came with a micro-ATX motherboard.

I run FreeNAS for my home server. FreeNAS is FreeBSD based with an HTML GUI
for configuration. The server's primary purpose is to be a ZFS file server
with me as the only user. I chose FreeNAS over some variation of Solaris or
illumos [2] (even though I'm an ex-Sun [RIP] employee) because I had used
it in the past and liked it for its ease of use (though it's GUI really
needs the loving attention of someone who knows something about user
interfaces) and for it's home and multi-media oriented plugins that I might
want to use in the future.

Another advantage of FreeNAS, which I've never seen mentioned, is that you
can save its basic configuration in a single file like you would a router.
This makes for easy disaster recovery and allows one to play with alpha and
beta releases and then easily roll back to a stable release just by
reinstalling and loading the config file.

May I ask how much your ebay purchase cost you? I probably shouldn't
> but can't help but wonder how badly I got beat up.
>

I don't mind answering this at all. Hopefully others will find it useful as
well. The short answer is $140:

Supermicro 1U short case with 280 watt power supply.
Supermicro X7DVA-L motherboard

2 x LGA 771 CPU sockets
2 x Intel NICs
6 x SATA II (Plenty for my purposes)
6 x DDR2 memory slots for up to 48 GB of ECC memory (DDR2 ECC memory is
very inexpensive.)

Several PCIe and one PCI slots

Built in VGA & audio (Technically it's probably a workstation, not a server
MB.)

Room for two 3.5 inch drives internally. (No tray less slots, but I really
don't need them.)

2 x Xeon L5420 CPUs for 8 cores total
8 GB (2 x 4) DDR2 ECC RAM (leaving 4 empty slots)
1 x 500 GB WD Blue HD

Here's the long version if you're interested:

I wanted an inexpensive server for ZFS that had ECC memory. I was already
doing what many do for a DIY home NAS: repurposing a reasonably lowish
power old PC and getting by with only 4 GB of non-ECC memory, but I really
wanted to get that last bit of protection that ECC memory offers and I knew
that I was eventually going to need more memory for additional services
that I want to run. (N.B. I think that the warnings about using ZFS with
less than 8 GB of memory (some even say 16) given by some rather outspoken
people (one in particular) in the FreeNAS community are a rather overblown
if all you're doing is running a simple file server for a few people.)

After doing some research I found that it would cost me $250 [3] to ~$400+
[4] for current generation hardware that used ECC memory. After deciding
that I could live without some of the benefits of new hardware (very low
power usage, AES-NI instructions for on-the-fly encrypted filesystems, the
latest virtualization features, the latest vector instructions, Quick Sync
for transcoding, IPMI remote management) I started looking on eBay for used
options.

I first found a couple of variations of Dell servers that had been custom
built for early cloud servers that were coming onto the used market in
large quantities for less than $200. But if I recall correctly there was a
hardware incompatibility with FreeBSD that would have been to expensive to
work around.

I then searched on "supermicro" and "xeon" or "ECC", sorted by price and
found a bunch of 7 series Supermicro systems in 1U or 2U cases for very
reasonable prices. Some more research taught me about the low power (for
the time) L series Xeons that many of theses systems came with and in the
end I decided that the Xeon L5420 CPU was the best bang for my buck. The
Xeon L5420 has 4 cores, 50 W max TPD, ECC support, is 64 bits, has basic
VT-x support for virtualization and pretty good power saving features. It's
single core performance is about the same as an AMD Sempron 140 according
to GeekBench 3 (1400).

I eventually picked the Supermicro system I described above which is all
together more than good enough for a one person ZFS file server, maybe a
couple of BSD Jails (think Solaris Zone/Container) and eventually an
instance of Windows 8 in a VirtualBox as my media center. (After adding
more memory.)

The only real drawback to the system at the moment is that it's in a 1U
case so it's quite loud–even with the latest BIOS and the fans set on
quiet. I'm eventually going to have to put it in the basement and drill
some holes for cables, or use a different case.

As to the used, but not so power efficient CPU vs new, but low power with
new features CPU trade off (with back of the envelope calculations on my
part):

This system, as configured above but with three additional drives, uses
about 80 watts at idle. (Max TPD for an L5420 is 50 watts.) So at the
average US price for electricity it costs about $80 a year to run 24x7.

The optimal configuration for current low power hardware is an Intel Avoton
CPU/motherboard combo with 4 or 8 cores which starts at around $280. Add 8
GB of memory and we're up $360. An eight core version with 16 GB of RAM and
a bunch of drives will idle at around 40 watts I'm told.

So, just looking at out of pocket expenses (without trying to fairly value
each part of the system I bought vs a new system with it's additional
feature/performance and the PS and case that I already have that I would
put it in) we get $360 - $140 = $240 saved. The used system cost about $40
more a year to run. So it will take $240 / $40 = 6 years before I start to
lose money on the used system.

The lowest cost option for current hardware would be an open box server
class motherboard and a low end Haswell Pentium at $140 + $80 for memory =
$220. Power consumption jumps up to around 50 W TPD for the CPU and cores
go down to two with nearly double the single core performance, but half the
multi-core performance of two L5420s. I have no idea what a system like
this would idle at, but let's take a wild ass guess and say 60 watts. So:

$220 - $140 = $80 hardware savings with used system.
$80 - $60     = $20 higher annual operating cost.
$80 / $20 ---> 4 years before I start losing money on the used system.

Perhaps the fairest comparison: Lenovo TS140 with case, bronze power
supple, DVD, Core i3-4130 and 8 GB of memory would be about $300, but
probably $30 or $40 more unless you catch it at it's lowest ever sale
price. The Core i3-4130 is a little over twice as fast for single core
performance and has about 2/3 of the multi-core performance of two L5420s.
Amazon buyers report idle power usage of less than 20 watts for the base
system. So:

$300 - $140 = $160 (or more, see above) saved.

Despite a couple of the new hardware options being not much more expensive
than the used option, out of pocket costs was the overriding factor for me
at the moment so I went with the used system. The money I saved allowed me
to buy a couple of 3TB hard drives without too much quilt.

Another advantage of the used system is that it uses DDR2 ECC memory. Used
DDR2 ECC memory is dirt cheap.

So far I'm happy with my choice.

I bought the HP xw8600 with 2x Xeon (5470) cpus, 3.33 Ghz, ecc memory and 32
> GB of it.
>
> I paid $775. The case and all the rest are in really superb shape and
> it does seem to be a workhorse and powerhouse.
>
> So, was I robbed?
>

I'm not familiar enough with your hardware or what you intend to use it for
to have an opinion on that, though if you can't get the SAS part working
that will certainly take away from its value.


Phil


[1] It's Linux based and appears to be quite similar in features to what
companies like Synology and Thecus provide with their NAS boxes. It's only
$20 so it might be interesting to buy a copy to play with.

[2] While there are plenty of other options for a ZFS server like napp-it,
Eon (which I've used in the past) and Nexenta; as I said above I really
like FreeNAS for it's binary plug-ins. Another option, now that there's a
reasonably priced way to connect eSATA or USB3 drives to Thunderbolt, would
have been to connect my big drives to my iMac, which has 12 GB of memory,
and use Open-ZFS; but I really want the safety of ECC memory, which an iMac
can't provide.

[3] The Lenovo TS140 entry level model server is a great deal at $250—the
lowest price I've seen for it in the last few months.

[4] Current Supermicro board, 8-16 GB memory, some flavor of Haswell ECC
supporting CPU.