[OpenIndiana-discuss] Shopping for an all-in-one server

[jason matthews]

On Jun 2, 2014, at 12:38 AM, Jim Klimov <jimklimov at cos.ru> wrote:

> The box should be a reliable rackable server with remote management, substantial ECC RAM for efficient ZFS and VM needs (128-256gb likely, possibly more), CPUs with all those VT-* bits needed for illumos-kvm and a massive amount of cores (some large-scale OS rebuilds from source are likely a frequent task), and enough disk bays for rpool (hdd or ssd), ssd-based zil and l2arc devices (that's already half a dozen bays), possibly an ssd-based scratch area (raid0 or raid1, this depends), as well as several TB of HDD storage. Later expansions should be possible with JBODs.


SmartOS might be a good match for your VM issue. 

As far as servers go, I am using Dell r820 with 64 vcores (4x E5-4650L)  and r720 ewith 48 vcores (2x e5-2697v2). Plenty of cores. Each chassis offers 16 2.5inch bays which populate with Intel DC S3700 800gb drives.

I pull the MegaRAID cards out of the Dells and replace them with LSI 9207-8i HBAs. You can return the MegaRAID cards after the fact but you can’t order the systems without them. Hardware compatibility is 100%. Avoid expanders/port multipliers. In the case of dell, avoid the r720xd unless you are using SAS as that uses an expander to get to 24 drive bays.

The HP RAID cards are also LSI derived but probably not compatible so plan on using a pair or more of 9207-8i or similar. HP RAID cards also require drives to be in some kind of raid container, there is no pass through.

Don’t mix SAS and SATA. If using SATA for primary storage, I would use the DC S3700s. If I had SAS for primary storage I would use the HGST SSDs. If you have to have both, then use separate HBAs.

The 9207 is rated at like 700,000 IOPS. Consider taking that with a grain of salt.  The DC S3700 800gb drives can do, in my testing 22k operations per second (Intel rates at 20k/s) so you would need five stripes to break 100k IOPs. If using mirrors, mirror across controllers. I would worry too much about over loading a 9207 unless you know you are going to sustain a few hundred thousand IOPs.

I am also fond of external disk chassis. Historically I have used Newisys/Sanima NDS-2240 chassis which work great. If you order the drives from Newisys they come pre-burned with (in my case) no infant mortality.  I am about to order a handful of Intel made JBOD chassis to give those a whirl.