[OpenIndiana-discuss] Recommendations for fast storage

[Sašo Kiselkov]

On 04/14/2013 05:15 PM, Wim van den Berge wrote:
> Hello,
> 
> We have been running OpenIndiana (and its various predecessors) as storage
> servers in production for the last couple of years. Over that time the
> majority of our storage infrastructure has been moved to Open Indiana to the
> point where we currently serve (iSCSI, NFS and CIFS) about 1.2PB from 10+
> servers in three datacenters . All of these systems are pretty much the
> same, large pool of disks, SSD for root, ZIL and L2ARC, 64-128GB RAM,
> multiple 10Gb uplinks. All of these work like a charm. 
> 
> However the next system is  going to be a little different. It needs to be
> the absolute fastest iSCSI target we can create/afford. We'll need about
> 10-12TB of capacity and the working set will be 5-6TB and IO over time is
> 90% reads and 10% writes using 32K blocks but this is a data analysis
> scenario so all the writes are upfront. Contrary to previous installs, money
> is a secondary (but not unimportant) issue for this one. I'd like to stick
> with a SuperMicro platform and we've been thinking of trying the new Intel
> S3700 800GB SSD's which seem to run about $2K. Ideally I'd like to keep
> system cost below $60K.
> 
> This is new ground for us. Before this one, the game has always been
> primarily about capacity/data integrity and anything we designed based on
> ZFS/Open Solaris has always more than delivered in the performance arena.
> This time we're looking to fill up the dedicated 10Gbe connections to each
> of the four to eight processing nodes as much as possible. The processing
> nodes have been designed that they will consume whatever storage bandwidth
> they can get.
> 
> Any ideas/thoughts/recommendations/caveats would be much appreciated.

Hi Wim,

Interesting project. You should definitely look at all-SSD pools here.
With the 800GB DC S3700 running in 3-drive raidz1's you're looking at
approximately $34k CAPEX (for the 10TB capacity point) just for the
SSDs. That leaves you ~$25k you can spend on the rest of the box, which
is *a lot*. Be sure to put lots of RAM (512GB+) into the box.

Also consider ditching 10GE and go straight to IB. A dual-port QDR card
can be had nowadays for about $1k (SuperMicro even makes motherboards
with QDR-IB on-board) and a 36-port Mellanox QDR switch can be had for
about $8k (this integrates the IB subnet manager, so this is all you
need to set up an IB network):
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idcategory=7&idproduct=158

Cheers,
--
Saso