[OpenIndiana-discuss] Recommendations for fast storage

[Günther Alka]

I would think about the following

- yes, i would build that from SSD
- build the pool from multiple 10 disk Raid-Z2 vdevs,

- use as much RAM as possible to serve most of reads from RAM
example a dual socket 2011 system with 256 GB RAM

- if you need sync writes/ disabled LU write back cache, use dedicated 
DRAM based log-devices (ZEUSRAM)
For multiple 10 Gbe you may need several of them or disable sync/enable 
LU cache when possible,
I would calculate one ZEUSRAM per 10 GBe adapter (about 2000$ each), 
analyze ZIL usage first.

- if possible, avoid expander with Sata disks

- do not fill a pool above 50% if you need max performance
read about fillrate vs throughput: 
http://blog.delphix.com/uday/2013/02/19/78/

- tune ip (Jumboframes, MPIO, Trunking) and iSCSI blocksize

- think about using OmniOS (a little more up to date than OI)


The rest is some math,you need:

a case (like a 50 x 3,5" bay Chenbro or a up to 72 bay SuperMicro)
with a 7 x pci-e mainboard, CPU, RAM, 3 x SAS2 HBA controller,
4 x dual 10 Gbe adapters:

ex: Chenbro 50 x 3,5" case without expander:
4 x dual 10 Gbe + 3 x LSI 16 channel HBA

or Supermicro cases with expander, up to 72 x 2,5" bays
with up to 3 x 8-16 channel HBA

         say 10000 $


The rest is for SSD and ZIL
If you like to use 10 TB and want to have 20TB capacity for performance 
reasons:
with your 800GB Intel, you have about 6,5 TB usable for 10 disks (Z2)
You need 30 of them ex 2000$ per SSD: 60000$ (without ZIL and spare),

gives a total o 70000$ without ZIL and spare.

other Option:
use 500-600 GB SSD like Intel 320 or 520.
You need more of them but they are cheaper regarding TB/$

allow 80% SSD usage, check ARC usage to eventually reduce amount if SSD
(RAM is cheaper than using only 50% of SSD capacity)

keep enough slots free to optionally add more SSD for better performance 
or higher capacity
care about needed capacity for snaps
add 10% spare disks.




On 14.04.2013 17:15, Wim van den Berge wrote:
> Hello,
>
>   
>
> We have been running OpenIndiana (and its various predecessors) as storage
> servers in production for the last couple of years. Over that time the
> majority of our storage infrastructure has been moved to Open Indiana to the
> point where we currently serve (iSCSI, NFS and CIFS) about 1.2PB from 10+
> servers in three datacenters . All of these systems are pretty much the
> same, large pool of disks, SSD for root, ZIL and L2ARC, 64-128GB RAM,
> multiple 10Gb uplinks. All of these work like a charm.
>
>   
>
> However the next system is  going to be a little different. It needs to be
> the absolute fastest iSCSI target we can create/afford. We'll need about
> 10-12TB of capacity and the working set will be 5-6TB and IO over time is
> 90% reads and 10% writes using 32K blocks but this is a data analysis
> scenario so all the writes are upfront. Contrary to previous installs, money
> is a secondary (but not unimportant) issue for this one. I'd like to stick
> with a SuperMicro platform and we've been thinking of trying the new Intel
> S3700 800GB SSD's which seem to run about $2K. Ideally I'd like to keep
> system cost below $60K.
>
>   
>
> This is new ground for us. Before this one, the game has always been
> primarily about capacity/data integrity and anything we designed based on
> ZFS/Open Solaris has always more than delivered in the performance arena.
> This time we're looking to fill up the dedicated 10Gbe connections to each
> of the four to eight processing nodes as much as possible. The processing
> nodes have been designed that they will consume whatever storage bandwidth
> they can get.
>
>   
>
> Any ideas/thoughts/recommendations/caveats would be much appreciated.
>
>   
>
> Thanks
>
>   
>
> W
>
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss