[OpenIndiana-discuss] ZFS remote receive

[Richard Elling]

On Oct 31, 2012, at 5:53 AM, Roy Sigurd Karlsbakk <roy at karlsbakk.net> wrote:

>> 2012-10-30 19:21, Sebastian Gabler wrote:
>>> Whereas that's relative: performance is still at a quite miserable
>>> 62
>>> MB/s through a gigabit link. Apparently, my environment has room for
>>> improvement.
>> 
>> Does your gigabit ethernet use Jumbo Frames (like 9000 or up to 16KB,
>> depending on your NICs, switches and other networking gear) for
>> unrouted (L2) storage links? It is said that traditional MTU=1500
>> has too many overheads with packet size and preamble delays between
>> packets that effectively limit a gigabit to 700-800Mbps...
> 
> Erm… That's not true. IPv4 header is 20 bytes. TCP header the same, meaning 40 bytes in total out of 1500 bytes "payload", leaving 1460 bytes left for real payload, or 97.3%. An overhead of 20-30% is *not* correct. The ~3% overhead matches well what I see in practice on my networks. You will get a gain with jumboframes, but mostly lower CPU use for handling the packets, and especially in iSCSI environments, but not much for the lower packet size overhead…

In the bad old days, you could get one interrupt per packet, but those days
are long gone for high-speed NICs. Today, interrupt coalescing is quite common,
further reducing the benefits of jumbo frames.

NB, setting MTU to 9000 can actually work against performance when coalescing
on some popular OSes.  Ideally, the MTU should be set to what your workload
needs and not more.

Finally, a data point: using MTU of 1500 with ixgbe you can hit wire speed on a
modern CPU.
 -- richard

--

Richard.Elling at RichardElling.com
+1-760-896-4422