[OpenIndiana-discuss] ZFS remote receive

[Jim Klimov]

2012-10-31 13:58, Sebastian Gabler wrote:
>> 2012-10-30 19:21, Sebastian Gabler wrote:
>>> >Whereas that's relative: performance is still at a quite miserable 62
>>> >MB/s through a gigabit link. Apparently, my environment has room for
>>> >improvement.
>> Does your gigabit ethernet use Jumbo Frames (like 9000 or up to 16KB,
>> depending on your NICs, switches and other networking gear) for
>> unrouted (L2) storage links? It is said that traditional MTU=1500
>> has too many overheads with packet size and preamble delays between
>> packets that effectively limit a gigabit to 700-800Mbps...


> The MTU is on 1500 on source and target system, and there are no
> fragmentations happening.

The point of Jumbo frames (in unrouted L2 ethernet segments) is to
remove many overheads - CSMA/CD delays being a large contributor -
and send unfragmented chunks of 9-16Kb in size, increasing the local
network efficiency.

 > On the target system I am seeing writes up to
> 160 MB/s with frequent zpool iostat probes. When iostat probes are up to
> 5s+, there is a steady stream of 62 MB/s.

I believe this *may* mean that your networking buffer receives data
into memory (ZFS cache) at 62Mb/s, then every 5s the dirty cache
is sent to disks during TXG commit at whatever speed in can burst
(160Mb/s in your case).

 > At this time I am not sure if
> that is indeed a networking issue. I am also not sure how jumbo frames
> could provide an intersting benefit here. The usually alleged 15% (which
> are already on the high side) are not in the scope of making or breaking
> the use case.

Mostly elaborated above.

Other ways to reduce networking lags were discussed by other
responders, including use of netcat to pipe the stream quickly,
ssh without encryption/with cheap encryption/with HPC patches.

Based on some experience with NFS and OpenVPN I might also
suggest to try UDP vs. TCP (i.e. with netcat), though this
would probably play on the unsafe side - UDP-based programs
include retries like NFS (or accept the drop of data like VoIP),
as they deem necessary, and ZFS-send probably doesn't do this;
it is rather fragile already.

//Jim