[OpenIndiana-discuss] 10G/40G net perf - what are you doing?

[Jason Matthews]

*So my 10G/40G network performance sucks. I have a combination of 
Chelsio and Intel 10 and 40G NICs. The 10G Intels do a little better. I 
have dug deeper into systems with Chelsio cards than Intel cards but 
what I found is that the receive buffer on the Chelsio card fills up and 
then it starts sending flow control frames. I have been unable to alter 
this behavior. I would say the average top rate for Chelsio is about 
3Gbps (independent of whether it is a 10G/40G card)and the Intel cards 
and reach but cannnot sustain 5Gbps and then regress towards 3Gbps.

The base use case is backing up data to four systems. Some of them are 
Intel/Kontron S5520UR with X5650 CPUs and Chelsio T520-LL-CR cards  and 
the others are SuperMicro H11SSL-C EPYC 7401 based systems with 
T580-SO-CR 40G cards all connected to Force10/Dell S4048-ON switches. No 
errors on any interfaces other than flow control frames coming from the 
servers. The performance is the same running back to back on 10G or 40G.

I setup two systems back to back and had Chelsio login to take a look. 
They they set some tcp buffers and locked one ore more CPUs to the NICs 
and then gave up. Here are somethings that I have tried.

*

*   ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 15000
    ndd -set /dev/tcp tcp_time_wait_interval 10000

    # default 128
    ndd -set /dev/tcp tcp_conn_req_max_q   4096

    # setup for high transaction volume and drop super slow talkers
    # default 1024
    ndd -set /dev/tcp tcp_conn_req_max_q0 98304

    # default 180000
    ndd -set /dev/tcp tcp_keepalive_interval  600000

    # default 180000
    ndd -set /dev/tcp tcp_ip_abort_cinterval 30000

    # default 60000
    ndd -set /dev/tcp tcp_rexmit_interval_max 240000

    # default 32768
    /usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 2049

    # hpn-ssh related values
    ndd -set /dev/tcp tcp_recv_hiwat 491520
    ndd -set /dev/tcp tcp_xmit_hiwat 491520
    ndd -set /dev/tcp tcp_cwnd_max 655340
*

*   # disable lso
ndd -set /dev/ip ip_lso_outbound 0# increase tcp buffersipadm set-prop 
-p max_buf=4194304 tcp ipadm set-prop -p recv_buf=1048576 tcp ipadm 
set-prop -p send_buf=1048576 tcp What are you doing differently to get 
something near wired speed performance? I will take any delta I can get. 
Thanks,*

*j. *