[OpenIndiana-discuss] persistent infiniband MTU setting

[Dustin Marquess]

The MTU is 65520 because unlike in Linux, OI/Solaris default running
the interface in Connected mode instead of Unreliable Datagram Mode.
This is the linkmode=cm or linkmode=ud setting in dladm.  From
ibp(7D):

     By default, Connected Mode will be used by the each IB link.
     This  behavior can be modified by changing the linkmode pro-
     perty of the data link. See  the  EXAMPLES  section  of  the
     dladm(1M) manual page for information .

     Because ibp over connected mode attempts to use a large  MTU
     (65520 bytes), applications should adapt to the large MTU to
     get better performance, for example, adopting  a  large  TCP
     window size.

If you switch it to ud, then you'll see the MTU change.  Again, it
won't let you change the MTU.  Why? Because also unlike Linux,
OI/Solaris are smart enough to use the MTU that the subnet manager is
set to, which is the proper thing to do.

In theory, you shouldn't need to change anything, since it's smart
enough to do it itself:

     If only one side has enabled Reliable Connected  mode,  com-
     munication  falls  back to datagram mode. The connected mode
     instance uses Path MTU discovery to automatically adjust the
     MTU  of a unicast packet if an MTU difference exists. Before
     Path MTU discovery reduces the MTU for a  specific  destina-
     tion,  several packets whose size exceeds the MTU of Unreli-
     able Datagram mode is dropped.

-Dustin

On Mon, Feb 20, 2012 at 5:42 AM, Johan Guldmyr <johan.guldmyr at csc.fi> wrote:
> Hello!
>
> This is with oi_151a2 on a HP DL360 G7 with a "592520-B21   HP IB 4X QDR
> CX-2 PCI-e G2 Dual Port HCA".
> I'm trying to set the MTU to the same on this host as it is on the existing
> nodes on the IB network.
>
> Setting it with ifconfig goes through, but "dladm show-linkprop -p mtu"
> still says it's 65520.
>
> How to make the MTU setting persistent? If it's only set with ifconfig is it
> actually using the specified MTU?
>
> http://docs.oracle.com/cd/E23824_01/html/821-1475/ibp-7d.html
>
> Refers to a /kernel/drv/ibp.conf
>
> But this file is quite empty.
>
> ---
> root at oi:~# dladm create-part -l ibp0 -p mtu=2044 -P FFFF pFFFF.ibp0
> dladm: partition ffff creation over ibp0 failed: operation not supported
>
> root at oi:~# dladm create-part -l ibp0 -P FFFF pFFFF.ibp0
> root at oi:~# ifconfig pFFF.ibp0 plumb
>
> root at oi:~# dladm show-link
> LINK        CLASS     MTU    STATE    BRIDGE     OVER
> ibp0        phys      65520  up       --         --
> ibp1        phys      65520  up       --         --
> bnx0        phys      1500   up       --         --
> bnx2        phys      1500   down     --         --
> bnx3        phys      1500   down     --         --
> bnx1        phys      1500   down     --         --
> pFFFF.ibp0  part      65520  up       --         ibp0
>
> root at oi:~# ifconfig pFFFF.ibp0 10.0.0.1 netmask 255.255.255.0 mtu 2044 up
> root at oi:~# ifconfig pFFFF.ibp0
> pFFFF.ibp0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU>
> mtu 2044 index 4
>        inet 10.0.0.1 netmask fffffc00 broadcast 10.0.0.255
>        ipib 80:0:0:4a:fe:80:0:0:0:0:0:0:0:2:c9:3:0:4c:7f:f1
>
> root at oi:~# dladm set-linkprop -p mtu=2044 pFFFF.ibp0
> dladm: warning: cannot set link property 'mtu' on 'pFFFF.ibp0': operation
> not supported
>
> root at oi:~# dladm show-linkprop -p mtu
> LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
> ibp0         mtu             r-   65520          65520          1500-65520
> ibp1         mtu             r-   65520          65520          1500-65520
> bnx0         mtu             rw   1500           1500           1500
> bnx2         mtu             rw   1500           1500           1500
> bnx3         mtu             rw   1500           1500           1500
> bnx1         mtu             rw   1500           1500           1500
> pFFFF.ibp0   mtu             r-   65520          65520          65520
>
> Connectivity works fine (can ping another host over IPoIB).
>
> --
> Johan Guldmyr
>
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss