[OpenIndiana-discuss] persistent infiniband MTU setting

[Rich]

Aside: Linux defaults to running the IPoIB interface in CM, not UD, as
of OFED 1.3 (released in 2008).

As far as I'm aware, only Windows defaults to UD mode these days, and
that should go away in the next OFED release.

- Rich

On Thu, Mar 1, 2012 at 5:00 PM, Dustin Marquess <dmarquess at gmail.com> wrote:
> The MTU is 65520 because unlike in Linux, OI/Solaris default running
> the interface in Connected mode instead of Unreliable Datagram Mode.
> This is the linkmode=cm or linkmode=ud setting in dladm.  From
> ibp(7D):
>
>     By default, Connected Mode will be used by the each IB link.
>     This  behavior can be modified by changing the linkmode pro-
>     perty of the data link. See  the  EXAMPLES  section  of  the
>     dladm(1M) manual page for information .
>
>     Because ibp over connected mode attempts to use a large  MTU
>     (65520 bytes), applications should adapt to the large MTU to
>     get better performance, for example, adopting  a  large  TCP
>     window size.
>
> If you switch it to ud, then you'll see the MTU change.  Again, it
> won't let you change the MTU.  Why? Because also unlike Linux,
> OI/Solaris are smart enough to use the MTU that the subnet manager is
> set to, which is the proper thing to do.
>
> In theory, you shouldn't need to change anything, since it's smart
> enough to do it itself:
>
>     If only one side has enabled Reliable Connected  mode,  com-
>     munication  falls  back to datagram mode. The connected mode
>     instance uses Path MTU discovery to automatically adjust the
>     MTU  of a unicast packet if an MTU difference exists. Before
>     Path MTU discovery reduces the MTU for a  specific  destina-
>     tion,  several packets whose size exceeds the MTU of Unreli-
>     able Datagram mode is dropped.
>
> -Dustin
>
> On Mon, Feb 20, 2012 at 5:42 AM, Johan Guldmyr <johan.guldmyr at csc.fi> wrote:
>> Hello!
>>
>> This is with oi_151a2 on a HP DL360 G7 with a "592520-B21   HP IB 4X QDR
>> CX-2 PCI-e G2 Dual Port HCA".
>> I'm trying to set the MTU to the same on this host as it is on the existing
>> nodes on the IB network.
>>
>> Setting it with ifconfig goes through, but "dladm show-linkprop -p mtu"
>> still says it's 65520.
>>
>> How to make the MTU setting persistent? If it's only set with ifconfig is it
>> actually using the specified MTU?
>>
>> http://docs.oracle.com/cd/E23824_01/html/821-1475/ibp-7d.html
>>
>> Refers to a /kernel/drv/ibp.conf
>>
>> But this file is quite empty.
>>
>> ---
>> root at oi:~# dladm create-part -l ibp0 -p mtu=2044 -P FFFF pFFFF.ibp0
>> dladm: partition ffff creation over ibp0 failed: operation not supported
>>
>> root at oi:~# dladm create-part -l ibp0 -P FFFF pFFFF.ibp0
>> root at oi:~# ifconfig pFFF.ibp0 plumb
>>
>> root at oi:~# dladm show-link
>> LINK        CLASS     MTU    STATE    BRIDGE     OVER
>> ibp0        phys      65520  up       --         --
>> ibp1        phys      65520  up       --         --
>> bnx0        phys      1500   up       --         --
>> bnx2        phys      1500   down     --         --
>> bnx3        phys      1500   down     --         --
>> bnx1        phys      1500   down     --         --
>> pFFFF.ibp0  part      65520  up       --         ibp0
>>
>> root at oi:~# ifconfig pFFFF.ibp0 10.0.0.1 netmask 255.255.255.0 mtu 2044 up
>> root at oi:~# ifconfig pFFFF.ibp0
>> pFFFF.ibp0: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU>
>> mtu 2044 index 4
>>        inet 10.0.0.1 netmask fffffc00 broadcast 10.0.0.255
>>        ipib 80:0:0:4a:fe:80:0:0:0:0:0:0:0:2:c9:3:0:4c:7f:f1
>>
>> root at oi:~# dladm set-linkprop -p mtu=2044 pFFFF.ibp0
>> dladm: warning: cannot set link property 'mtu' on 'pFFFF.ibp0': operation
>> not supported
>>
>> root at oi:~# dladm show-linkprop -p mtu
>> LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
>> ibp0         mtu             r-   65520          65520          1500-65520
>> ibp1         mtu             r-   65520          65520          1500-65520
>> bnx0         mtu             rw   1500           1500           1500
>> bnx2         mtu             rw   1500           1500           1500
>> bnx3         mtu             rw   1500           1500           1500
>> bnx1         mtu             rw   1500           1500           1500
>> pFFFF.ibp0   mtu             r-   65520          65520          65520
>>
>> Connectivity works fine (can ping another host over IPoIB).
>>
>> --
>> Johan Guldmyr
>>
>> _______________________________________________
>> OpenIndiana-discuss mailing list
>> OpenIndiana-discuss at openindiana.org
>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss