[OpenIndiana-discuss] Network device config problem

[Udo Grabowski (IMK)]

Hi,
we have strange problems on oi 151a7 with network device management.
On our X4540 thumper boxes, the onboard e1000g* network entries are
sometimes randomly overwritten by net* entries (machine had crashed
before), which cannot be removed and block the use of the original
device entries (trying to add it back to the aggregation aggr1 where
it originally belongs to). Tried to modify /etc/dladm/datalink.conf,
but net0 gets readded, and /dev/net/net0 stays, e1000g0 does not
appear there. Tried to move net0 to e1000g0, but, expectedly, this
caused havoc... We have network/physical:default active, but it looks
like nwam is somehow interferring, although completely disabled in
SVM. /etc/path_to_inst entries go to the correct /devices/ links,
all /dev/e1000g* are correct, all link targets are there with the
correct major/minor entries. Tried datalink-management restart
after modifications, reboot etc., no change.

What's going on here ? Where do the net* entries come from ?
Where they are permanently recorded ?

It goes like this:

# dladm show-phys
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
e1000g0      Ethernet             unknown    0      half      e1000g0
e1000g1      Ethernet             up         1000   full      e1000g1
e1000g2      Ethernet             up         1000   full      e1000g2
e1000g3      Ethernet             up         1000   full      e1000g3

# ll /dev/e1000g*
lrwxrwxrwx   1 root     root          32 Sep 25 14:06 /dev/e1000g -> 
../devices/pseudo/clone at 0:e1000g
lrwxrwxrwx   1 root     root          57 Sep 25 14:06 /dev/e1000g0 -> 
../devices/pci at 0,0/pci1022,7458 at 10/pci8086,1011 at 1:e1000g0
lrwxrwxrwx   1 root     root          59 Sep 25 14:06 /dev/e1000g1 -> 
../devices/pci at 0,0/pci1022,7458 at 10/pci8086,1011 at 1,1:e1000g1
lrwxrwxrwx   1 root     root          57 Sep 25 14:06 /dev/e1000g2 -> 
../devices/pci at 0,0/pci1022,7458 at 10/pci8086,1011 at 2:e1000g2
lrwxrwxrwx   1 root     root          59 Sep 25 14:06 /dev/e1000g3 -> 
../devices/pci at 0,0/pci1022,7458 at 10/pci8086,1011 at 2,1:e1000g3

# ll /dev/net   (e1000g0 is missing, instead net0 uses its entry)
total 0
crw-rw-rw-   1 root     sys      211, 1001 Feb 19 14:42 aggr1
crw-rw-rw-   1 root     sys       23,  2 Feb 19 14:45 e1000g1
crw-rw-rw-   1 root     sys       23,  3 Feb 19 14:45 e1000g2
crw-rw-rw-   1 root     sys       23,  4 Feb 19 14:42 e1000g3
crw-rw-rw-   1 root     sys       23,  1 Feb 19 14:45 net0

# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
e1000g0     phys      1500   ?        --         --
e1000g1     phys      1500   up       --         --
e1000g2     phys      1500   up       --         --
e1000g3     phys      1500   up       --         --
aggr1       aggr      1500   up       --         e1000g1 e1000g2

# dladm show-aggr -L (the working aggregation which had e1000g0 previously)
LINK        PORT         AGGREGATABLE SYNC COLL DIST DEFAULTED EXPIRED
aggr1       e1000g1      yes          yes  yes  yes  no        no
--          e1000g2      yes          yes  yes  yes  no        no

# dladm show-aggr -x
LINK        PORT           SPEED DUPLEX   STATE     ADDRESS            PORTSTATE
aggr1       --             1000Mb full    up        0:14:4f:41:35:41   --
             e1000g1        1000Mb full    up        0:14:4f:41:35:41   attached
             e1000g2        1000Mb full    up        0:14:4f:41:35:42   attached

# dladm add-aggr -l e1000g0 aggr1    (can't add it to aggregation)
dladm: add operation failed: object not found

# dladm add-aggr -l net0 aggr1       (net0 effectively is e1000g0)
dladm: invalid <link> 'net0': invalid argument

# ifconfig e1000g0 plumb  (can plumb this)

# ifconfig -a
aggr1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
         inet xxxxxxxxxxxx    netmask fffffc00 broadcast xxxxxxxx.255
         ether 0:14:4f:41:35:41
e1000g0: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 6
         inet 0.0.0.0 netmask 0
         ether 0:14:4f:41:35:40

# dladm show-phys (state seems ok after plumb)
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
e1000g0      Ethernet             up         1000   full      e1000g0
e1000g1      Ethernet             up         1000   full      e1000g1
e1000g2      Ethernet             up         1000   full      e1000g2
e1000g3      Ethernet             up         1000   full      e1000g3

# cat /etc/dladm/datalink.conf  (deleted the net0 entry)
e1000g0 
class=int,1;media=int,4;phyinst=int,1;phymaj=int,23;devname=string,e1000g0;cpus=string,0-31;
e1000g1 
class=int,1;media=int,4;phyinst=int,2;phymaj=int,23;devname=string,e1000g1;cpus=string,0-31;
e1000g2 
class=int,1;media=int,4;phyinst=int,3;phymaj=int,23;devname=string,e1000g2;cpus=string,0-31;
e1000g3 
class=int,1;media=int,4;phyinst=int,4;phymaj=int,23;devname=string,e1000g3;cpus=string,0-31;
aggr1 
class=int,4;media=int,4;lacp_timer=int,1;lacp_mode=int,1;force=boolean,0;fix_macaddr=boolean,0;policy=int,7;portnames=string,e1000g0:e1000g1:e1000g2:;nports=int,3;key=int,0;cpus=string,0-31;


-- 
Dr.Udo Grabowski    Inst.f.Meteorology a.Climate Research IMK-ASF-SAT
www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology            http://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany  T:(+49)721 608-26026 F:-926026