[OpenIndiana-discuss] NFSv4 problems between Ubuntu 10.04 and OI 151a

[Albert Chin]

We have an oi_151a server as our NFS/ZFS/iSCSI server. We have one
interface (aggregated across two physical interfaces) dedicated to NFS
traffic. We wanted to add a second interface dedicated to KVM guest
disk image I/O traffic.

(Solaris NFS server)
# cat /etc/release
                      OpenIndiana Development oi_151a X86
        Copyright 2010 Oracle and/or its affiliates. All rights reserved.
                        Use is subject to license terms.
                           Assembled 01 September 2011
# ifconfig aggr0 (sanji.il.thewrittenword.com)
aggr0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 10.191.57.70 netmask ffffff00 broadcast 10.191.57.255
# ifconfig igb2 (sanji.vmnfs.il.thewrittenword.com)
igb2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5
        inet 10.191.62.1 netmask ffffff00 broadcast 10.191.62.255

One of our NFS clients is an Ubuntu Lucid 10.04LTS server:
# ifconfig eth0 (trunks.il.thewrittenword.com)
eth0      Link encap:Ethernet  HWaddr 00:1e:67:1c:fd:d4  
          inet addr:10.191.57.22  Bcast:10.191.57.255 Mask:255.255.255.0
          inet6 addr: fe80::21e:67ff:fe1c:fdd4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:38915848 errors:0 dropped:6 overruns:6 frame:0
          TX packets:38670920 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:34440950064 (34.4 GB)  TX bytes:47295732818 (47.2 GB)
          Memory:b1d20000-b1d40000 
# ifconfig eth5 (trunks.vmnfs.il.thewrittenword.com)
eth5      Link encap:Ethernet  HWaddr 00:1b:21:d3:f6:09  
          inet addr:10.191.62.2  Bcast:10.191.62.255 Mask:255.255.255.0
          inet6 addr: fe80::21b:21ff:fed3:f609/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:23382 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1075 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1915540 (1.9 MB)  TX bytes:140018 (140.0 KB)
          Memory:b1b00000-b1b80000 

NFSv4 mounts between 10.191.57.22 on the Ubuntu client and
10.191.57.70 on the OpenSolaris server work fine:
  (Ubuntu client)
  # nfsstat -m
  /opt/tww from sanji:/opt/tww/
   Flags: rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.191.57.22,minorversion=0,local_lock=none,addr=10.191.57.70

  /opt/vms from sanji:/opt/vms/
   Flags: rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.191.57.22,minorversion=0,local_lock=none,addr=10.191.57.70

  # ls -l /opt/vms
  total 3567662
  drwxr-xr-x  5 root root          5 2011-09-07 12:52 esxi
  drwxr-xr-x  7 root root          7 2011-11-15 01:38 images
  drwxrwxr-x 81 root root         81 2012-04-19 17:48 iso
  drwxr-xr-x  6 root root          6 2012-04-24 06:19 kvm
  -rw-rw-rw-  1 root root    7610401 2011-12-09 02:14 ogata-boot.tar.bz2
  -rw-rw-rw-  1 root root 4852152320 2011-12-09 02:29 ogata-root.tar
  -rw-------  1 root root 1001652224 2011-12-09 03:13 ogata-root.tar.bz2
  drwxrwxrwt 23 root src          23 2011-02-04 18:33 vmwareserver

Now, if I try to mount on the second set of interfaces:
  (Ubuntu client)
  # mount sanji:/opt/vms /mnt
  # nfsstat -m
  /mnt from sanji.vmnfs:/opt/vms/
   Flags: rw,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.191.62.2,minorversion=0,local_lock=none,addr=10.191.62.1
  # ls -l /mnt
  total 3
  drwxr-xr-x 7 root root 7 2011-11-15 01:38 images
  # mount sanji.vmnfs:/opt/tww /mnt
  mount.nfs: access denied by server while mounting sanji.vmnfs:/opt/tww

  (Solaris NFS server)
  # zfs get sharenfs tww/opt/vms
  tww/opt/vms  sharenfs rw,root=trunks.il.thewrittenword.com:trunks.iscsi.il.thewrittenword.com:trunks.vmnfs.il.thewrittenword.com local
  # zfs get sharenfs tww/opt/tww
  tww/opt/tww  sharenfs rw=.il.thewrittenword.com:.vmnfs.il.thewrittenword.com  local
  # sharemgr show -vp zfs
  ...
      zfs/tww/opt/vms nfs=() nfs:sys=(root="trunks.il.thewrittenword.com:trunks.iscsi.il.thewrittenword.com:trunks.vmnfs.il.thewrittenword.com" rw="*")
            /opt/vms
            /opt/vms/images
            /opt/vms/images/hpvm
            /opt/vms/images/kvm
            /opt/vms/images/vios
            /opt/vms/images/vmware
            /opt/vms/images/xen
  ...

Why doesn't ls -l of /opt/vms look like ls -l of /mnt? And how can I
get them to behave the same?

-- 
albert chin (openindiana-discuss at mlists.thewrittenword.com)