[OpenIndiana-discuss] Symbolic links broken on NFS clients

[Martyn Klassen]

I narrowed down the issue to being related to using RDMA over my Infiniband connection. When mounting without using RDMA, ie tcp, the symbolic links come through as expected. I tried snooping on the Infiniband interface, but I get no traffic at all when using NFS over RDMA (snoop was running during mounting, symbolic link access, and unmounting), which I assume means snoop cannot intercept the RDMA protocol communication, since I get lots of traffic when during the same operations except using TCP on the IPoIB connection. 

The clients mount NFS over RDMA shares from another CentOS based server and do not have any issues with the symbolic links, so I am fairly confident that the issue is in the OI server and not the clients. 

I would like to get RDMA working, since it is the default NFS protocol and my network supports it, but my OI/Solaris expertise is limited. However I am willing to try any debugging suggestions that might provide more helpful information.

Martyn

> 
> On Mon, Apr 9, 2012 at 10:41 AM, Martyn Klassen <mklassen at imaging.robarts.ca> wrote:
> 
>> I have a OI 151a2 server sharing numerous filesystems via NFSv4 to CentOS
>> 5/6 clients, and symbolic links are broken on the clients. I get IO errors
>> trying to read the symbolic links on the clients. However, I can destroy
>> and create a symbolic link on the client and it will work for the one
>> client until the NFS share is unmount and remounted, then the symbolic link
>> is broken again. Curiously, not all symbolic links are broken, only
>> approximately 99% of them. It does not seem to be related to size,
>> destination, or location of the symbolic link. I have also tried NFSv3 with
>> the same results. The links are always fine on the OI server. Any
>> suggestion on what might be causing this would be appreciated.
>> 
>> The nfs server settings are
>> 
>> $ sharectl get nfs
>> servers=16
>> lockd_listen_backlog=32
>> lockd_servers=20
>> lockd_retransmit_timeout=5
>> grace_period=90
>> server_versmin=2
>> server_versmax=4
>> client_versmin=3
>> client_versmax=3
>> server_delegation=on
>> nfsmapid_domain=****
>> max_connections=-1
>> protocol=ALL
>> listen_backlog=32
>> device=
>> 
>> The shares are created from ZFS filesystems using sharenfs set to
>> 
>> ro=@
>> 10.0.0.0/8,rw=@10.2.40.0/24:@10.2.20.0/24:@10.2.150.0/24:@10.1.40.0/24:@10.1.20.0/24:@10.1.150.0/24,root=@10.2.40.0/24:@10.2.20.0/24:@10.2.150.0/24:@10.1.40.0/24:@10.1.20.0/24:@10.1.150.0/24
>> 
>> Because the filesystems will also be shared via CIFS they were created
>> with casesensitivity=mixed and nbmand=on as the Solaris documents suggests
>> for CIFS shared filesystems.
>> 
>> Martyn
>> 
> 
> A snoop trace showing the readdir and open attempts may be useful here. (Do
> a full 'snoop -o blah.out host <client>' and follow connections in
> Wireshark or similar tool).
> 
> -Albert