[OpenIndiana-discuss] CIFS performance issues

[Robin Axelsson]

On 2012-01-24 21:59, James Carlson wrote:
> Robin Axelsson wrote:
>>> If you have two interfaces inside the same zone that have the same IP
>>> prefix, then you have to have IPMP configured, or all bets are off.
>>> Maybe it'll work.  But probably not.  And was never been supported that
>>> way by Sun.
>> The idea I have with using two NICs is to create a separation between
>> the virtual machine(s) and the host system so that the network activity
>> of the virtual machine(s) won't interfere with the network activity of
>> the physical host machine.
> Nice idea, but it unfortunately won't work.  When two interfaces are
> plumbed up like that -- regardless of what VM or bridge or hub or
> virtualness there might be -- the kernel sees two IP interfaces
> configured with the same IP prefix (subnet), and it considers them to be
> completely interchangeable.  It can (and will!) use either one at any
> time.  You don't have control over where the packets go.
>
> Well, unless you get into playing tricks with IP Filter.  And if you do
> that, then you're in a much deeper world of hurt, at least in terms of
> performance.
Here's what the virtualbox manul says about bridged networking:

"*Bridged networking*:

This is for more advanced networking needs such as network simulations 
and running servers in a guest. When enabled, VirtualBox connects to one 
of your installed network cards and exchanges network packets directly, 
circumventing your host operating system's network stack.

With bridged networking, VirtualBox uses a device driver on your 
_*host*_ system that filters data from your physical network adapter. 
This driver is therefore called a "net filter" driver. This allows 
VirtualBox to intercept data from the physical network and inject data 
into it, effectively creating a new network interface in software. When 
a guest is using such a new software interface, it looks to the host 
system as though the guest were physically connected to the interface 
using a network cable: the host can send data to the guest through that 
interface and receive data from it. This means that you can set up 
routing or bridging between the guest and the rest of your network.

For this to work, VirtualBox needs a device driver on your host system. 
The way bridged networking works has been completely rewritten with 
VirtualBox 2.0 and 2.1, depending on the host operating system. From the 
user perspective, the main difference is that complex configuration is 
no longer necessary on any of the supported host operating systems."

>> The virtual hub that creates the bridge between the VM network ports and
>> the physical port tap into the network stack of the host machine and I
>> suspect that this configuration is not entirely seamless. I think that
>> the virtual bridge interferes with the network stack so letting the
>> virtual bridge have its own network port to play around with has turned
>> out to be a good idea, at least when I was running OSOL b134 - OI148a.
> I think you're going about this the wrong way, at least with respect to
> these two physical interfaces.
>
> I suspect that the right answer is to plumb only *ONE* of them in the
> zone, and then use the other by name inside the VM when creating the
> virtual hub.  That second interface should not be plumbed or configured
> to use IP inside the regular OpenIndiana environment.  That way, you'll
> have two independent paths to the network.
Perhaps the way to do it is to create a dedicated jail/zone for 
VIrtualBox to run in and "plumb the e1000g2" to that zone. I'm a little 
curious as to how this would affect the performance I'm not sure if you 
have to split up the CPU cores etc between zones or if that is taken 
care of as the zones pretty much share the same kernel (and its task 
scheduler).
>> I suppose I could try to configure the IPMP, I guess I will have to
>> throw away the DHCP configuration and go for fixed IP all the way as
>> DHCP only gives two IP addresses and I will need four of them. But then
>> we have the problem with the VMs and how to separate them from the
>> network stack of the host.
> It's possible to have DHCP generate multiple addresses per interface.
> And it's possible to use IPMP with just one IP address per interface (in
> fact, you can use it with as little as one IP address per *group*).  And
> it's possible to configure an IPMP group with some static addresses and
> some DHCP.
In order to make DHCP generate more IP addresses I guess I have to 
generate a few (virtual) MAC addresses. Maybe ifconfig hadles this 
internally.
>
> But read the documentation in the man pages.  IPMP may or may not be
> what you really want here.  Based on the "isolation" demands mentioned,
> I suspect it's not.  The only reason I mentioned it is that your current
> IP configuration is invalid (unsupported, might not work, good luck with
> that) without IPMP -- that doesn't mean you should use IPMP, but that
> you should rethink the whole configuration.
>
> One of the many interesting problems that happens with multiple
> interfaces configured on the same network is that you get multicast and
> broadcast traffic multiplication: each single message will be received
> and processed by each of the interfaces.  Besides the flood of traffic
> this causes (and the seriously bad things that will happen if you do any
> multicast forwarding), it can also expose timing problems in protocols
> that are listening to those packets.  When using IPMP, one working
> interface is automatically designated to receive all incoming broadcast
> and multicast traffic, and the others are disabled and receive unicast
> only.  Without IPMP, you don't have that protection.
>
> Another interesting problem is source address usage.  When the system
> sends a packet, it doesn't really care what source address is used, so
> long as the address is valid on SOME interface on the system.  The
> output interface is chosen only by the destination IP address on the
> packet -- not the source -- so you'll see packets with source address
> "A" going out interface with address "B."  You might think you're
> controlling interface usage by binding some local address, but you're
> really not, because that's not how IP actually works.  With IPMP,
> there's special logic engaged that picks source IP addresses to match
> the output interface within the group, and then keeps the connection (to
> the extent possible) on the same interface.
>
> But those are just two small ways in which multiple interfaces
> configured in this manner are a Bad Thing.  A more fundamental issue is
> that it was just never designed to be used that way, and if you do so,
> you're a test pilot.
This was very interesting and insightful. I've always wondered how 
Windows tell the difference between two network connections in a 
machine, now I see that it doesn't. Sometimes this can get corrupted in 
Windows and sever the internet connection completely. If I understand 
correctly, the TCP stack in Windows is borrowed from Sun. I guess this 
is a little OT, it's just a reflection.
>> I will follow these instructions if I choose to configure IPMP:
>> http://www.sunsolarisadmin.com/networking/configure-ipmp-load-balancing-resilience-in-sun-solaris/
> Wow, that's old.  You might want to dig up something a little more
> modern.  Before OpenIndiana branched off of OpenSolaris (or before
> Oracle slammed the door shut), a lot of work went into IPMP to make it
> much more flexible.
I'll see if there is something more up-to-date. There are no man entries 
for 'ipmp' in OI and 'apropos' doesn't work for me.
>
>>> If you run "/sbin/route monitor" when the system is working fine and
>>> leave it running until a problem happens, do you see any output produced?
>>>
>>> If so, then this could fairly readily point the way to the problem.
>>>
>> WIth one port I mean that only one port is physically connected to the
>> switch, all other ports but one are disconnected. So I guess ifconfig
>> <port_id>  unplumb would have no effect on such ports.
> Not so!
>
> In terms of getting the kernel's IRE entries correct, it doesn't matter
> so much where the physical wires go.  It matters a whole lot what you do
> with "ifconfig."
Ok, but when it is not connected it has no IP address (as it is 
configured over DHCP) that can interfere with the multicast and the IP 
setup. Maybe this is a problem when the address is static.
>
>> I managed to reproduce a few short freezes while "/sbin/route monitor"
>> was running over ssh but it didn't spit out any messages, perhaps I
>> should run it on a local terminal instead.
> If the freeze is damaging network communication, then doing that to some
> destination (such as ">  /tmp/monitor.out") that won't be affected by the
> network is probably a good idea.
>
> It's a guess.  I'm guessing that what's really going on is that you have
> either interfaces, IREs, or ARP entries flapping in the breeze due to
> the odd configuration on this system.  "route monitor" should reveal that.
>
> But it could be almost anything else.  Looking at kstats and/or
> capturing network traffic may be necessary in order to find the problem.
I'll report back when I experience the next freeze-up...
>> I looked at the time stamps
>> of the entries in the /var/adm/messages and they do not match the
>> freeze-ups by the minute.
> I assume that refers to the NWAM messages previously reported.  No, I
> don't think those are the proximate cause of your problem.
>

I've been playing around with 'ifconfig <interface> unplumb' (as a 
superuser of course) but it doesn't appear to do anything on e1000g2. As 
a memory refresher, here's the setup:

e1000g1: 10.40.137.185, DHCP (the computer name is associated with this 
address in the /etc/hosts file)
e1000g2: 10.40.137.171, DHCP (bridged network of the VM is attached to 
this port)
rge0: <no IP address>, DHCP (no cable attached)

No error message comes after that command and when issuing 'ifconfig -a' 
everything looks the same, and the '/sbin route monitor' doesn't yield 
any messages. If I do the same on rge0 it will disappear in the IPv4 
section of the 'ifconfig -a' output but remain in the IPv6 section. I 
also see messages coming out of the route monitor; RTM_IFINFO, 
RTM_DELETE and RTM_DELADDR... as a result of the unplumb command on rge0 
which I think should be expected.

I can see that e1000g2 is operating in Promiscuous state (Whatever that 
means) which the other ethernet connections are not.

I tried with 'ifconfig e1000g2 down' and the "UP" flag disappeared from 
the port in 'ifconfig -a'. Once again this only applies to IPv4, IPv6 
remains unaffected. The route monitor yielded messages this time. The IP 
address 10.40.137.171 is still there however and the bridged network 
connection of the VM seems to be unaffected by this command (which would 
be desired if that command really severed the IFP from the host).

If I try to unplumb it again (ifconfig e1000g2 unplumb) I get the error:
"ifconfig: cannot unplumb e1000g2: Invalid argument provided"
which is a bit strange. The route monitor yields the message (in 
digested form):
RTM_LOSING (Kernel Suspects partitioning) <DST,GATEWAY,NETMASK,IFA> 
default 10.40.137.1 default default
right after this command.

As a comparison; if I do the same on the rge0 which is already unplumbed 
I get:
"ifconfig: cannot unplumb rge0: Interface does not exist"
and no messages on the route monitor.

Maybe I'm using a bad monkey wrench...

Robin.