[OpenIndiana-discuss] CIFS performance issues

[James Carlson]

Robin Axelsson wrote:
>> If you have two interfaces inside the same zone that have the same IP
>> prefix, then you have to have IPMP configured, or all bets are off.
>> Maybe it'll work.  But probably not.  And was never been supported that
>> way by Sun.
> The idea I have with using two NICs is to create a separation between
> the virtual machine(s) and the host system so that the network activity
> of the virtual machine(s) won't interfere with the network activity of
> the physical host machine.

Nice idea, but it unfortunately won't work.  When two interfaces are
plumbed up like that -- regardless of what VM or bridge or hub or
virtualness there might be -- the kernel sees two IP interfaces
configured with the same IP prefix (subnet), and it considers them to be
completely interchangeable.  It can (and will!) use either one at any
time.  You don't have control over where the packets go.

Well, unless you get into playing tricks with IP Filter.  And if you do
that, then you're in a much deeper world of hurt, at least in terms of
performance.

> The virtual hub that creates the bridge between the VM network ports and
> the physical port tap into the network stack of the host machine and I
> suspect that this configuration is not entirely seamless. I think that
> the virtual bridge interferes with the network stack so letting the
> virtual bridge have its own network port to play around with has turned
> out to be a good idea, at least when I was running OSOL b134 - OI148a.

I think you're going about this the wrong way, at least with respect to
these two physical interfaces.

I suspect that the right answer is to plumb only *ONE* of them in the
zone, and then use the other by name inside the VM when creating the
virtual hub.  That second interface should not be plumbed or configured
to use IP inside the regular OpenIndiana environment.  That way, you'll
have two independent paths to the network.

> I suppose I could try to configure the IPMP, I guess I will have to
> throw away the DHCP configuration and go for fixed IP all the way as
> DHCP only gives two IP addresses and I will need four of them. But then
> we have the problem with the VMs and how to separate them from the
> network stack of the host.

It's possible to have DHCP generate multiple addresses per interface.
And it's possible to use IPMP with just one IP address per interface (in
fact, you can use it with as little as one IP address per *group*).  And
it's possible to configure an IPMP group with some static addresses and
some DHCP.

But read the documentation in the man pages.  IPMP may or may not be
what you really want here.  Based on the "isolation" demands mentioned,
I suspect it's not.  The only reason I mentioned it is that your current
IP configuration is invalid (unsupported, might not work, good luck with
that) without IPMP -- that doesn't mean you should use IPMP, but that
you should rethink the whole configuration.

One of the many interesting problems that happens with multiple
interfaces configured on the same network is that you get multicast and
broadcast traffic multiplication: each single message will be received
and processed by each of the interfaces.  Besides the flood of traffic
this causes (and the seriously bad things that will happen if you do any
multicast forwarding), it can also expose timing problems in protocols
that are listening to those packets.  When using IPMP, one working
interface is automatically designated to receive all incoming broadcast
and multicast traffic, and the others are disabled and receive unicast
only.  Without IPMP, you don't have that protection.

Another interesting problem is source address usage.  When the system
sends a packet, it doesn't really care what source address is used, so
long as the address is valid on SOME interface on the system.  The
output interface is chosen only by the destination IP address on the
packet -- not the source -- so you'll see packets with source address
"A" going out interface with address "B."  You might think you're
controlling interface usage by binding some local address, but you're
really not, because that's not how IP actually works.  With IPMP,
there's special logic engaged that picks source IP addresses to match
the output interface within the group, and then keeps the connection (to
the extent possible) on the same interface.

But those are just two small ways in which multiple interfaces
configured in this manner are a Bad Thing.  A more fundamental issue is
that it was just never designed to be used that way, and if you do so,
you're a test pilot.

> I will follow these instructions if I choose to configure IPMP:
> http://www.sunsolarisadmin.com/networking/configure-ipmp-load-balancing-resilience-in-sun-solaris/

Wow, that's old.  You might want to dig up something a little more
modern.  Before OpenIndiana branched off of OpenSolaris (or before
Oracle slammed the door shut), a lot of work went into IPMP to make it
much more flexible.

>> If you run "/sbin/route monitor" when the system is working fine and
>> leave it running until a problem happens, do you see any output produced?
>>
>> If so, then this could fairly readily point the way to the problem.
>>
> WIth one port I mean that only one port is physically connected to the
> switch, all other ports but one are disconnected. So I guess ifconfig
> <port_id> unplumb would have no effect on such ports.

Not so!

In terms of getting the kernel's IRE entries correct, it doesn't matter
so much where the physical wires go.  It matters a whole lot what you do
with "ifconfig."

> I managed to reproduce a few short freezes while "/sbin/route monitor"
> was running over ssh but it didn't spit out any messages, perhaps I
> should run it on a local terminal instead.

If the freeze is damaging network communication, then doing that to some
destination (such as "> /tmp/monitor.out") that won't be affected by the
network is probably a good idea.

It's a guess.  I'm guessing that what's really going on is that you have
either interfaces, IREs, or ARP entries flapping in the breeze due to
the odd configuration on this system.  "route monitor" should reveal that.

But it could be almost anything else.  Looking at kstats and/or
capturing network traffic may be necessary in order to find the problem.

> I looked at the time stamps
> of the entries in the /var/adm/messages and they do not match the
> freeze-ups by the minute.

I assume that refers to the NWAM messages previously reported.  No, I
don't think those are the proximate cause of your problem.

-- 
James Carlson         42.703N 71.076W         <carlsonj at workingcode.com>